Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ive been using moz for just a minute now , i used it to check my website and find quite a number of errors , unfortunately i use a wordpress website and even with the tips , is till dont know how to fix the issues.
ive seen quite a number of errors on my website hipmack.co a wordpress website and i dont know how to begin clearing the index errors or any others for that matter , can you help me please? ghg-1.jpg
Moz Pro | | Dogara0 -
Advanced popup for WP website
Hi guys, I'm looking for advanced popup for WP website.
Moz Pro | | EdmondHong87
Im looking something with advanced filters. For example: Filter the visitors by the operation system (For mobile).Will show to new visitors, but exclude to returns visitors.
Will exclude visitors that sign up to the website.
Filter the traffic...Popup will show only to visitors from FB, Google organic etc... Well, i believe that you understand what i need...
Anyone know something like this ? Thank you!0 -
Should I block .ashx files from being indexed ?
I got a crawl issue that 82% of site pages have missing title tags
Moz Pro | | thlonius
All this pages are ashx files (4400 pages).
Should I better removed all this files from google ?0 -
Page Authority and Google updates favouring websites with black hat practices ?
Can someone explain how is it that most of the competitors I have online and that rank in first page of the search results almost entirely get links ( in the thousands) and still have higher or equal domain/page authority than mine? I went 1 by 1 checking all their links and they mostly come from sex pages, and non related sites. I say stop creating angry pandas and penguins and start taking out of the game people that just play dirty. Thanks.
Moz Pro | | AbellSEO0 -
Can increasing website pages decrease domain authority?
Hello Mozzers! Say there is a website with 100 pages and a domain authority of 25. If the number of pages on this website increases to 10,000 can that decrease its domain authority or affect it in any way?
Moz Pro | | MozAddict0 -
Stats from Moz bar and OSE not changed over last three months - new update?
I have noticed for a few of my websites that the SEOMoz stats have not changed over the last three months; PR, MR and even total external links. Is both the tool bar and OSE going through further updates and not yet pulling through accurate results? Any advice on this would be much appreciated.
Moz Pro | | Adido-1053990 -
Using Keyword Difficulty Report along with OSE to calculate SEO Pricing?
Has anyone thought of or created a algorithm or Excel function that uses the results from the Keyword Difficulty Report percentage of difficulty and SERP ranking analysis set against potential client's own OSE rankings to come up with an estimated SEO pricing tool?
Moz Pro | | easystreetint0