Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Different statistics in Google Analytics
Hi! A client of mine is having some trouble on his Google Analytics overview. When he checks the yearly report with number of visitors, new visitors, returning, ... he gets statistics that doesn't match mine. Also, he gets a notation from Google that no hits were fired, although the script is still there. On the other hand, I (logged in with another account) am seeing all statistics correctly. FYI: no filters were installed, nor segments, ... Why does the client see different and incorrect statistics? There should be a setting or something I'm overseeing. Hope you guys can help.
Reporting & Analytics | | conversal0 -
How to Configure & Tracking Images in Google Analytic?
Hello Experts, Can anyone tell me for my eCommerce site how to configure images tracking in google analytic? And where can I see the image traffic in google analytic? Not sure I am asking correct question but confuse about image traffic. Thanks! Wrights!
Reporting & Analytics | | wright3350 -
Webmaster Tools and Geolocation
Our site gets a significant amount of traffic from Venezuela, about 9% based on our Google Analytics (the US being 63%). According to Webmaster Tools we get 48% of our clicks from Venezuela. That discrepancy is massive, considering we are an English site. My theory is that they are not allowed/capable of using secure search in Venezuela for some legal reason and therefore their searches are much more 'visible'. Does anyone have any insight?
Reporting & Analytics | | evansluke0 -
Google Analytics question
Hi, I recently relaunched my site on a wordpress CMS. The relaunch has gone well except for a category of traffic within Google Analytics which is referral traffic from google.co.uk (its a UK site) which has dropped off hugely. The non-paid search traffic from google.co.uk is the same or better than it was. Can anyone tell me what the difference is between referral traffic from google.co.uk and non-paid search traffic from google.co.uk. Thanks for your help. Ben
Reporting & Analytics | | benacuity0 -
Spike in 404 Errors from a redirecting domain
All, The non www version of a site I own redirects to the www version. Recently, WMT began showing a big spike in 404 errors (1,000+) on the non www version of the site. Site traffic is off about 15% since the spike in 404 errors. There was also a brief period about two weeks ago, where the site went down due to an issue with the code that has now been resolved. Any ideas how WMT is showing 404 errors on redirected pages? Thanks, John
Reporting & Analytics | | JSOC0 -
Roger says I have 378 404 errors
Hi The report shows that I have 404 errors in the about of almost 300 but they appear to me to be active and part of the site. ??? Examples: NOTE: warning - these pages discuss BDSM and have some nude images here are some on the list http://bestslavetraining.com/groups/discipline-and-punishment/?acpage=6 http://bestslavetraining.com/groups/general-b-d-s-m/?acpage=13 also here is one that does not work http://bestslavetraining.com/activity/activity/ WHY does it show 404 errors for urls that work??? what am I missing and what in the world am I to correct if they appear on the SEOmoz error log.???
Reporting & Analytics | | Cuffsmaster0 -
Export all links on page with anchor text
As the question says really. Is there a tool that will export all the links on a page and anchor text related to those links into Excel? Thanks.
Reporting & Analytics | | RikkiD220 -
Google and bing search filed commands
Dose someone have / know a full list / resource with commands for google and bing ? Including filters for those commands ? (site:domain.com -filter etc) (like: site:domain.com, link:domain.com etc) I use the basic ones b ut I know there are much more and that there are several filters that can be used with success to filter down results. Thanks.
Reporting & Analytics | | eyepaq1