Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query on Google Experiment?
Hello All, I am doing A/B testing via google analytic experiment now my query is for my ecommerce site homepage i am trying to add newsletter in 3 different way so it will be variation A, Variation B and Variation C. So my query is what should be then original page? Currently there is no newsletter form on homepage. Do I consider original page as Variation A? i.e. abcd.com?variation-A as original page? Then google will decide winner between original page i.e. variation A and Variation B & C Thanks!
Reporting & Analytics | | dsouzac0 -
In google analytic for google /cpc it is showing url with 404 which even not exists in my database
Hello All, In google analytic for google /cpc it is showing url with 404 which even not exists in my database that also more than 300 per day. How can it is possible? it is showing /black-friday-offers but I don't have such page. Thanks!
Reporting & Analytics | | pragnesh96390 -
Google Analytics to sub domains
Hi, I have a site xyz.com and two separate sites on sub domains xyz.com/abc and xyz.com/def. What's the best way to go about with GA such that I can get all the data in the same place. Should I use the GA code for xyz.com on sub domains as well? Or should I create separate profiles?
Reporting & Analytics | | mayanksaxena0 -
How does switching to HTTPS effect Google Analytics?
We are looking at making our site HTTPS. We have been using the same Google Analytics account for years and I like having the historical data. All of our pages will be the same, we are just going to redirect from the http to https. Does anything need to be done with Google Analytics? What about other addons such as Optimizely, Crazy Egg, or Share this?
Reporting & Analytics | | EcommerceSite0 -
Mobile Site on Google Analytics
Hi mozzers, We just launched a mobile site and I was wondering what are the main steps to follow for gettting your mobile site tracked via GA (m.example.com)? We have a profile for www.example.com GATC: javascript or PHP to install? Should the profile be on a subdomain? What else to consider when implementing a mobile site on GA? Thanks
Reporting & Analytics | | Ideas-Money-Art0 -
Why Webmaster tools shows different avg. positions?
Hi everyone, I am new to Seomoz, I loved this amazing SEO school 🙂 My questions is about webmaster tool. Webmaster tool shows that my site's avg. positions is 34 and its getting better every week. I even see some new queries, new avg. positions for that queries. That's great ! But when I search the avg position it is between 54 -60 and I don't see any new queries . Shouldnt we rely on what webmaster tells us about avg. positions? Thanks a lot
Reporting & Analytics | | EzgiGunyel-InfinPixels0 -
Google Analytics - New account creation broken
Is it just me or is the new Google Account Creation button not working. After i set it up, it just simply doesn't save.
Reporting & Analytics | | TheGrid0 -
Google Analytics All Kinds of Messed Up
Anyone have their google analytics go straight crazy on Thursday, 8/11/2011? 30% increase in visits No increase in visitors Huge drop in pageviews and time site Higher bounce rate More return users
Reporting & Analytics | | ChrisKolmar0