Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz [email protected]
Also, you should check
- https://mza.seotoolninja.com/learn/seo/robotstxt
- https://support.google.com/webmasters/answer/6062608?hl=en
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Warnings, Notices, and Errors- don't know how to correct these
I have been watching my Notices, Warnings and Errors increase since I added a blog to our WordPress site. Is this effecting our SEO? We now have the following: 2 4XX errors. 1 is for a page that we changed the title and nav for in mid March. And one for a page we removed. The nav on the site is working as far as I can see. This seems like a cache issue, but who knows? 20 warnings for “missing meta description tag”. These are all blog archive and author pages. Some have resulted from pagination and are “Part 2, Part 3, Part 4” etc. Others are the first page for authors. And there is one called “new page” that I can’t locate in our Pages admin and have no idea what it is. 5 warnings for “title element too long”. These are also archive pages that have the blog name and so are pages I can’t access through the admin to control page title plus “part 2’s and so on. 71 Notices for “Rel Cononical”. The rel cononicals are all being generated automatically and are for pages of all sorts. Some are for a content pages within the site, a bunch are blog posts, and archive pages for date, blog category and pagination archive pages 6 are 301’s. These are split between blog pagination, author and a couple of site content pages- contact and portfolio. Can’t imagine why these are here. 8 meta-robot nofollow. These are blog articles but only some of the posts. Don’t know why we are generating this for some and not all. And half of them are for the exact same page so there are really only 4 originals on this list. The others are dupes. 8 Blocked my meta-robots. And are also for the same 4 blog posts but duplicated twice each. We use All in One SEO. There is an option to use noindex for archives, categories that I do not have enabled. And also to autogenerate descriptions which I do not have enabled. I wasn’t concerned about these at first, but I read these (below) questions yesterday, and think I'd better do something as these are mounting up. I’m wondering if I should be asking our team for some code changes but not sure what exactly would be best. http://www.seomoz.org/q/pages-i-dont-want-customers-to-see http://www.robotstxt.org/meta.html Our site is http://www.fateyes.com Thanks so much for any assistance on this!
Moz Pro | | gfiedel0 -
Competitive Domain Analysis.. Can anyone explain this
As you can see from the following 3 images below that 3 out of 4 companies have declined in Domain Authority however one has managed to actually increase and stay up. Can anyone explain how this may be for the company that has stayed up has lost total external followed link and has only increased linking C-block domains by 1 uLTyDWz,QvBCoka,xT1rsod#2 uLTyDWz,QvBCoka,xT1rsod#1 uLTyDWz,QvBCoka,xT1rsod#0
Moz Pro | | MeteorElec0 -
How many competitors can you track?
Hi All, How many competitors can you track in a campaign? It appears to only allow 3 competitors. Thanks,
Moz Pro | | toby-02
Toby1 -
Yo-Yo-ing rankings - what's going on?
I'm getting a little fed up and exasperated - if I'm honest! Our seoMoz rankings are yo-yoing on a weekly basis - extremely. One week we're in the top 3, the next week, we're not in the top 50. The week after that, we're back in the top 3. Then we're outside the top 50 again. I understand rankings go up and down sometimes - but this is every week, without fail. Not only that, but I also check my rankings with cuteRank and Firefox's Rank Checker and while those rankings are consistent with each other - seoMoz's rankings are always different and the only ones that Yoyo. Further, it's not just our rankings that Yoyo - it's those of our competitors too. They follow the same pattern except that when we go up, they go down, and vice versa, I'm just a little lost and unsure what to do, and I can't believe really that this is a true reflection of what is happening to us on Google every week. I know rankings change but surely Google aren't throwing us around like this? It is making doing any SEO on the site impossible because I never know where I stand. Any ideas or help would be wonderful. Thanks.
Moz Pro | | JetBookMike0 -
Is The Crawl Diagnostic tool working correctly?
The Crawl Diagnostic tool shows issues and displays a graph but they don't display the page specific results/suggestion like it used to. I get the "Congratulations, there are no pages affected by this issue!" message.
Moz Pro | | -PAUL-0 -
Blocking all robots except rogerbot
I'm in the process of working with a site under development and wish to run the SEOmoz crawl test before we launch it publicly. Unfortunately rogerbot is reluctant to crawl the site. I've set my robots.txt to disallow all bots besides rogerbot. Currently looks like this: User-agent: * Disallow: / User-agent: rogerbot Disallow: All pages within the site are meta tagged index,follow. Crawl report says: Search Engine blocked by robots.txt Yes Am I missing something here?
Moz Pro | | ignician0 -
What's with the new Q&A set up?
In the old Q&A set up your response was automatically given a thumbs up by you so that you got a mozpoint for every response you gave. Now you have to manually thumb up your responses. Why was this change made? Can we get it back to the way it was?
Moz Pro | | DanDeceuster1 -
Why isn't my Domain Authority much higher than my competitors'?
Looking at the Competitive Link Analysis report shows my domain outperforming my competitors on Domain MozRank, Domain MozTrust, total links, linking root domains, followed linking root domains, and linking c-blocks. But Domain Authority scores are very similar across all four sites with one competitor actually scoring higher than my domain. How should I unpick this result? I'm assuming that there must a problem with my link profile. Is that a valid assumption? Thanks, Tom fc7Ez.png
Moz Pro | | trbaldwin0