Can't work out robots.txt issue.
-
Hi
I'm getting crawl errors that MOZ isn't able to access my robots.txt file but it seems completely fine to me? Any chance anyone can help me understand what might be the issue?
-
I would follow the advise from Roman, I tried to access the file via a few ways as well and it doesn't seem to be an issue to load it at all.
-
Ok, I made a quick test of your robot.txt file and looks fine, making HTTP status code test and shows me 200 code which is ok. Also, you need to make sure that your robot.txt file is accessible for the Moz crawler
- Remember to put your file is in the top-level directory of your web server.
- Also, check if your hosting provider is not blocking third-party crawlers (server level)
Here you can test the status code of your robot.txt file** https://httpstatus.io/ **If you're still having trouble this is the email of Moz [email protected]
Also, you should check
- https://mza.seotoolninja.com/learn/seo/robotstxt
- https://support.google.com/webmasters/answer/6062608?hl=en
Best of luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not working
Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh0 -
When Should I Ignore Moz's Report Canonical Missing?
I'm dealing with an eCommerce website which has a category, subcategory, products. Moz is showing all of these and the individual products as missing a canonical. The site is very thin on content at the moment, but all the pages are clearly different, and I don't see why they need a canonical unless this is some rule that eCommerce sites have to follow. Should I ignore Moz's missing canonical report? My understanding is if the product appears in multiple categories, then a canonical should be put in place to the product. Any advice would be appreciated. Christina
Moz Pro | | ChristinaRadisic0 -
Competitive Domain Analysis.. Can anyone explain this
As you can see from the following 3 images below that 3 out of 4 companies have declined in Domain Authority however one has managed to actually increase and stay up. Can anyone explain how this may be for the company that has stayed up has lost total external followed link and has only increased linking C-block domains by 1 uLTyDWz,QvBCoka,xT1rsod#2 uLTyDWz,QvBCoka,xT1rsod#1 uLTyDWz,QvBCoka,xT1rsod#0
Moz Pro | | MeteorElec0 -
Why don't Google+ URL's work in OSE?
Is there any reason why Google+ URLs does not work in OSE? Is it just that it is a secure URL or is there something bigger there? Why? Be cool to determine every website the person has been published on; especially if it is rel="author" verified. Jeff
Moz Pro | | WebBizIdeas1 -
Error on duplicated content, but when checking shouldn't been possible
Dear all, Every week I look at the different crawl reports for our website, since the start of my SeoMoz membership the Errors for duplicated content and duplicated Title is rising. But if I take out the .csv file and look in more detail, and select a pages which is marked as duplicated content, a canonical is actually existing on this page. So it shouldn't be an warning, I have no idea what the issue could be. For example pagesare marked as duplicated content, <colgroup><col width="966"></colgroup>
Moz Pro | | Letty
| http://www.zylom.com/es/descargar-juegos/3-en-raya/?sortby=2 |
| http://www.zylom.com/es/descargar-juegos/3-en-raya/?startnumber=60&sortby=2 |
| http://www.zylom.com/es/descargar-juegos/3-en-raya/?startnumber=80&sortby=2 | the parameters after '?' (question mark) are necessary for our internal system. To overcome duplicated content we coded that a canonical tag onis placed on every page with parameters and the main page is http://www.zylom.com/es/descargar-juegos/3-en-raya/ but it doesn't seem to work, because my error warnings are still rising. Please advice me Kind regards, Ms Letty van Eembergen0 -
My OnPage dosent work, why?
When im trying to use the on-page grade analysis seomoz says that is empty and there is now keywords.. this happens with 3of 5 of my campaigns.. Please help
Moz Pro | | majako0 -
How can a site have a backlink from Barclays website?
Hi, I have entered a competitiors website www.my-wardrobe.com into Open Site to see who they get links from and to my surprise they have a load from Barclays Business Banking. When I visit the page I can not see the links. But if I search the pages source code for my-wardrobe, there I have it, a link to my-wardrobe.com. How have they done this? Surely Barclays haven't sold them it? And more so, why are they receiving link juice when you cant even see the link on the Barclays page in question - http://www.barclays.co.uk/BusinessBanking/P1242557952664 Thanks | |
Moz Pro | | YNWA
| | <a <span="">href</a><a <span="">="</a>http://www.my-wardrobe.com" class="popup" title="Link opens in a new window" rel='' onmousedown="dcsMultiTrack('DCS.dcsuri','BusinessBankingfromBarclays/Footer/wwwmywardrobecom', 'WT.ti', '','WT.dl','1');"> |
| | www.my-wardrobe.com |
| |
|
| | |0 -
Where can I view all of the Questions that I have Subscribed to?
I recently asked a question about viewing the questions I have answered already: http://www.seomoz.org/q/where-can-i-view-all-of-the-questions-that-i-have-answered It turns out the link was right under my nose (but it appears a lot of people were having this issue). I'm now having the problem of not being able to see where I can view all the Q&As that I have subscribed to. Again, I may just be missing a link that's right in front of me but it certainly isn't obvious. Any help much appreciated!
Moz Pro | | NickPateman810