Crawl diagnostic issue?
-
I'am sorry if my English isn't very good, but this is my problem at the moment:
On two of my campagnes I get a weird error on Moz Analytics:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
Moz Analytics points to an url that starts with: http:/**/None/**www.????.com. We don't understand how Moz indexed this non-existing page that starts with None? And how can we solve this error?
I hope that someone can help me.
-
Hi MOZ,
I'am sorry that I have not previously responded. The problem has been solved. Thanks!
Also thanks to Pixel for the response!
Greetz,
Sam
-
Hi Nettt!
I apologize for any confusion and can confirm there is no issue on your side. One of our crawlers failed causing some campaigns crawled on Aug 29th attempt to follow the strange /None/ URL you are seeing in your diagnostics. I've submitted a re-crawl for all of your campaigns affected so you should see updated data by this Friday.
Hope this helps!
-
"I have checked the URL, and it is not our own website that has the error."
is this the problem?
Could you take a screen grab of the problem it might help better.
-
Thanks for the respons, Pixelbypixel!
I have checked the URL, and it is not our own website that has the error.
We have checked the robots.txt and it should not cause any problem. We have n't recently changed it.
I Think that Moz is causing it, but I am not sure..
-
Is the URL correct on Moz pro? It also seems like your robots.txt is blocking Moz which you may want to look into.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving from http to https: image duplicate issue?
Hello everyone, We have recently moved our entire website virtualsheetmusic.com from http:// to https:// and now we are facing a question about images. Here is the deal: All webpages URLs are properly redirected to their corresponding https if they are called from former http links. Whereas, due to compatibility issues, all images URLs can be called either via http or https, so that any of the following URLs work without any redirect: http://www.virtualsheetmusic.com/images/icons/ResponsiveLogo.png https://www.virtualsheetmusic.com/images/icons/ResponsiveLogo.png Please note though that all internal links are relative and not absolute. So, my question is: Can that be a problem from the SEO stand point? In particular: We have thousands of images indexed on Google, mostly images related to our digital sheet music preview image files, and many of them are ranking pretty well in the image pack search results. Could this change be detrimental in some way? Or doesn't make any difference in the eyes of Google? As I wrote above, all internal links are relative, so an image tag like this one: Hasn't changed at all, it is just loaded in a https context. I'll wait for your thoughts on this. Thank you in advance!
Intermediate & Advanced SEO | | fablau0 -
International Href Lang Tag Parameter Issue
Hey, let's say I'm on the following page.. site.com/product-name/product-code/?d=womens I view the page source and it looks like this.. My question is, should I remove the parameter for the hreflang tag???? I just need some clarification that NO parameter page should have a canonical tag and / or href lang with parameters..
Intermediate & Advanced SEO | | ggpaul5620 -
Pages with rel "next"/"prev" still crawling as duplicate?
Howdy! I have a site that is crawling as "duplicate content pages" that is really just pagination. The rel next/prev is in place and done correctly but Roger Bot and Google are both showing duplicated content + duplicate page titles & meta's respectively. The only thing I can think of is we have a canonical pointing back at the URL you are on - we do not have a view all option right now and would not feel comfortable recommending it given the speed implications and size of their catalog. Any experience, recommendations here? Something to be worried about? /collections/all?page=15"/>
Intermediate & Advanced SEO | | paul-bold0 -
Rankings disappeared on main 2 keywords - are links the issue?
Hi, I asked a question around 6 months ago about our rankings steadily declining since April of 2013. I did originally reply to that topic a few days ago, but as it's so old I don't think it's been noticed. I'm posting again here, if that's an issue I'm happy to delete. Here it is for reference: http://moz.com/community/q/site-rankings-steadily-decreasing-do-i-need-to-remove-links Since the original post, I have done nothing linkbuilding-wise except posting blog posts and sharing them on Facebook, G+ and Twitter. There are some links in there which don't look great (ie spammy seo directories, which I'm sending removal requests to) although quite a lot of others are relevant. Here's my link profile: <a rel="nofollow" target="_blank">http://www.opensiteexplorer.org/links?site=www.thomassmithfasteners.com</a> I've tried to make the site more accessible - we now have a simple, responsive design and I've tried to make the content clear and concise. In short, written for humans rather than search engines. As of the end of November, 'nuts and bolts' has now disappeared completely, and 'bolts and nuts' is page 8. There are many pages much higher which are not as relevant and have no links. We still rank highly for more specialised terms - ie 'bsw bolts' and 'imperial bolts' are still page 1, but not as high as before. We get an 'A' grade on the on-page grader for 'nuts and bolts, and most above us get F. I was cautious about removing links as our profile doesn't seem too bad but it does seem as if it's that. There are a fair few questionable directories in there, no doubt about that, but our overall practice in recent years has been natural building and link earning. So - I've created a spreadsheet and identified the bad links - ie directories with any SEO connotations. I am about to submit removal requests, I thought two polite requests a couple of weeks apart prior to disavowing with Google. But am I safe to disavow straight away? I say this as I don't think I'll get too many responses from those directories. I am also gradually beefing up the content on the shop pages in case of any 'thin content' issues after advice on the previous post. I noticed 100s of broken links in webmaster tools last week due to 2 broken links on our blog that repeated on every page and have fixed those. I have also been fixing errors W3C compliance-wise. Am I right to do all this? Can anyone offer any suggestions? I'm still not 100% sure if this is Panda, Penguin or something else. My guess is Penguin, but the decline started in March 2013, which correlates with Panda. Best Regards and thanks for any help, Stephen
Intermediate & Advanced SEO | | stephenshone0 -
Crawl Budget on Noindex Follow
We have a list of crawled product search pages where pagination on Page 1 is indexed and crawled and page 2 and onward is noindex, noarchive follow as we want the links followed to the Product Pages themselves. (All product Pages have canonicals and unique URLs) Orr search results will be increasing the sets, and thus Google will have more links to follow on our wesbite although they all will be noindex pages. will this impact our carwl budget and additionally have impact to our rankings? Page 1 - Crawled Indexed and Followed Page 2 onward - Crawled No-index No-Archive Followed Thoughts? Thanks, Phil G
Intermediate & Advanced SEO | | AU-SEO0 -
Duplicate Content Issue
Why do URL with .html or index.php at the end are annoying to the search engine? I heard it can create some duplicate content but I have no idea why? Could someone explain me why is that so? Thank you
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Duplication Issue?
One of our copywriters has just written a blog to be posted on our own company blog to be reviewed by myself, however I had noticed that the blog post has some duplication issues with one of our own product pages, about 60% duplication, is it still worth posting? Will search engines still index the blog post? Kind Regards,
Intermediate & Advanced SEO | | Paul780 -
202 error page set in robots.txt versus using crawl-able 404 error
We currently have our error page set up as a 202 page that is unreachable by the search engines as it is currently in our robots.txt file. Should the current error page be a 404 error page and reachable by the search engines? Is there more value or is it a better practice to use 404 over a 202? We noticed in our Google Webmaster account we have a number of broken links pointing the site, but the 404 error page was not accessible. If you have any insight that would be great, if you have any questions please let me know. Thanks, VPSEO
Intermediate & Advanced SEO | | VPSEO0
