Why doesn't Moz crawler follow robots.txt?
-
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
-
Which I am ok with, but why am I getting duplicate content?
-
Yes, it doesn't tell them which pages not to crawl - just not to index them
-
It has been used correctly. The site is a Magento site and they have it built in. There are a lot of filters for products so it uses rel=canonical to tell Google which to index.
-
rel=canonical is not really an robots instruction file - rel=canonical is to help with duplicate copy where you have the same or similar pages and your telling search engines which pages is the preferred page.
If you don't want pages crawling you have to tell Search engines in the robots file
-
Hi There,
Rel=canonical tags tell robots, which page is actually to index out of many.
For SEOs, canonicalization refers to individual web pages that can be loaded from multiple URLs. This is a problem because when multiple pages have the same content but different URLs, links that are intended to go to the same page get split up among multiple URLs. This means that the popularity of the pages gets split up. Unfortunately for web developers, this happens far too often because the default settings for web servers create this problem.
https://mza.seotoolninja.com/learn/seo/canonicalization
I feel you have not used it correctly, check the above article and see if it helps.
Thanks,
Vijay
-
So I made a mistake it isn't the robots.txt that is the issue. I am getting hit with a ton of duplicate content penalties so I figured that was it. The problem is that I have pages with rel=canonical tags that it is ignoring. Does Roger not read those?
-
Hi
Have to agree with the above, Rogerbot does listen to robot.txt file, unlike Bing - while they are getting better Bing ignores the robots.txt file frequently.
Ive analysed quite a few server logs over the years and Roger has always listened to the file - its usually a mistake the in the robots file.
There is an option to test your robots.txt file in GCS - while this is testing to see if Google will crawl the page - usually Roger has the same instructions as Google.
However if you are still pretty certain that Roger is ignoring robots.txt please DM your Server Logs and your website and I will take a look and analyse it for you (free of course).
Thanks
Andy
-
All major search engines, including Moz's crawler Rogerbot and Internet Archives, respect Robots.txt as a standard “robots exclusion protocol” to communicate with web crawlers and web robots.
In case you wish to exclude some specific information from all Search Engines, you can use the following sample code as reference to block specific directories.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/However, if you want to specifically block Mz's Rogerbot from crawling specific sections of your website. You may take the following reference code to block specific areas / directories in your website from rogerbot:
User-agent: Rogerbot
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/I hope this helps, If you have specific questions, please feel free to respond, I will be happy to answer them.
Regards,
Vijay
-
Hi there! Moz's crawler, rogerbot, does follow robots.txt. When he's not following robots.txt, it's usually because the robots.txt protocol is formatted improperly. Learn more about formatting your page here: https://mza.seotoolninja.com/learn/seo/robotstxt
For more information on Roger, including how to block him, head here: https://mza.seotoolninja.com/help/guides/moz-procedures/what-is-rogerbot
And if you want to test your formatting, try the Robots Checker here: https://support.google.com/webmasters/answer/6062598
If you're still unable to determine why rogerbot is crawling your site, feel free to write in to [email protected]!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
Site: www.kpmg.us Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter". Updated robots.txt to allow rogerbot full access: User-agent: rogerbot
Link Explorer | | KPMG-Search-Social
Disallow: Any ideas how to get roger to crawl my site????1 -
Apart from spying on competitors back link what else can be done in MOZ?
I've recently started trail version in MOZ and I want to explore every possible way that can help me in building my website better. My website Sanctum Consulting (Visas and Immigration)
Link Explorer | | Manifeat90 -
Strange error in MOZ report
I get the following warning about our domain name in Link Explorer Moz tool You entered the URL debtacademy.com which redirects to www.hugedomains.com/domain_profile.cfm?d=debtacademy&e=com. Click here to analyze www.hugedomains.com/domain_profile.cfm?d=debtacademy&e=com instead. Please advice me. How I can fix it.
Link Explorer | | jeffreyjohnson0 -
Does moz pro standard give access to open site explorer & keyword explorer?
Does moz pro standard give access to open site explorer & keyword explorer?
Link Explorer | | SearchOpt1 -
My title has a TM symbol and Moz says I don't have the keyword in my title
Moz gave me a poor rating because I don't have the keyword in my title but I do, it just has a trademark symbol. Pelican 1500 Case is the keyword I'd like to rank for on this page: http://www.thepelicanstore.com/pelican-1500-case-1014.aspx Moz says the keyword doesn't exist on the page. Should I ignore it, remove it or otherwise change it? I feel the trademark symbol would help with CTR in the title.
Link Explorer | | bREALmarketing0 -
Error getting your data in moz ose
Why am I receiving a "There was an error getting your data" in moz ose? Everything worked fine yesterday but now I'm having trouble getting link metrics for my site.
Link Explorer | | TitanDigital1 -
Why are no-follow links in my blog comments across the web showing up as "equity-passing"?
When I make comments on blog posts (and there's a link in my comment or my name is the link to my site), the links are always no-follow (as they should be). But, when I check Open Site Explorer, the new links show up as equity-passing. Are they actually passing equity or is this a mistake?
Link Explorer | | infotrust20 -
How can I use Open Site Explorer to Identify why my own URL's are appearing as inbound links
I'm a new member of Moz, and am trying to understand why I am seeing my sites URL's in my Inbound Link results in Open Site Explorer. There are quite a few, and they are all product category URL's that are generated by my stores database. This is not happening with .html URL's. Any thoughts?
Link Explorer | | Artfx0