Why doesn't Moz crawler follow robots.txt?
-
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
-
Which I am ok with, but why am I getting duplicate content?
-
Yes, it doesn't tell them which pages not to crawl - just not to index them
-
It has been used correctly. The site is a Magento site and they have it built in. There are a lot of filters for products so it uses rel=canonical to tell Google which to index.
-
rel=canonical is not really an robots instruction file - rel=canonical is to help with duplicate copy where you have the same or similar pages and your telling search engines which pages is the preferred page.
If you don't want pages crawling you have to tell Search engines in the robots file
-
Hi There,
Rel=canonical tags tell robots, which page is actually to index out of many.
For SEOs, canonicalization refers to individual web pages that can be loaded from multiple URLs. This is a problem because when multiple pages have the same content but different URLs, links that are intended to go to the same page get split up among multiple URLs. This means that the popularity of the pages gets split up. Unfortunately for web developers, this happens far too often because the default settings for web servers create this problem.
https://mza.seotoolninja.com/learn/seo/canonicalization
I feel you have not used it correctly, check the above article and see if it helps.
Thanks,
Vijay
-
So I made a mistake it isn't the robots.txt that is the issue. I am getting hit with a ton of duplicate content penalties so I figured that was it. The problem is that I have pages with rel=canonical tags that it is ignoring. Does Roger not read those?
-
Hi
Have to agree with the above, Rogerbot does listen to robot.txt file, unlike Bing - while they are getting better Bing ignores the robots.txt file frequently.
Ive analysed quite a few server logs over the years and Roger has always listened to the file - its usually a mistake the in the robots file.
There is an option to test your robots.txt file in GCS - while this is testing to see if Google will crawl the page - usually Roger has the same instructions as Google.
However if you are still pretty certain that Roger is ignoring robots.txt please DM your Server Logs and your website and I will take a look and analyse it for you (free of course).
Thanks
Andy
-
All major search engines, including Moz's crawler Rogerbot and Internet Archives, respect Robots.txt as a standard “robots exclusion protocol” to communicate with web crawlers and web robots.
In case you wish to exclude some specific information from all Search Engines, you can use the following sample code as reference to block specific directories.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/However, if you want to specifically block Mz's Rogerbot from crawling specific sections of your website. You may take the following reference code to block specific areas / directories in your website from rogerbot:
User-agent: Rogerbot
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/I hope this helps, If you have specific questions, please feel free to respond, I will be happy to answer them.
Regards,
Vijay
-
Hi there! Moz's crawler, rogerbot, does follow robots.txt. When he's not following robots.txt, it's usually because the robots.txt protocol is formatted improperly. Learn more about formatting your page here: https://mza.seotoolninja.com/learn/seo/robotstxt
For more information on Roger, including how to block him, head here: https://mza.seotoolninja.com/help/guides/moz-procedures/what-is-rogerbot
And if you want to test your formatting, try the Robots Checker here: https://support.google.com/webmasters/answer/6062598
If you're still unable to determine why rogerbot is crawling your site, feel free to write in to [email protected]!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz show me that my backlinks lost every day in my some important keywords
hi, moz show me that my backlinks lost every day in my some important keywords like: ثبت تغییرات شرکت ، ثبت شرکت ، .... but , i think that's not correct. can you explain to me that is for what? my website is: https://vanak.org thank you....
Link Explorer | | alirezatrade01230 -
Why my website Backlinks not getting Crawl by Moz?
Hi, I've the query related to backlinks of my website. Some of the high authority sites give backlinks to my website but these links are still not showing in Moz. My website is 6 months old and also continuously getting backlinks from high authority sites but still these are not showing on Moz and also not improving DA and PA of my website. I've also attached the screenshot of Moz link explorer results please check and guide me what to do with website so Moz will consider it and give some authority. And also guide me How much Moz takes time to crawl website backlinks and shown in their link explorer. Website URL: https://www.welderexpert.com Moz Expert Suggestions needed. Thanks, overview?site=welderexpert.com⌖=domain
Link Explorer | | KOidue0 -
Moz crawling doesn't show all of my pages
Hello, I'm trying to make an SEO on-pages audit of my website. When using the SEO crawl, i see only one page while i have much more pages in this website (200+). Is it an issue with the sitemap ? The robot.txt ? How can i check and correct this. My website is discoolver.com
Link Explorer | | MK-Discoolver1 -
Weekly and other data not available in moz tools
Dear All, Since last few days we are not getting updated results in moz tools. We get following warnings in the pages. "Hello! Weekly data for some campaigns is a bit behind schedule. Don't worry; it's processing now and will be available soon. Check out http://health.moz.com for the latest status on all of our tools. And thanks for your patience!" How much time it will take to show us the current results, so that we can properly work on it. Right now we are not able to measure any changes affected our site. Thanks, Om
Link Explorer | | omverma0 -
Open Site Explorer is finding old html Files that havn't been on my site in two years... even after a 301 Redirect. HELP!
Hello!
Link Explorer | | morganlindsaycole
My problem started when I became aware that when I checked my backlinks for the past two years, it states that no backlinks have been found. When I ran a site analysis on SEMrush - No backlinks are found on the URL, or Domain. There are 7 Backlinks on the Root Domain and those were configured in 2012. I made a second domain www.columbusweddingphotographersreviews.comwhere I linked to my domain at www.morganlindsayphotography.com so I could test that google had crawled both websites and after, still no backlink was found. I have also been published on a dozen or so wedding websites that has linked to my website where they are follow links and still nothing. (http://www.brendasweddingblog.com/blogs/2015/2/23/an-elegant-fall-wedding-in-ohio-with-morgan-lindsay-photography) **Website Background- **
In 2012 I had two separate websites - One for Seniors that was an HTML website I build in Dreamweaver at www.morganlindsayphotography/seniors and another for Wedding Clients found at www.morganlindsayphotography.com/Wedding - (wordpress) I had a Splash page wish was found atwww.morganlindsayphotography.com. Two years ago when I became aware splash pages were frowned upon in Google, I combined the two websites and stayed with the Wordpress which was www.morganlindsayphotography.com/Wedding
Because I did not want users to have to go to www.morganlindsayphotography.com/Wedding to view my url, Godaddy moved my wordpress site from thewww.morganlindsyphotography.com/Wedding directory towww.morganlindsayphotography.com When I ran the Open Site Explorer with Moz I found after runningwww.morganlindsayphotography.com the TOP pages on this domain according to Page Authority are old HTML files from my senior website, as well as old Posts from when my wordpress site was found atwww.moragnlindsayphotography.com/Weddings
No current pots or pages are showing up besideswww.morganlindsyphotography.com I do run a cache management system to speed up my system and recently cleaned out my .htcacess folder and still had no luck. This is difficulty something **Last night I made a 301 Redirect in my htaccess for all the old links pointing to the new links as best as I could. My htacess folder looks like this.. BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]</ifmodule> END WordPress Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /Wedding http://www.morganlindsayphotography.com/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /Wedding/ http://www.morganlindsayphotography.com/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /about.html http://www.morganlindsayphotography.com/about-morgan-lindsay/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /app.html http://www.morganlindsayphotography.com/blog/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /experience.html http://www.morganlindsayphotography.com/senior-sessions/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /index.html http://www.morganlindsayphotography.com/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /senior.html http://www.morganlindsayphotography.com/ohio-senior-photographer/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /seniorsconstruction.html http://www.morganlindsayphotography.com/ohio-senior-photographer/ Permanent URL redirect - generated by www.rapidtables.com Redirect 301 /Wedding/2012/06/22/brittany-reis-jason-mcclaflin-tiffin-ohio-wedding/ http://www.morganlindsayphotography.com/holy-family-church-columbus-wedding/ After I ran the open site moz explorer and the www.morganlindsayphotography/Wedding was still there..0 -
Moz can't crawl domain due to IP Geo redirect loop
Hi, I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages. In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time. Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use. Thanks, Mel
Link Explorer | | SalvationArmy0