Rogerbot not crawling our site
-
Has anyone else had issues with Roger crawling your site in the last few weeks? It shows only 2 pages crawled. I was able to crawl the site using Screaming Frog with no problem and we are not specifically blocking Roger via robots.txt or any other method. Has anyone encountered this issue? Any suggestions?
-
Hi there! Kristina from Moz's Help Team here.
I noticed that you're currently in conversation with my colleague, David, who is helping you out on this. It looks like he might have an idea of what's causing this to occur and he's waiting on the Crawl Test results to confirm if it's fixed for you.
I'll let David continue assisting you on this one, but as always you can reach out to our team with any technical issues such as this one by emailing us at [email protected].
Thanks so much,
-Kristina -
Who is your host? Some are starting to crack down on crawlers across the board with many blocking the likes of SEMrush. Might be worth checking if there are no other technical issues.
-Andy
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not crawling on my site
Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh0 -
Merging two Sites into One - how is Final Domain Authority Calculated?
Hi, If we merge two sites say, one with DA 35 and another 20 what will be resulting DA of the merged site? Secondly, will the result vary based on which Domain is kept and which one is 301ed? ps :- all content is transferred from one to another using 301. Regards,
Product Support | | Praveenshah10 -
Crawl test
I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?
Product Support | | Karen_Dauncey0 -
Link to Moz Pro home 404s, and then I stumbled upon Moz's staging site...
I'm wary of giving more information as it's probably best that little to no one be able to access Moz staging for one reason or another. I should also mention that this was done very unintentionally on my part. That said, from the 404 from Moz Pro home, I tried to access "My QA", and that's when I realized that everything I did thereafter was within staging.moz.com (and posted a question similar to this one). Does Moz permit access to their staging site, or did I stumble upon a mistake - or Moz-stake, if you will?
Product Support | | Lumina0 -
I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.
Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.
Product Support | | potterharry0 -
SEO Moz PRO app Isn't Crawling Anymore
Hi, We find the SEO Moz PRO app a great tool for us. What is the reason that it is not re-crawling the websites included in our campaigns anymore?
Product Support | | solution.advisor0 -
Latest Crawl Stats Not Showing
I got the email that Moz crawled my site for the week of February 8. I expected to log in and find some shinny new data, but new stats are not showing in my dashboard yet. Is there a delay? Is there something wrong?
Product Support | | ChadC0 -
Deleting a site
If we have a 3 of 5 of our "active site" allocation currently used, and then delete one, would we then only have 2 of 5 sites active sites used? Or would the previously used active site still count against our allocation?
Product Support | | lincolndigitalgroup1