Unsolved Site Crawler not working but on-demand crawler working
-
Hi,
In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully.
I just like to know if in both these cases, it is roberbot that is used!
And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. .
Could you please us understand how could we possibly make site-crawler work?
I am happy to provide more details if you need any.Thanks
-
Hi,
I will double-check with firewall settings in our servers. Could you please share moz-pro site-crawler roger bot IP addresses/range? We will verify against our firewall rules.
Thanks
Shashi -
I am looking for roger bot site crawler IP addresses Please provide.
Thanks
-
@Aditi_08
Could you please help me on how to get IP addresses of Site Crawler? Just please note, Site Crawler is working before November so IP addresses were not blocked.Like it is mentioned before,
- no change in robots.txt
- no issue with rate limiting
- no changes in site-crawler configuration
-
@gilesd If you're experiencing issues with Moz Pro's Site Crawler showing that the site is banned while the On-Demand Crawler works fine, it might be due to changes or updates since November/December 2023. Both tools likely use the same crawler, "rogerbot," but differ in their operational schedules. The problem could be due to rate limiting or blocking by your server, IP blocking, changes in your robots.txt file, or updates in the Site Crawler configuration. To resolve this, check your robots.txt file to ensure it allows Moz's crawler, review server logs and firewall settings to ensure the crawler’s IP addresses aren’t blocked, and adjust rate limiting settings if necessary. Also, double-check the settings in Moz Pro to make sure there are no configurations causing the issue. If the problem persists, contact Moz support with detailed information about the error messages and any recent changes to your site’s configuration. Regular monitoring of your site’s interactions with automated tools and coordinating with your hosting provider can help prevent such issues in the future.
-
I am not sure why my reply not appearing here. Just for confirmation, replying again,
I like to confirm you -
There is no modification in Robots.txt
No issues with rate limit
Moz Pro settings are not changedWe are looking for your help to identify the issue.
Thanks
-
Thanks for your trouble shooting tips.
I assure you there has been nothing changed in robots.txt file or any settings in MozPro.
And there is frequency limit, Site Crawler triggers only once in 2 weeks.Thanks
-
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair -
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz Crawl seems to be stuck?
Hi all, It seems like moz has been stuck on crawling our site for a while now - I had a message of 'you will get a notification for when your site crawl is complete' for about 2 weeks now, and it doesn't seem to finish it? Any ideas why this happens and how to fix it? Thank you in advance.
Moz Tools | | StevenWalley0 -
Solved Site Crawl Won't Complete
How can I start/restart a new site crawl? I requested one 2 days ago on one of my sites, and it won't complete. It's only 150 pages -
Product Support | | PaulBarrs0 -
Infographic distribution sites, ideas & tools
What is the best way to get a infographic distributed for pure marketing? Is there a good way to get both digital and news related channels? big-mistake.php
Moz Pro | | jdcline1 -
Multiple Sites/Internal Pages Campaign
Can MOZ do reports/ranks/campaigns separately for each of our sites, then do separate keyword campaigns for specific internal content pages of each of the websites? For a law firm with both a defense and family law sites. We have multiple pages within each site, we will need to do separate and individual campaigns for assault, burglary, traffic tickets, ect. in our defense site. And we'll need other specific campaigns for other specific content in out family law site. Will Moz be able to accommodate our needs with their $99/mo plan or is that service only available for higher packages? (or is it even possible in ANY of their packages???) Thanks.
Moz Pro | | Wallin_Klarich0 -
SEOmoz crawler and duplicate content
Does anybody know if the SEOmoz crawler picks up canonical tags when its looking for duplicate content? I've got a ton of errors in one of my projects even though they all have canonical tags. Thanks!
Moz Pro | | neooptic0 -
Open Site Explorer Missing URL's
I see a link to my site on a couple different url's, but they are not listed in OSE. The links have been active for a long time too. Does OSE not track all inbound links from all sites? Thanks, Stephen
Moz Pro | | stats440 -
Unsubscribe to weekly crawl notifications never works
Hello! All of my campaigns have the box 'Weekly crawl completed for campaign ...' unticked under Campaign Settings, yet for all of them I still receive an email regularly with the subject 'New crawl completed for ...'. How do I stop this? Is there a bug here? Adam Bishop
Moz Pro | | arbishop0