Crawler triggering Spam Throttle and creating 4xx errors
-
Hey Folks,
We have a client with an experience I want to ask about.
The Moz crawler is showing 4xx errors. These are happening because the crawler is triggering my client's spam throttling. They could increase from 240 to 480 page loads per minute but this could open the door for spam as well.
Any thoughts on how to proceed?
Thanks! Kirk
-
Thank you Dave!
-
Hey Kirk! We built our crawler to obey robots.txt crawl-delay directives. In the future, if this is ever an issue, you can use the crawl delay to slow Rogerbot down to a more reasonable speed. However, we don't recommend adding a crawl delay larger than 10 or Rogerbot might not be able to finish the crawl of your site.
Just add a crawl delay directive to your robots.txt file like this:
User-agent: rogerbot
Crawl-delay: 10Here's a good article that explains more about this technique: https://mza.bundledseo.com/learn/seo/robotstxt. I hope this helps, feel free to reach out if you have any other questions!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
H1 Errors and False Positives
Since the inception of our new website back in 2018, we have had no H1 issues, but now, we are popping positive for H1 errors. As seen in the attached image, we have H1 tags, but it doesnt seem that your crawlers are identifying them now. Is there a reason for its? qYbGp6P.jpg
Moz Bar | | nshelton56830 -
Moz Crawl - 804 : HTTPS (SSL) error encountered when requesting page.
Got an issue sending a Crawl Request to https://www.usernamebuddy.com/ " "804 : HTTPS (SSL) error encountered when requesting page." I have tried to recrawl several times now same issue keeps occurring. I cannot see an error when I access the site am I missing something, if so how can I diagnose the issue and sort the problem? I have reviewed the source and cannot use any http: resources.
Moz Bar | | GrouchyKids0 -
Error: 804 : HTTPS (SSL) error encountered when requesting page
In my crawl report I'm getting the error: 804 : HTTPS (SSL) error encountered when requesting page. How can I fix this? .
Moz Bar | | Yesi.Ortega0 -
MozBar 3.1.54 released: Spam Score and Smart Icon
Hey guys! We just launched MozBar version 3.1.54 for Chrome. 2 exciting new features: we now have Spam Score integrated into the MozBar (for Logged In users) the MozBar icon itself will now show DA when the MozBar is off. Click the icon to toggle this state (you can also click again to turn it completely off) We also fixed a few bugs, and removed Google Authorship from the Markup tab. Those of you who already have the MozBar installed, it will automatically update. For those new to the MozBar - you can download it here - https://chrome.google.com/webstore/detail/mozbar/eakacpaijcpapndcfffdgphdiccmpknp?hl=en Also big shout out to all of our MozBar users, we are now at 243k Chrome Installs. A full 110k increase from the version 3 launch in June last year. Thank you all! Love to hear your feedback below! Jon
Moz Bar | | jon.white3 -
Create a report with keyword, label, difficulty, global search volume, and ranking?
Is it possible to create a report with containing keyword, label, difficulty, global search volume, and ranking? Currently in order to get the data, it seems like I need to manage two lists, the keyword list we are tracking and the keyword list in the Difficulty tool then somehow manually combine the data. Is there an easier way?
Moz Bar | | promfgsystems2 -
Moz crawler finding my homepage multiple times
Hi and thank you in advance for your help! I have a Moz Pro campaign running (I am a complete Moz novice by the way) for one of my websites (balloonsutah.com). After crawling my site, the Moz crawler informed me that I have 3 pages with duplicate content. While I am not sure why exactly this is happening, the crawler indexed my homepage 3 times under different url's. -balloonsutah.com
Moz Bar | | Keenan-Price
-balloonsutah.com/
-balloonsutah.com/index.html I checked my FTP server and I cannot figure out for the life of me why the crawler is finding anything other than the index.html file. I suppose I need to do something regarding a rel="Canonical" but I am not terribly familiar with that either. Any suggestions would be greatly appreciated!
Keenan0 -
Why is the exact same URL being seen as duplicate and showing an error in my SEO reports
Well, I am still having duplicate page issues. I have a question about one of the errors SEO is giving me when I download a crawl report. I am going to attach a screen shot of part of the report so you can see for yourself, along with explaining it here. SEO shows the list of URL's that it crawled in the report. In this(see attachment) portion of the report it has 321 results for the exact same URL. It also says all of these exact same URL's have received a 404 error. What I want to know is how does it make 321 results for the same URL? And with this error that I don't see when I look at the page? 0hkRDST
Moz Bar | | JoshMaxAmps0 -
I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
The website is www.bigbluem.com and is a wordpress site. I'm getting the following error: 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag But what is weird is the domain it lists below that is http://None/BigBlueM.com Any advice?
Moz Bar | | TumbleweedPDX1