What is the IP address of the MOZ crawlbot?
-
Our preproduction site is restricted to a specific IP range.
Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
-
Hi,
Moz is hosted on Amazon. See this sample log entry from today:
54.146.235.41 - - [17/Sep/2017:09:48:13 +0200] "GET /best-deals.html/page/12/ HTTP/1.0" 200 15803 "-" "rogerbot/1.2 (https://mza.bundledseo.com/help/guides/moz-procedures/what-is-rogerbot, [email protected])"[XF 54.146.235.41]
That IP is part of Amazon's ASN.
whois -h whois.radb.net -- '-i origin AS16509' | grep ^route
You can either use something like mod_authz to allow the domain amazonaws.com to connect or you can get each IP from AS16509 and allow that.
Hope it helps.
Regards
Chris
-
Just to expand on this, I found the following quote from a Moz member of staff:
"I wish I could give you IP addresses, but they change all the time since we host Roger in the cloud. There's not even a reliable range of IPs to give you. You can totally whitelist the useragent rogerbot, but that's the only reliable information about the crawler you can go off of. I hope that helps but let me know if there's any other solution you can think of. Thank you! "
From a previous question: https://mza.bundledseo.com/community/q/is-there-a-whitelist-of-the-rogerbot-ip-addresses
-
Unfortunately this is all I can find: https://mza.bundledseo.com/help/guides/moz-procedures/what-is-rogerbot with no IP range shown.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword difficulty with Moz Pro
Hello, I want to know how moz pro calculates the keyword difficulty? What are the basic things which moz pro consider while calculating keyword difficulty? Whenever i check my site(Sehhaty app) it shows different keyword difficulty. why so? Thanks.
Getting Started | | emmaava60 -
I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
Getting Started | | blueprintatl0 -
Why is Moz unable to crawl my site?
Was hoping someone could advise why Moz is unable to crawl my site at https://www.oceaniacruises.com **Moz was unable to crawl your site on Oct 5, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. Any help would be appreciated. Thanks!
Getting Started | | jbarinaga0 -
Moz not recognising url to setup a new account
I am attempting to setup a new account for my client but Moz is having a problem and not registering the url http://www.esc.uk.net/. I get the "oops" message & read the info on the FAQ page it refers to. Why do I keep getting an "Oops" message when I type and submit my campaign URL? To create a campaign, we need to receive a valid HTTP response from the web server hosting your site. While the site may be accessible from a standard web browser, other user-agents and search engine bots may receive a different response. If you receive this error, we recommend running a test by sending an HTTP response using any online HTTP tool. Using an online HTTP tool I believe I received an HTTP response. What else can I do? Regards Terry
Getting Started | | Buzzin0 -
How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages. Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site. Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
Getting Started | | longclimber0 -
Google + Business Page and Moz Pro
Is there a way to connect my Google + Business page with Moz? The only option I have available to connect when I press the button is my personal Google + Page. Thanks!
Getting Started | | BenhardtConstruction0 -
New to using MOZ. Familiar with Google Analytics. With MOZ is there a code snippet to include on my site?
Just taken over web marketing responsibilities at my company. Will be doing some major website upgrades soon. I'm not familiar with MOZ and don't want to overwrite anything. So when setting up MOZ, is there a code snippet that goes anyplace on the site like there is with Google analytics? Thanks.
Getting Started | | NanoLumens0 -
Moz profile
Hi, For some reason when I try and add my site's link in my Moz profile I keep getting an error. My site an SSL so I used https instead of http. I tried again without the 's' as my domain redirects to the https version, but this didn't work. Then I tried without http or https and this didn't work either. Why does Moz hate my site? It's actually really good!!! Thanks, Amelia
Getting Started | | CommT0