How to get rogerbot whitelisted for application firewalls.
-
we have recently installed an application firewall that is blocking rogerbot from crawling our site. Our IT department has asked for an IP address or range of IP addresses to add to the acceptable crawlers. If rogerbot has a dynamic IP address how to we get him added to our whitelist? The product IT is using is from F5 called Application Security Manager.
-
Hi Joel,
Just wondering if you guys came up with a fix for this one? I have the same issue myself...
Thanks!
-
We have a similar issue where we block the Amazon cloudspace but want to allow rogerbot. Our firewall finds it much easier to block by IP range rather than user-agent (rogerbot).
Can you please tell us more about the solution you suggested?
-
Hi, our IP is changing all the time. We do have an option that may resolve your issue though. I'm going to create a ticket as I will need some personal information regarding your account.
Thanks,
Joel
-
Hi kgoss -
I'm not an SEOmoz employee - but just fyi, here are the IP addresses rogerbot used when accessing my site since the start of 2012:
50.17.63.219
184.72.211.58
50.19.77.29
174.129.177.94
184.73.93.243
107.20.64.186
23.22.26.208
107.22.19.89
50.19.188.211
50.19.195.34
107.22.107.114
107.22.137.134
216.244.72.3
75.101.205.127
67.202.42.32
216.176.191.232
23.20.247.207
216.244.72.9
23.22.48.203
216.176.191.234
184.73.13.203
107.20.11.69
50.17.96.67
184.73.137.220
184.73.22.42
184.72.175.148
67.202.13.192
23.21.33.19
107.22.0.173
107.22.157.76
50.16.164.143
50.16.43.91
107.21.143.222
216.176.191.201
107.20.91.74
50.16.88.111
204.236.213.120
184.73.118.141
50.17.38.180
107.21.150.218
216.244.72.10
216.244.72.11
216.244.72.12
23.22.53.56Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting my top keywords separated out in SEOmoz reports
I am using the standard functionality to produce weekly Moz reports. There does not seem to be a setting to show rankings of my most important keywords. It would be nice to have those high-level keywords on the first page. For example, I have 200 keywords in an account. I want the report to show on a page my 10 most important keywords. Is there a way to set up a Label to my keywords in order to product a report page just for those keywords?
Moz Pro | | clicktoshop0 -
What should be cols value if I want to get Backlinks?
Hi, I am forming below url to get backlinks. http://lsapi.seomoz.com/linkscape/url-metrics/".$trimurl."?Cols=2048 &AccessID=".$accessID." &Expires=".$expires." &Signature=".$urlSafeSignature; For Example, if I keep $trimurl = "www.tatvic.com/" , I get [uid] => 633. Is this a right way to get number of backlinks ? If not, what should be the 'Cols' value? Also, how can I ensure that the number of links I am getting is correct ? Is there any way to compare this number with Google search results? This is very essential to check as I got different number of backlinks on different APIs. Thank you.
Moz Pro | | Ravi_Pathak0 -
Rogerbot not showing in logs
Hi All Rogerbot has recently thrown up 403 errors for all our pages - no changes had been made to the site so I asked our ISP for assistance. They wanted to have a look at what rogerbot was doing and so went to the logs but rogerbot was not listed anywhere in the logs by name - any ideas why? Regards Craig
Moz Pro | | CraigWiltshire0 -
Getting your site totally indexed by SEOMOZ
Hi guys! Ijust started using SEOMOZ software and wondered how it could be that my site has over 10.000 pages but in the Pro Dashboard it only indexed about 1500 of them. I've been waiting a few weeks now but the number has been stable ever since. Is there a way to get the whole site indexed by SEOMoz software? Thanks for your answers!
Moz Pro | | ssiebn70 -
Is the Linkscape Index getting updated?
I know it only gets a big refresh once a month...The last being in the early part of January. But I released a map that went viral (front page of CNET, Wired, Scientific American) that has gotten me hundreds of links from different domains: https://www.google.com/search?q=perceptionbuilder.com+map I got these links in early to mid December and OSE is showing no data at all. Similarly, I have been doing some linkbuilding for clients and links that I got in October and November and those links aren't showing up either. Anyone else experiencing this? Many thanks, Matt
Moz Pro | | coppersix0 -
Rogerbot Ignoring Robots.txt?
Hi guys, We're trying to block Rogerbot from spending 8000-9000 of our 10000 pages per week for our site crawl on our zillions of PhotoGallery.asp pages. Unfortunately our e-commerce CMS isn't tremendously flexible so the only way we believe we can block rogerbot is in our robots.txt file. Rogerbot keeps crawling all these PhotoGallery.asp pages so it's making our crawl diagnostics really useless. I've contacted the SEOMoz support staff and they claim the problem is on our side. This is the robots.txt we are using: User-agent: rogerbot Disallow:/PhotoGallery.asp Disallow:/pindex.asp Disallow:/help.asp Disallow:/kb.asp Disallow:/ReviewNew.asp User-agent: * Disallow:/cgi-bin/ Disallow:/myaccount.asp Disallow:/WishList.asp Disallow:/CFreeDiamondSearch.asp Disallow:/DiamondDetails.asp Disallow:/ShoppingCart.asp Disallow:/one-page-checkout.asp Sitemap: http://store.jrdunn.com/sitemap.xml For some reason the Wysiwyg edit is entering extra spaces but those are all single spaced. Any suggestions? The only other thing I thought of to try is to something like "Disallow:/PhotoGallery.asp*" with a wildcard.
Moz Pro | | kellydallen0 -
How to get rid of the message "Search Engine blocked by robots.txt"
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest. Regards, Prashakth Kamath
Moz Pro | | 1prashakth0 -
Anyone know how/where to get keyword yield ratio?
I'm trying to incorporate keyword yield ratio (the ratio of keywords to pages yielding search traffic) in my seo report. I want to be able to compare on a monthly basis the percentage ratio of keywords to a landing page yielding search traffic. So for example, in July we might have 5 keywords that earned a click to landing page A. But in September, the number of keywords grew to 8 keywords that earned a click to landing A. I would like to be able to report monthly on the % ratio growth of number of keywords to landing pages that yield search traffic. Aim is to grow # of keywords per page... Is there a way I can get this data through seomoz tools or google analytics? Thanks, Gemma
Moz Pro | | RBA0