SEO Web Crawler IP addresses
-
What are the IP addresses for the SEO Web Crawler?
There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses.
Thank you for your time
-
Hi Jason,
This is Megan from the SEOmoz Help Team. Unfortunately, since we crawl from the cloud, RogerBot's IP address is changing all of the time so we cannot specify a particular IP address. Also, we do need the site to be live before we can crawl it, but you could remove the firewall and block all other bots except rogerbot so it won't get indexed anywhere else before it's ready.
I hope this helps! Please let me know if you have any other questions.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help with best seo strategy
Hello all, I am a little bit confused about the "content suggestion'' feature in Moz. If I understood well, this feature suggests Anchor texts we can use in other page to link to the page we are trying to Optimize. So far, I've been using page keyword as Anchor Text in other articles. For example, I have a page about the best places to visit in Morocco, where the keyword is 'places to visit in Morocco'. I am using this same keyword as Anchor text in other pages to link to that page. While Moz suggests '10 best places' as Anchor text. What does it mean ? Is '10 best places' anchor text better ? Or should I use both ? Thanks in advance for your enlightenment.
Moz Pro | | sportygirl0 -
I'm searching for a Freelancer SEO Professional
Hi guys, I'm searching for a freelancer seo professional to fix my website errors on SEO Moz, I'm from Brazil, I got a supplement webstore and have no time to fix this errors here in SEO Moz Someone here for this job? Thanks
Moz Pro | | megaformasuplementos0 -
Crawlers crawl weird long urls
I did a crawl start for the first time and i get many errors, but the weird fact is that the crawler tracks duplicate long, not existing urls. For example (to be clear): there is a page: www.website.com/dogs/dog.html but then it is continuing crawling:
Moz Pro | | r.nijkamp
www.website.com/dogs/dog.html
www.website.com/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dogs/dog.html
www.website.com/dogs/dogs/dogs/dogs/dogs/dog.html what can I do about this? Screaming Frog gave me the same issue, so I know it's something with my website0 -
Can I see when SEO Moz has crawled my website?
I would like to know if it's possible to see (maybe in my Google Analytics) if SEO Moz has crawled my website. I'm also curious if and where I can see when the robot of Google visited my website. Thanks!
Moz Pro | | Spotler0 -
2 SEO questions
Hello everyone 🙂 I have two questions for all you fine seo people this morning... 1 - Is there a website that I could go to to make sure that I do not have duplicate content floating around on the web that I am not aware of? Sometimes people take information from my site and post it as their own and I want to make sure that google does not ping me for it. 2 - Does anyone know how I can report a spam site to google? I have filled out the reports many times over the past year and posted it in the webmaster discussion forum and it is still up there 🙂 I sent one email to bing and the next day they contacted me with a thank you email and indicated that I was 100% correct and they removed the site from their index. Thank you all!!!!
Moz Pro | | nazmiyal0 -
SEO Moz Tools - too many on the page links result driving me nuts
A while back I remember Rand and I having a conversation about how many links on the page and up until that point I had followed the NO MORE THAN 100 links on a page rule - which is what the MOZ tools are telling me now in the campaigns I have running. But then during a seminar both of us were holding this 100 link rule question came up and Rand commented that this was probably old hat now as the search engines can crawl a much greater number of links in the page. I was encouraged by his answer especially where ecommerce websites are concerned. But the MOZ tool is driving me nuts telling me that this 100 link rule is still something to be adhered too. It is especially frustrating when we are discussing ecommerce website sites with editable mega menus. Examples to support this question are www.bohemiadesign.co.uk or www.flowersbuydelivery.co.uk which are 2 ecommerce sites I am aware of using such mega menus that are editable and give a link count greater than 100. and I am sure there are many more sites like this, even Amazon for example. So, how much notice do we take of this warning in MOZ tools that is telling me about excessive numbers of links on the pages it lists as needing corrected?
Moz Pro | | ICTADVIS0 -
Top ranking with no SEO
I ran an SEOMoz keyword difficulty report for the term "interim CFO services" and found that all of the websites that ranked really well for that term have good page authority and domain authority scores and plenty of incoming links.....except 1. result #5 has a page and domain authority of 1 and no incoming links, however it's ranked #5 in google SERP for this highly competitive term. see screenshots: http://screencast.com/t/NJiOn8N4l http://screencast.com/t/x6tU1RhX4j Is this ranking based solely on the domain name being 100% relevant? Did they just get lucky and hit the google rankings jackpot? If this is the case, by the same logic I am trying achieve high rankings for the keyword "outsourced accounting firm" and purchased the domain name outsourcedaccountingfirm.com and am not even ranked in the top 50 with google for that keyword. (it's very new though) any feedback on either questions above would be greatly appreciated. Thanks!
Moz Pro | | texmeix0 -
SEO Targeting: Ideal monthly keyword search volume?
What's your SEO sweet spot for monthly keyword search volume*? At what volume do you actually expect to see traffic? (I know this depends heavily on competition for said keywords). (*Currently measuring volume via Google AdWords Keyword Tool / SEOmoz keyword difficulty tool. || Comparing exact match and broad match.)
Moz Pro | | JessicaCox0