Roger Bot
-
Hi Mozzers,
I have a dev site that I want to run your crawl text on (Roger Bot) but I want to ensure the other engines don't crawl it.
What is the Robots.txt line that I need to make sure only Roger bot can get in and not Google etc?
Please advise
Thanks
Gareth
-
HI Gareth, Your robots.txt should look like this; User-agent: * Disallow: / User-agent: rogerbot Allow: /
-
User-agent: *
Disallow: /
User-agent: rogerbot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved The Moz.com bot is overloading my server
0 -
GOOGLE ANALYTIC SKEWED DATA BECAUSE OF GHOST REFERRAL SPAM ND CRAWL BOTS
Hi Guys, We are having some major problems with our Google Analytics and MOz account. Due to the large number of ghost/referral spam and crawler bots we have added some heavy filtering to GA. This seems to be working protecting the data from all these problems but also filtering out much needed data that is not coming through. In example, we used to get a hundred visitors a day at the least and now we are down to under ten. ANYBODY PLEASE HELP. HAVE READ THROUGH MANY ARTICLES WITH NO FIND TO PERMANENT SOLID SOLUTION (even willing to go with paid service instead of GA) Thank You so Much, S.M.
Moz Pro | | KristyKK0 -
What Are Roger's Super Powers?
I ended up with a box of SEOMOZ swag. (Thanks! As to how this came to pass...I shall draw a veil, as they used to say in Victorian novels.) My upstairs neighbour Max, age 5, enjoys a rich fantasy life and is very much into superheroes and costumes. Naturally, he ended up with a lot of the Roger stickers. Alas, I was unable to answer all of Max's questions. When he asked: "What does Roger do?" I replied: "Roger makes your computer work." Pretty good, I thought. But then Max asked: "What does the antenae do?" I was kind of stumped. Then it got worse. Max asked what Roger's superpowers are and if he could beat Spiderman. I tried to change the subject. Max wasn't impressed. What are the answers? Enquiring five year old minds want to know!
Moz Pro | | DanielFreedman6 -
Will SEOMoz offer URL data relating to Bot visits
Does SEOMoz in the future plan to report on Bot visits for each URL, when they are spidered and when they appear in for example Google's index ?
Moz Pro | | NeilTompkins0 -
Hug From Roger
I have long since had enough Moz Points to get my hug and STILL have nothing. WHERE'S MY HUG!!!??
Moz Pro | | TheGrid1 -
SEOmoz bot and "noindex"
As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x40 -
Help with Roger finding phantom links
It Monday and Roger has done another crawl and now I have a couple of issues: I have two pages showing 404->302 or 500 because these links do not exist. I have to fix the 500 but the 404 is trapped correctly. http://www.oznappies.com/nappies.faq & http://www.oznappies.com/store/value-packs/\ The issue is when I do a site scan there is no anchor text that contains these links. So, what I would like to find out is where is Roger finding them. I cannot see any where in the Crawl Report that tells me where the origin of these links is. I also created a blog on Tumblr and now every tag and rss feed entry is producing a duplicate content error in the crawl stats. I cannot see anywhere in Tumblr to fix this issue. Any Ideas?
Moz Pro | | oznappies0 -
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev840