Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to block my web from SeoMOz Crawler

Technical SEO

1132

Locked

Gomu2 last edited by

Hi,

I want to block mysite.com/forum from SeoMoz Crawler, how do i do that? i just want to block the forum and let the other part of my site still crawlable by seomoz. Thanks

Regards
1 Reply Last reply
Reply Quote 0
TomRayner last edited by

Howdy

You'll need to block the SEOMoz crawler in your robots.txt file. Edit the file and insert this bit of code:

User-Agent: rogerbot Allow: /
Disallow: /forum/

If that doesn't work, you might want to add a Wildcard too, making it:

Disallow: *****/forum/

If you're not too sure about editing the robots.txt file, then it might be an idea to check the SEOMoz guide: http://www.seomoz.org/learn-seo/robotstxt - if you're really unsure, it's best to get someone else to do it for you, as you don't want to block all of your site to all of the crawlers

Hope this helps
1 Reply Last reply
Reply Quote 2

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

3,511 Pages Indexed and 3,331 Pages Blocked by Robots

Morning, So I checked our site's index status on WMT, and I'm being told that Google is indexing 3,511 pages and the robots are blocking 3,331. This seems slightly odd as we're only disallowing 24 pages on the robots.txt file. In light of this, I have the following queries: Do these figures mean that Google is indexing 3,511 pages and blocking 3,331 other pages? Or does it mean that it's blocking 3,331 pages of the 3,511 indexed? As there are only 24 URLs being disallowed on robots.text, why are 3,331 pages being blocked? Will these be variations of the URLs we've submitted? Currently, we don't have a sitemap. I know, I know, it's pretty unforgivable but the old one didn't really work and the developers are working on the new one. Once submitted, will this help? I think I know the answer to this, but is there any way to ascertain which pages are being blocked? Thanks in advance! Lewis
Technical SEO | | PeaSoupDigital

0
Blocking subdomains without blocking sites...

So let's say I am working for bloggingplatform.com, and people can create free sites through my tools and those sites show up as myblog.bloggingplatform.com. However that site can also be accessed from myblog.com. Is there a way, separate from editing the myblog.com site code or files, for me to tell google to stop indexing myblog.bloggingplatform.com while still letting them index myblog.com without inserting any code into the page load? This is a simplification of a problem I am running across. Basically, Google is associating subdomains to my domain that it shouldn't even index, and it is adversely affecting my main domain. Other than contacting the offending sub-domain holders (which we do), I am looking for a way to stop Google from indexing those domains at all (they are used for technical purposes, and not for users to find the sites). Thoughts?
Technical SEO | | SL_SEM

1
Suddenly Many 301 Redirects captured by SEOMOZ

On April the 7th SeoMOZ captured 6000 301 redirect on my site, but I cant seem to understand how SEOMOZ finds these links Example http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Makes a 301 Redirect to the following page beneath SEOMOZ says http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html The weird thing is that both urls work, but if i browse my site in a normal matter this link will never be created i that way. The -4a in the end os the link is not the normal link structure on the site and has never been like that before. So how does SEOMOZ Create that link? http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Also google only has the right one that are this one beneath http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html People would normal come to the category with this url http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/ And page 2 would be http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html AND NOT http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Can anyone find out what is going on?
Technical SEO | | noerdar

0
Blocked URL's by robots.txt

In Google Webmaster Tools shows me 10,936 Blocked URL's by robots.txt and it is very strange when you go to the "Index Status" section where shows that since April 2012 robots.txt blocked many URL's. You can see more precise on the image attached (chart WMT) I can not explain why I have blocked URL's ? because I have nothing in robots.txt.
My robots.txt is like this: User-agent: * I thought I was penalized by Penguin in April 2012 because constantly i'am losing visitors now reaching over 40%. It may be a different penalty? Any help is welcome because i'm already so saturated. Mera robotstxt.jpg
Technical SEO | | meralucian37

0
I was googling the word "best web hosting" and i notice the 1st and 3rd result were results with google plus. Does Google plus now play a role in improving ranking for the website?

I was googling the word "best web hosting" and i notice the 1st and 3rd result were results with google plus. Does Google plus now play a role in improving ranking for the website?I see a person's name next to the website too
Technical SEO | | mainguy

0
Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?

Hello, I see hundreds of notices for blocked by meta-robots/meta robots nofollow and it appears it is linked to the comments on my site which I assume I would not want to be crawled. Is this the case and these notices are actually a positive thing? Please advise how to clear them up if these notices can be potentially harmful for my SEO. Thanks, Talia
Technical SEO | | M80Marketing

0
Web config redirects not working where a trailing slash is involved

I'm having real trouble with getting working redirects in place to use on a site we're re-launching with a modified url structure. Old URL: http://www.example.com/example_folder/ New URL: http://www.example.com/example-of-new-folder/ Now, where the old URL's have a trailing slash the web.config simply will not accept it. It says the URL can start with a slash, but not end with a slash. However, many of my URL's do end with a slash so I need a workaround. These are the rules I'm putting in place: <location path="example_folder/"></location> Thanks
Technical SEO | | AndrewAkesson

0
How to create GA Profile for SEOMoz

Hi ! I get this : Oh No! We did not find any Google Analytics profiles associated with this account. So I was wondering how to create GA Profiles for SEOMOZ. I'm admin of 3 websites in my GA accounts ... Thanks !
Technical SEO | | ovivo

0