Best blocking solution for Google
-
Posting this for Dave SottimanoI Here's the scenario: You've got a set of URLs indexed by Google, and you want them out quickly Once you've managed to remove them, you want to block Googlebot from crawling them again - for whatever reason. Below is a sample of the URLs you want blocked, but you only want to block /beerbottles/ and anything past it: www.example.com/beers/brandofbeer/beerbottles/1 www.example.com/beers/brandofbeer/beerbottles/2 www.example.com/beers/brandofbeer/beerbottles/3 etc.. To remove the pages from the index should you?: Add the Meta=noindex,follow tag to each URL you want de-indexed Use GWT to help remove the pages Wait for Google to crawl again If that's successful, to block Googlebot from crawling again - should you?: Add this line to Robots.txt: DISALLOW */beerbottles/ Or add this line: DISALLOW: /beerbottles/ "To add the * or not to add the *, that is the question" Thanks! Dave
-
Following up here -- did this answer Dave's question?
-
I would put noindex,follow on those page and wait a little until they disappear for Google index. Of course, if you have only a few pages, I would do it manually in GWT. If you have rather big site with a good crawl rate, this should be done in a few days.
When you don't see them anymore, you may use DISALLOW */beerbottles/ but this could be annoying later. I would recommend to use the meta robots as you have more control on it. It will allow page rank to flow in the beerbottles pages too !
-
I believe you can confirm the block via the webmaster tools also.
-
Hi Goodnewscowboy,
To block the whole folder you dont need to use the wild card (*)
and I advise you to also do these steps:
- Verify your ownership of the site in Webmaster Tools.
- On the Webmaster Tools home page, click the site you want.
- On the Dashboard, click Site configuration in the left-hand navigation.
- Click Crawler access, and then click Remove URL.
- Click New removal request.
- Type the URL of the page you want removed, and then click Continue. Note that the URL is case-sensitive—you will need to submit the URL using exactly the same characters and the same capitalization that the site uses.
- Select Remove page from cache only.
- Select the checkbox to confirm that you have completed the requirements listed in this article, and then clickSubmit Request.
Cheers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Deindexed homepage by Google
I just noticed that my homepage was de-indexed by google. Any thoughts would be appreciated.
Technical SEO | | Jenny_H0 -
Google News problem
Hello to all. The latest Google algorithm changes have had a big impact on the way that Google news features stories, at least in my country. I've been featured heavily in Google News until about 6th of october, when the changes had the biggest impact, but since then, I haven't been featured at all. Prior to this, I would be featured for keywords on almost any article, not necessarily on the 1st position, but I was almost always there. Posts still show up in the dedicated News category, but not in the main search pages. I've seen a lot of websites being impacted, but some with lower ranks than mine still show up there. I haven't done any changes prior to the 6th of october, and I haven't done any link building campaings, just getting links from higher ranking news sites in my country, for articles I wrote. What I'd like to know is if there were any major changes for Google News and I'm not complying with any of them, or If i could check to see if there are any other problems. I don't have any penalties disclosed by Google, and no new errors in the Webmasters console, I'm just baffled by the fact that overnight the website was completely cut off from being featured in Google News. And one other strange thing, I'm now ranking better for searches that are kind of opposite to my website's main theme. Think about mainly writing about BMW, and less about AUDI, but ranking a lot better for the latter, and a lot less for the other. Thank you.
Technical SEO | | thefrost0 -
Google Sitelinks
Hello, Good afternoon. I am having a site issue with Sitelinks. For some reason when I search Google for the brand I represent "California Olive Ranch" Sitelinks are not being generated. When I search for "Cal Olive Ranch" our site links are being generated. Our domain is Californiaoliveranch.com. Is there a way to tell Google to to change the site links to match our domain and brand name? Is this something that can be done in Google Webmasters? Thank you very much for your help. Adam P
Technical SEO | | apost40 -
Google is changing the title
Hi! Lately i have seen that Google changed the page titles for some clients, not all... its about 30% of them. For example, the title looks likes this after the Google change: Company name: SEO and Pay per click management But in on that page it looks like this: SEO and Pay per click management - Company name Does anyone know why?
Technical SEO | | DanielNordahl1 -
How to correct a google canonical issue?
So when I initially launched my website I had an issue where I didn't properly set my canonical tags and all my pages got crawled. Now in looking at the search engine results I see a number of the pages that were meant to be canonical tagged to the correct page showing up in the results. What is the best way to correct this issue with google? Also I noticed that while initially I was ranking well for the main pages, now those results have disappeared entirely and deeper in the rankings I am finding the pages that were meant to be canonical tagged. Please Help.
Technical SEO | | jackaveli0 -
Google Places - What is the best Service Areas Strategy?
I've found a lot of useful info on this topic in these forums, but still can't seem to find the answer to my specific question. Client has one physical location and services many areas. I have seen various comments that claim setting a service area actually has a negative effect on rankings and the login makes sense to me, so we don't want to do that. Using the actual physical address, seems to be what google would prefer, but the address is actually on the outskirts of the city and would mean that competitors that have addresses closer to the city center would show up before us. Our current places listing has the actual address, but the previous SEO put the larger city, with the smaller city zip on the on the website. City Center: San Diego, 92101 Actual: Street Address, El Cajon, 92020 On website: San Diego, 92020 It this large City + Actual zip code strategy any good? Which of these 3 strategies should we use to standardize all of our listings? *we will not be considering a location or mailbox per service are to use multiple listings at this time
Technical SEO | | vernonmack0 -
When is the best time to submit a sitemap?
What changes to a website constitute resubmitting a sitemap? For example, if I add new in-site links, should I then resubmit? Or is it more for changes to URLs, Page titles, etc?
Technical SEO | | MichaelWeisbaum0 -
How to disallow google and roger?
Hey Guys and girls, i have a question, i want to disallow all robots from accessing a certain root link: Get rid of bots User-agent: * Disallow: /index.php?_a=login&redir=/index.php?_a=tellafriend%26productId=* Will this make the bots not to access any web link that has the prefix you see before the asterisk? And at least google and roger will get away by reading "user-agent: *"? I know this isn't the standard proceedure but if it works for google and seomoz bot we are good.
Technical SEO | | iFix0