How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Schema Markup for eCommerce Category Pages?
My research indicates that applying an ItemList schema markup to our category pages is likely the best way to go. However, I've also ready that Google discourages schema markup on category pages. I'm just wondering if any of you have applied schema markup to your category pages and, if so, how did you do it? John,
On-Page Optimization | | JohnBrown75
Essay Writer1 -
No-index all the posts of a category
Hi everyone! I would like no-indexing all the posts of a specific category of my wordpress site. The problem is that the structure of my URL is composed without /category/: www.site-name.ext/date/post-name/
On-Page Optimization | | salvyy
so without /category-name/ Is possibile to disallow the indexing of all the posts of the category via robots.txt? Using Yoast Plugin I can put the no-index for each post, but I would like to put the no-index (or disallow/) a time for all the post of the category. Thanks in advance for your help and sorry for my english. Mike0 -
Duplicate Content - Category Pages 2+
I have my Wordpress SEO settings to deindex past page 1 of each category. However, Google Webmasters is selling me I have 210 pages with duplicate title tags. My site tanked last weekend and I don't know if it was Google Panda or what. I have been getting some fantastic backlinks and it seems like they just decided to disregard all of them as I am completely off the SERPs. Is this duplicate content a contributing factor? How can I get google to deindex my category pages past page 1? (I do need the first page to index as that does bring me organic traffic) Thanks.
On-Page Optimization | | 2bloggers0 -
Do you use <nofollow>and rel=nofollow?</nofollow>
I just read http://www.thoughtmechanics.com/does-nofollow-attribute-work-google-says-yes-studies-say-otherwise/ . Is it really better to avoid using nofollow for local links (from one site to itself)?
On-Page Optimization | | fleetway0 -
Directory Structure
Hi, We are creating a new content directory for online courses hosted on our site. Like a typical directory, we have high level categories and then more granular subcategories. A course will typically only be in one high level category and then multiple subcategories. What would be the best URL structure for an individual course? Should we force users to pick one 'master' subcategory that gets included in their URL? Or should we just not include the subcategory at all in the URL? Right now we've been thinking about: OurUrl.com/upper-category/sub-category/course-title or OurUrl.com/upper-category/course-title
On-Page Optimization | | mindflash0 -
Installing a site on top level domain directory VS deeper directory
How important it is for seo to install a site on a top level directory vs deeper directory? example: www.mysite.com VS www.mysite.com/catalog many eCommerce scripts such as oscommerce and cre loaded will install by default to /catalog and that's what I've been doing for most of my customer. does it mean it will be harder for them to get good seo results? Thanks in advance for the input...
On-Page Optimization | | zigi0 -
Product merchandising category creation
I work on a site selling clothing, and we break up our clothing into categories of types of clothing. Right now in my accessories, I have an other category, which cover the miscellaneous items that there aren't enough to warrant their own categories. I was curious what people thought about further breakdowns of this category. Do I create a category that only has one item in it? Is there a certain threshold of number of items which should signal that these items need their own categories or length of time they're expected to be available to buy? Right now, I'm not targeting the one-off items for SEO purposes because we tend not to carry them from season to season or continue with them long enough to be the best use of my SEO time.
On-Page Optimization | | kennyrowe0 -
Photogallery and Robots.txt
Hey everyone SEOMOZ is telling us that there are to many onpage links on the following page: http://www.surfcampinportugal.com/photos-of-the-camp/ Should we stop it from being indexed via Robots.txt? best regards and thanks in advance... Simon
On-Page Optimization | | Rapturecamps0