How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using h2 for category on ecommerce website
Hi, I am working on an ecommerce site and the main category - lets call them car widgets - is using a h1 at the top of the page which is great. There are 4 sub categories on the page - lets call one of them red widget. The only content on the page is the sub category name and an image. Should the sub category red widget use a h2? Thanks S
On-Page Optimization | | bedynamic0 -
Is my plan the right plan? - What do you suggest/do?
When trying to rank a page high for a certain keyword, I usually go by the following plan. 1. Focus on on-Page SEO. Title tags, webpage URL, unique quality content. In other words, make the page the best you can for the user using the keyword you want to rank for. 2. Promote the page via social media. 3. Try to link build to the page. Contact high-quality sites and have them try to link to your page. I was wondering if I am missing any steps, or do you use other methods when trying to rank for a certain keyword?
On-Page Optimization | | trumpfinc0 -
How to exclude URL filter searches in robots.txt
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505 How can I exclude all of these filters in the robots.txt? I think it'll be: Disallow: /*?color=$ Is that the correct syntax with the $ sign in it? Thanks!
On-Page Optimization | | neenor0 -
Lost Page Rank after directing http:// to WWW?
Hi I am trying to redirect all the non www urls to WWW. After I redirected them, most of my category page PR are dropped to 0. Can someone please tell me if this is the normal after effect after the redirect? Example url: this is PR2 before the redirect http://www.ilovebodykits.com/category/95/Body_Kits_Front_Bumpers.html
On-Page Optimization | | ilovebodykits0 -
The crawl diagnosis indicated that my domain www.mydomain.com is duplicate with www.mydomain.com/index.php. How can I correct this issue?
How can I fix this issue when crawl diagnosis indicated that my www.mydomain.com is duplicate with www.mydomain.com/index.php? That suppose to be the same page and not duplicate, right?
On-Page Optimization | | jsevilla0 -
How to make FB comments crawlable by Google? <noscript>?</noscript>
We get tons of FB comments, but it's all in iframe, so Google doesn't give us any credit for it. We found a solution - turn all the comments into HTML and hide it from readers with <noscript>. </p> <p> </p> <p>Will this help? I heard that Google considers <noscript> a scammy practice. Is that true?</p> <p>How do you guys make your FB comments SEO friendly?</p> <p> </p></noscript>
On-Page Optimization | | Alexey_mindvalley0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0 -
Why does SEOmoz use /blog/content-title vs /category/content-title? Any difference?
Assume a brand new blog being designed and all other things equal. What are the pros & cons between using the url structure /blog/content-title vs. /category/content-title? Note:
On-Page Optimization | | JasonJackson
Both scenarios would be using categorical archiving.0