How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of using spaces or no spaces on product category page varient keywords
Hello, all fellow Mozzers,
On-Page Optimization | | JamesDavison
I have taken over a project and this account, so can't change the username according to MOZ.🙃 We run an eCommerce website, and to me, some of the content is conflicting as some pages have more information content than what I would put in a commerce page, but this is how the boss wants it to work, personally, I would separate the content out.
The page I'm working on:
https://www.longstonetyres.co.uk/tyres/205-70-14.html
and this is an example of the rest of these types of pages, I will be tackling:
https://www.longstonetyres.co.uk/tyres/125-15.html I was tasked to improve SEO ranking, when using the MOZ page grader I had a score of 24 out of 27 83% SEO score and 3-page problems. 7th position in Google for the search term 205/70 R14 As it is a generic product listing page, It was pointless to add to the URL and the Internal links I can't reduce as these are links to products, so I went to reduce the
keyword stuffing and making the page content more natural, this improved the page to 25 out of 27, 87% SEO score and 2-page problems. Improvement to 3rd position in Google, but he wants to chase 1st place to be above his competitors, which is fair enough. It turns out that in the past, they have used this type of page to try and get a high ranking for several search terms, as it is a different variation on a tyre size terms are:
205/70 R14, 205/70R14, 205/70 R 14
205/70 X 14, 205/70X14, 205/70 X14
and so on for all the different ways you can search for this tyre size. He is also convinced Google will see these as different search terms, and while I agree to an extent, this causes Keyword Stuffing on the page, which in turn was harming the rankings. Each product listed on the page already has its own title 205/70 R14, 205/70 HR14 and so on, so my question is. What is the best practice for writing content on these types of pages to gain high rankings for several Keywords, and what value does writing the same keyword with spaces and no spaces have? Any help or advice is welcome, so I have a better understanding of how to approach this for this page and the rest of the site. Cheers Mal0 -
Best website IA/structure for SEO?
What's the current thinking on the best structure of information on a website for SEO? Structure for visitors can be best achieved through navigation menus, but I am more interested in how I should organise my URL structure so Google can make sense of the depth of my site topics. The website is an Asian travel blog so there are essentially two specific types of post on the site. One type is location specific (may be about an attraction, a city, a region or a country). The other type is general (usually about an aspect of travel like travel cash, visas, scams, etc). At the moment, all my general posts are organised like www.asiantraveltips.com/blog/[post-name]. My location-specific posts are organised like www.asiantraveltips.com/[country]/[region-or-city]/[place-name]/ so that Google can see I have depth of topics about each country and region. But I find it hard to keep consistency in this arrangement of URLs and I don't know if I might be better off to just have everything flat and tagged as a blog post like www.asiantraveltips.com/blog/[country]-[region-city]-[post-name]/? What's best practice these days? How are others organising travel blog websites?
On-Page Optimization | | Gavin.Atkinson0 -
Duplicate content from pagination and categories found in multiple locations
Hey Moz community, Really need help resolving duplicate content issues for an eCommerce utilizing Magento. We have duplicate content issues with category pagination and categories found in multiple locations. here's an example: "www.website.com/style/sequin-dresses" is also found at "www.website.com/features/sequin-dresses" *to resolve this issue do we just need to place a canonical tag on "www.website.com/features/sequin-dresses" pointing to "www.website.com/style/sequin-dresses"? In addition, the category "Sequin Dresses" also has pagination. to resolve duplicate content issues with pagination do we need to implement a rel=next/prev tag? (we do not have a view-all due to the amount of products featured) If anyone has experience with this or any insights on how to resolve these issues please let me know. Thanks!
On-Page Optimization | | LeapOfBelief0 -
Keyword repeats/presence in url's & over-optimisation
Hi I'm about to launch a redesigned site and worried about overdoing kw presence on-page, primarily using in url's since will already be using kw in titles as well as page content. What's current thinking re over optimisation: If kw is in titles and page content is it best not to repeat again in url structure i.e. less is more, even though this will cause things like SeoMoz on-page grade score to fall, or better to keep them/add them ? Personally i think it makes sense to include kw in url again since helps make the page relevant, and so long as matches the content should help as opposed to hinder rankings for the pages target keyword. However when i look into this some say don't do this since is over-optimisation The sites generally ranking quite well for its target kw which i obviously don't want to lose after re-launch & hopefully improve further, in the case of this example they are 'Sports Centre Services' & 'Sports Centre Equipment Rental'). The sites current url structure is similar to this below example: frankssportscentres.com/services/sports-centre-equipment-rental Would it be better to keep following existing/above format or to go with either of the below options i.e. more kw rich urls or less: frankssportscentres.com/sports-centre-services/sports-centre-equipment-rental Or frankssportscentres.com/sports-centre-services/equipment-rental Or even less frankssportscentres.com/services/equipment-rental Many Thanks in advance for any helpful comments Cheers Dan
On-Page Optimization | | Dan-Lawrence0 -
Wordpress category links not working
Hi All of sudden, my category links are not working. Any tips on figuring out what's causing this? Looks like permalink problem with newer wordpress version. I turned off all the plugins see if this cause any problems. Still not being able to find any option. Here's my site http://www.hibebefetaldoppler.com/fetal-doppler-questions-and-answers/ Thanks in advance
On-Page Optimization | | BistosAmerica0 -
Redirect a blog category page to the homepage?
Hi folks Following Penguin 1.1, I have a client site at number 5 for their primary keyword. (was creeping up page 2 with whitehat link building and tight on-site SEO.) However now the page ranking at No. 5 is for a blog category archive. What do people think the quickest / safest way to get this ranking directed back to the homepage is? Many thanks Simon
On-Page Optimization | | SCL-SEO0 -
Email in Local Search Directory Listings
Hi! I am relatively new to the world of SEO . . . let's say I'm still testing the waters. My role is at the company I work for is Local Search Specialist. I'm claiming/verifying/publishing listings for my employer's clients in attributing search directories. Yada yada yada. You guys are pros. Anyway, I have become apprehensive when listing a client's email in directories besides Google Places. I know that content matching between the client's website, Places, and attributing directories is a must; but spam is the worst. I'm trying to avoid spammers and sketchy directories contact the clients with sales pitches via email as much as possible. Should I create an additional email specifically for these directory listings and keep tabs to see if real people are using it as a means of contact? Not list the email at all? What should I do?!?!?!?!?!?!? Any insight/words of wisdom is appreciated!
On-Page Optimization | | CakeWebsites0 -
Include the company/domain name in page titles and urls?
I know this isn't something that I would use site-wide but I'm wondering if it helps or hurts me to use my company name (also my domain name) in pages below the homepage. As an example, let's say I'm Home Depot. In the category pages off the homepage should I use Page names and urls like Home and Garden Supplies or Home and Garden Supplies at Home Depot? Or does it hurt me to reuse my company/domain name on multiple pages?
On-Page Optimization | | kdieruf0