Should I use meta noindex and robots.txt disallow?
-
Hi, we have an alternate "list view" version of every one of our search results pages
The list view has its own URL, indicated by a URL parameter
I'm concerned about wasting our crawl budget on all these list view pages, which effectively doubles the amount of pages that need crawling
When they were first launched, I had the noindex meta tag be placed on all list view pages, but I'm concerned that they are still being crawled
Should I therefore go ahead and also apply a robots.txt disallow on that parameter to ensure that no crawling occurs? Or, will Googlebot/Bingbot also stop crawling that page over time? I assume that noindex still means "crawl"...
Thanks
-
Hi,
Thanks, I will do some testing to confirm that this behaves how I would like it to
-
if all pages are 100#5 not indexed then I would block it in robots.txt, Google's John Muller confirmed to me that Googlebot will continue to crawl every link to check to see if a nofollow or noindex has changed status.
So as a result we blocked our pages with robots.txt and saw a great increases in index/crawl rates on pages we want Google to pay attention to. It also reduces waste in server resources.
However if there are any pages that are index, if you block them in robots.txt then Googlebot will never be able to crawl the link to determine that it should be noindex. This means it could stay in a permanent stage of indexed.
I hope that answers all your questions?
-
When you say:
nofollow will tell the crawlers to not crawl the page
I believe you mean to say that this will tell the crawlers not to crawl the links on the page, the page itself is itself still "crawled" is it not?
But yes, you are right to say, that once robots.txt disallow is in place, the meta tag will not be seen and thus be moot (at which point I may as well take it off).
It would be nice to be able to say "don't crawl this and don't put it in the index"... but is there a way?
-
noindex only tells the search crawlers to not include the page in the index but still allows for them to crawl the page. nofollow will tell the crawlers to not crawl the page.
robots.txt will accomplish this as well but both I think would be overkill.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How safe is it to use a meta-refresh to hide the referrer?
Hi guys, So I have a review site and I'm affiliated with several partnership programs whose products I advertise on my site. I don't want these affiliate programs to see the source of my traffic (my site), so I'm looking for a safe solution to hide the referrer URL. I have recently added a rel="noreferrer" tag to all my affiliate links, but this method isn't perfect as not all browsers respect that rule. After doing some research and checking my competitors I noticed that some of them use meta-refresh, which seems more reliable in this regard. So, how safe is it to use meta-refresh as means of hiding referrer URL? I'm worrying that implementing a meta-refresh redirect might negatively affect my SEO. Does anybody have any suggestions on how to hide the referrer URL without damaging SEO? Thank you.
Intermediate & Advanced SEO | | Ibis150 -
What to do about new meta description character limit?
Hey Everyone, So as I'm sure everyone has heard the new meta description limits have been increased and I have been going back and forth with whether or not to update my descriptions for my client's websites. I know that Google is now dynamically generating descriptions based off of content, but is it still beneficial to write longer descriptions as well? Will Google display my longer description now if it is ranking well? Rand Fishkin at Moz and others say that you should and other people, including Danny Sullivan, have said you shouldn't worry about it and leave them at between 150-160. My questions is what should I do? I will be focusing heavily on making sure the site's content is very targeted and relevant for when Google dynamically generates the descriptions, but should I still edit my descriptions manually?
Intermediate & Advanced SEO | | kenturley2 -
Using Pagination for eComm Reviews Pages
Hi All, An eComm site has product pages where only 10 customer reviews are found in the source code on the product page, no matter how many reviews the product actually has. ALL reviews (including the 10 displayed on the product page) are located on a subdomain, split into many pages dependong on how many reviews a certain product has (some have well over 100 unique reviews). Reviews page: http://reviews.americanmuscle.com/0065-en_us/charcoalamr-18x8-0512-pirelli-stan/american-muscle-wheels-amr-charcoal-wheel-pirelli-tire-kit-18x8-05-14-all-reviews/reviews.htm Corresponding product page: http://www.americanmuscle.com/charcoalamr-18x8-0512-pirelli-stan.html I'm fearing a Panda related problem here, especially since thousands of products have only 1 or two reviews, duplicated on the reviews.americanmuscle.com page and the corresponding product page. I also do not want to lose the unique content on the second and third reviews pages simply by noindexing/canonicaling them to the product page. My question is whether or not I can paginate the reviews.am pages in a way that the product page is "page 1" and the first reviews page is "page 2," second reviews.am page is "page 3" and so forth. Are there issues associated with domain-to-subdomain pagination? Can I utilize the pagination tab in this manner in the first place? There are currently more than 57,000 of these review.americanmuscle.com pages in the index that I would like to clean up so any/all suggestions are appreciated.
Intermediate & Advanced SEO | | andrewv0 -
Business Uses Third Party for Phone Number
I have a client that uses a third party rental agency to book their cabins. The client has that 3rd party company's number on the site and I have been tasked with building citations. I feel this will create a problem by using another company's number instead of a custom one just for this business. Thoughts on this matter?
Intermediate & Advanced SEO | | Atomicx0 -
Meta Keywords Good or Bad
Hi All, I've been reading more about the meta keyword tag and why it may not be a good idea to include them on pages and am looking for thoughts/feedback on this idea. If you have employed this tactic, can you give me some insight into any results you saw. If you decided to not employ this tactic, why did you choose not to? I wan to understand all sides of this before employing any changes to my company's websites. Thank you for your help!
Intermediate & Advanced SEO | | airnwater0 -
Google showing high volume of URLs blocked by robots.txt in in index-should we be concerned?
if we search site:domain.com vs www.domain.com, We see: 130,000 vs 15,000 results. When reviewing the site:domain.com results, we're finding that the majority of the URLs showing are blocked by robots.txt. They are subdomains that we use as production environments (and contain similar content as the rest of our site). And, we also find the message "In order to show you the most relevant results, we have omitted some entries very similar to the 541 already displayed." SEER Interactive mentions that this is one way to gauge a Panda penalty: http://www.seerinteractive.com/blog/100-panda-recovery-what-we-learned-to-identify-issues-get-your-traffic-back We were hit by Panda some time back--is this an issue we should address? Should we unblock the subdomains and add noindex, follow?
Intermediate & Advanced SEO | | nicole.healthline0 -
What Wordpress Update Services Should You Be Using on Your Wordpress Blog?
I have been told that pingomatic.com is all that you need however yesterday I went to a conference and others were recommending to have a good list of pinging services to cover all your bases Here are 4 that have been recommended: pingomatic technorati blogsearch.google.com feedburner Any others that should be included on this list? My goal is not to spam these ping lists however want to make sure my content is getting indexed quickly
Intermediate & Advanced SEO | | webestate0 -
Link + noindex vs canonical--which is better?
In this article http://support.google.com/webmasters/bin/answer.py?hl=en&answer=66359 google mentions if you syndicate content, you should include a link and, ideally noindex, the content, if possible. I'm wondering why google doesn't mention including a canonical instead the link + noindex? Is one better than the other? Any ideas?
Intermediate & Advanced SEO | | nicole.healthline0