How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Referral Data Q's
1. We recently ran a promotion on both FB and Reddit, which is https, linking to our non-https site. We utilized UTM links to our landing page. Our GA campaign data returned extremely low hits in comparison to what we actually received (and recorded via FB/Reddit dashboard). Obviously our Direct traffic spiked during these times, caused by a secure to nonsecure referral, I'm sure. I'm also noticing a spike in referral traffic from lm.facebook.com that correlates to the ad times. Does this mean Facebook's link shim is stripping away my UTM data? My question is why we receive SOME properly UTM-tagged referral traffic in our campaigns? What's allowing some of it to go through? 2. I've tagged our email signature links with UTM as well, hoping to clean up some of our Direct traffic. I understand that external clients like Outlook and Thunderbird likely won't pass referral data, but do hosted clients like Gmail, Yahoo, and such? And if so, would the https to http difference obstruct this again? I'd love some insight onto these questions, especially if I'm off the mark with a few of my assumptions there.
Reporting & Analytics | | kirmeliux0 -
Why is this tag not firing in Google Analytics?
I setup Google Tag Manager on this site- http://germanhausbarn.com I am trying to setup event tracking for the donate, newsletter, and Contact Us button at the bottom of the page. The most recent version is published, and I ran debug and it shows that they fire, but nothing is coming up in analytics. Any thoughts?
Reporting & Analytics | | EcommerceSite0 -
Results for wrong keyword
I've started to look at ranking and visitor behaviour within a specific product category and I've come across this strange data in GA that i'm struggling to get my head around. keyword - wishbone necklace gold landing - ...necklace-dive-in.html bounce - 99.31% new visits - 0.00% The landing page is not anywhere in the results for the given keyword (nor does the keyword appear anywhere within the page). The data is spread out to date since Dec 2011 and the landing page accounts for about 97% of the traffic for that keyword. I then looked at browser which is 95% Safari (different versions) and breaking that down into City about 30% is (not set), 30% a single city and then there is a spread of locations. It might be fair to assume that therefore 60% could be the same location but there is still 40% to take into consideration. What I can't get my head around is how the landing page is being accessed so regularly for the wrong keyword. The correctly ranking URL only accounts for 5% of the traffic which is more in line with the estimated search volumes for that keyword. I've checked versions of the page going back and none contain the keyword. Am I missing something, or any ideas how to fix?
Reporting & Analytics | | MickEdwards0 -
Blended results in google ???
I am a little confused about my search results. My ranking on google is displayed as #42. HOWEVER, I am #1 in the LOCAL results... what does that mean? Am I really #42? or am I #1 because local is listed over the organic results?
Reporting & Analytics | | drshahprs0 -
Google Analytics traffic hijacking?
Ran into something interesting a week ago - the same Google Analytics code was installed on two different sites by accident. The account was reporting traffic from both domains. Haven't found a definitive answer on how to stop this yet if it were to be used maliciously?
Reporting & Analytics | | khemistry0 -
What's the final word on Image Search tracking in Google Analytics?
Sorry if this has been answered but I can't seem to get a straight answer to my questions by searching around. How is traffic referred by Google Images counted in Google Analytics? I know it used to be referral traffic from google.com/imgres. A lot of things I have read say that it should all be under google/organic now, but my site still gets referral traffic from google.com/imgres, so that can't be. However I also get traffic as google/organic that I am pretty sure is from image search, because we don't rank for the keyword normally, but we do for image search. What's the deal? How is traffic from an embedded image in a regular result page counted? How can I segment my image search traffic better? It would be great to see image search traffic as it's own medium. I found a script here -- http://jrom.net/google-images-in-google-analytics -- that looks promising, has anyone used it or can recommend another way? I haven't used the GA API very much so I want to make sure the script is kosher and won't screw up my numbers.
Reporting & Analytics | | tact0 -
Viewing 'overall' data for multiple Google Analytics accounts
Is there any way you can view data from all of your Google Analytics accounts? For example, if I wanted to view know how much mobile traffic all my sites had, could I do this? Rather than just looking at each site individually. Thanks
Reporting & Analytics | | intSchools0 -
Where are google analytics stats for iphone4
hi We were looking at the Google Analytics for one of our sites and noticed that there were NO pageviews from device=iphone and resolution=640x960 in the report. Given that iphone4 is supposed to be 640x960, and would be the most popular device (at least in our offices and everyone I know), it seems wierd. I sorted the Mobile Devices report by device and resolution to see what was available. The first 160 results were all device=not set. Finally got to device=iPhone and there were three entries: resolution 0x0 had 11 views resolution 320x396 had 45 views resolution 320x480 had 3,944 views. Hopefully all iphone4 users havent been classified as not set. Or is it possible that iphone4s claim to be 320x480 in browsers, as per http://www.alistapart.com/articles/a-pixel-identity-crisis/ Even worse, if I look at the Samsung Galaxy S II (myown phone), there are over 30 screen resolution combinations. Does anyone have anything to shed on this? I asked about it on the google analytics twitter account last week but havent had a response. Are there other analytics solutions that would distinguish between the iphones? Warning - this is a link to a large image, with the not set stats at the top. 6Sjji
Reporting & Analytics | | ozgeekmum0