Search directory - How to apply robots
-
Hi.
On the site I'm working on, we use a search directory to display our search results. It displays as follows -
With the dynamic search results appearing after the hash tag.
Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav.
How should I handle this?
Thanks.
-
That was my first instinct as well - but I wanted to check if there was a better move? Thanks for your help!
-
Have you considered switching to simply "noindex, follow" for these pages? This would both keep them out of the search index and would allow for the benefit of external links to these pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How does ARIA-hidden text appear to search engines
I'm having trouble getting my accessibility team to add alt text to our site's images for SEO benefits as they feel some of it would add additional noise for screen readers. They proposed using ARIA-hidden attributes to hide the text but I'm wondering if will that be interpreted as a cloaking tactic to search engines? Also, I'm wondering if it the alt text will carry the same weight if ARIA-hidden is used. Has anyone had any experience with this? I'm having trouble finding any research on the topic.
Web Design | | KJ6001 -
Disallow: /sr/ and Disallow: /si/ - robots.txt
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember. If any of you know what these directives do, then please let me know.
Web Design | | McTaggart0 -
Google Search Console Block
Am new to SEO. My clients site was completed using Yoast premium and then used Google search console to initiate the crawl. Initially setup an http:// property and all seemed good. Then i removed that under search console an created an https:// did the render and it appears google has put a block and placed their own robots.txt file which basically has rendered the site useless. Feedback most appreciated.
Web Design | | BoostMyBiz0 -
My news site not showing in "In the news" list on Google Web Search
I got a news website (www.tapscape.com) which is 6 years old and has been on Google News since 2012. However, whenever I publish a news article, it never shows up "In the news" list on Google Web Search. I have already added the schema.org/NewsArticle on the website and have checked it if it's working or not on Google structured data testing tool. I see everything shows on on the structured data testing tool. The site already has a news sitemap (http://www.tapscape.com/news-sitemap.xml) and has been added to Google webmaster tools. News articles show perfectly fine in the News tab, but why isn't the articles being shown on "In the news" list on the Google web search? My site has a strong backlink background already, so I don't think I need to work on the backlinks. Please let me know what I'm doing wrong, and how can I get it to the news articles on "In the news" list. Below is a screenshot that I have attached to this question to help you understand what I mean to say. 1qoArRs
Web Design | | hakhan2010 -
Robots.txt - Allow and Disallow. Can they be the same?
Hi All, I need some help on the following: Are the following commands the same? User-agent: * Disallow: or User-agent: * Allow: / I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots. Is that correct? Many thanks, Aidan
Web Design | | Presenter0 -
Can you recommend a responsive web design company of template to accommodate mobile searches
I need to change my old Dreamweaver website www.endeavourcottage.co.uk into a responsive website that can be viewed and used easily on mobile phones. Four years ago when the site was originally built it was one cottage on one basic website but it's grown into a small letting agency with several properties, so I think it's time to invest in a fully functional website. Looking at Google analytics around the fifth of the searches and now done on a mobile phone each month on this particular site, and obviously that's going to become a larger percentage in the future. Does anybody know of a particular company that can produce good quality responsive websites? Or alternatively used a particular responsive website template which was good. One of the guys that work for me is quite a good web designer if we got the right template. The old website is grown into quite a large beast with a blog so I think the idea would be to change the main pages with the properties on, just linking back to the old websites quite extensive blog / historical interest pages which I've built over the last few years. I need to keep the old blog webpages indexed because they have hundreds of back links coming with longtail keywords producing traffic. Have you gone through the process of changing your main business website for a resonsive redesign recently and how did you approach it?
Web Design | | whitbycottages0 -
Is there any difference in using an underscore vs. a dash in the directory portion of the url?
A friend who is a software developer asked this question regarding the directory portion of the url: Is it better to use dashes or underscores? I know in the domain name Matt Cutts recommends dashes, but what about the directory extension?
Web Design | | RobertFisher0 -
Need advice on diplaying content for Search
Hi every body, I am doing landing page redesign(s). Does any body know or can refer a content carousel that can rotate video and pictures? The "site with images" search option result is a compelling reason to showcase pictures if your space competeiveness (showroom, merchandise, etc) can be improved with a strong image presence. here is my main landing page http://www.shearerpainting.com I know there is alot of stuff, and confusing call to action, but I am looking for strategies to clean it up, clear fous on action (get bid, learn more), but allow users to see that they can dig for more content.
Web Design | | johnshearer0