Robots.txt advice
-
Hey Guys,
Have you ever seen coding like this in a robots.txt, I have never seen a noindex rule in a robots.txt file before - have you?
user-agent: AhrefsBot
User-agent: trovitBot
User-agent: Nutch
User-agent: Baiduspider
Disallow: /User-agent: *
Disallow: /WebServices/
Disallow: /*?notfound=
Disallow: /?list=
Noindex: /?*list=
Noindex: /local/
Disallow: /local/
Noindex: /handle/
Disallow: /handle/
Noindex: /Handle/
Disallow: /Handle/
Noindex: /localsites/
Disallow: /localsites/
Noindex: /search/
Disallow: /search/
Noindex: /Search/
Disallow: /Search/
Disallow: ?I have never seen a noindex rule in a robots.txt file before - have you?
Any pointers? -
Never seen this, doubt it's any useful as this isn't part of any search engines recommended statements to use. I don't think this would have any impact on what search engine robots would look at as it's not a statement in the robots.txt documentation.
-
Best I could find was-
Unlike disallowed pages, noindexed pages don’t end up in the index and therefore won’t show in search results. Combine both in robots.txt to optimise your crawl efficiency: the noindex will stop the page showing in search results, and the disallow will stop it being crawled
From-https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Very Old Pages Creeping Up - Advice
We are currently having very old pages dating back 5+ years ago appearing on moz all of a sudden, we don't necessarily get traffic from these links anymore and i doubt they still hold any weight. Currently they take you to a 404 page, would there be any worth in redirecting these links?
Intermediate & Advanced SEO | | JH_OffLimits0 -
Large robots.txt file
We're looking at potentially creating a robots.txt with 1450 lines in it. This will remove 100k+ pages from the crawl that are all old pages (I know, the ideal would be to delete/noindex but not viable unfortunately) Now the issue i'm thinking is that a large robots.txt will either stop the robots.txt from being followed or will slow our crawl rate down. Does anybody have any experience with a robots.txt of that size?
Intermediate & Advanced SEO | | ThomasHarvey0 -
Robots.txt - blocking JavaScript and CSS, best practice for Magento
Hi Mozzers, I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento. I'm concerned we are blocking bots from crawling essential information for page rank. My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not? You can view our robots.txt file here Thanks, Blake
Intermediate & Advanced SEO | | LeapOfBelief0 -
Screaming frog Advice
Hi I am trying to crawl my site and it keeps crashing. My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes. It gets to around 200k pages crawl and dies. Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site. Thanks in advance for any tips. Andy
Intermediate & Advanced SEO | | Andy-Halliday0 -
SEO advice with having a blog on sub domain.
Righto, so: I've been working on our company website www.nursesfornurses.com.au which is built on .asp which is a real pain because the site is built so messy and on a very dated CMS which means I have to go back to the dev every time I want to make a change. We've made the decision to move the site over to Wordpress in stages. So, (and I hope logically), i've started by making them a proper blog with better architecture to start targeting industry related keywords. I had to put it on a sub domain as the current hosting does not support Wordpress http://news.nursesfornurses.com.au/Nursing-news/
Intermediate & Advanced SEO | | 9868john
The previous blog is here: http://www.nursesfornurses.com.au/blog Its not live yet, so I'm just looking for SEO advice or issues I might encounter by having the blog on a sub domain. In terms of user experience, I realise that there needs a clearer link back to the main website, I'm just trying to work out the best way to do it... Any advice / criticism is greatly welcomed. Thanks0 -
Advice needed on how to handle alleged duplicate content and titles
Hi I wonder if anyone can advise on something that's got me scratching my head. The following are examples of urls which are deemed to have duplicate content and title tags. This causes around 8000 errors, which (for the most part) are valid urls because they provide different views on market data. e.g. #1 is the summary, while #2 is 'Holdings and Sector weightings'. #3 is odd because it's crawling the anchored link. I didn't think hashes were crawled? I'd like some advice on how best to handle these, because, really they're just queries against a master url and I'd like to remove the noise around duplicate errors so that I can focus on some other true duplicate url issues we have. Here's some example urls on the same page which are deemed as duplicates. 1) http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Holdings-and-sectors-weighting?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE&widgets=1 What's the best way to handle this?
Intermediate & Advanced SEO | | SearchPM0 -
Any advice for setting up a Job Board?
Hi- I've got a big client who is setting up an in-house job board, ie. as part of their own site. They are a diverse company that offers a wide variety of roles across the whole country. The software they have chosen to use is not in any way SEO-focused. Therefore I'll need to recommend some modifications to the sitemap created by the web design team, within the time and budget available to me. At this stage I am thinking along the lines of determining the major geographical areas and job sectors and creating summary (landing) pages such as /jobs/california/electrical-engineering which contain any currently available roles for that geo+sector. I've tried to find guidance on job board SEO optimization or even case studies but haven't found much at all. This seems useful though: http://www.jobboardmount.com/cm/features/seo_dashboard Does anyone have any tips or links to useful information on job board SEO? Thanks in advance! Jules
Intermediate & Advanced SEO | | Juller1 -
Blocking Dynamic URLs with Robots.txt
Background: My e-commerce site uses a lot of layered navigation and sorting links. While this is great for users, it ends up in a lot of URL variations of the same page being crawled by Google. For example, a standard category page: www.mysite.com/widgets.html ...which uses a "Price" layered navigation sidebar to filter products based on price also produces the following URLs which link to the same page: http://www.mysite.com/widgets.html?price=1%2C250 http://www.mysite.com/widgets.html?price=2%2C250 http://www.mysite.com/widgets.html?price=3%2C250 As there are literally thousands of these URL variations being indexed, so I'd like to use Robots.txt to disallow these variations. Question: Is this a wise thing to do? Or does Google take into account layered navigation links by default, and I don't need to worry. To implement, I was going to do the following in Robots.txt: User-agent: * Disallow: /*? Disallow: /*= ....which would prevent any dynamic URL with a '?" or '=' from being indexed. Is there a better way to do this, or is this a good solution? Thank you!
Intermediate & Advanced SEO | | AndrewY1