Robots.txt | any SEO advantage to having one vs not having one?
-
Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.
Is there any SEO advantage to having one anyways?
-
It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt
Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt
You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.
It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.
-
There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.
-
Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.
Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Negative SEO
I have got some idiot bombarding my site with spam links, most likely a negative SEO attempt. It started off very small and has ramped up to between five and 10 spam links per day. I think it may be having a slightly negative affect although difficult to tell. I can do a link cleanup, but I'm not sure that is a long term solution, I'm just going have to do it again in a few weeks time. Does anyone have any experience?
Technical SEO | | seoman100 -
Would this harm my SEO?
Hello, I own a shop, and I want to add the most searched keywords to my products page , example : Nike Aero , add this keywords to the product page : Nike Aero buy Where to buy Nike aero ect , I with the most searched keywords Would it harm my SEO? Would it be good? Thank you!
Technical SEO | | coolninja0 -
Redirection Impact on SEO
Need help urgently. There is the situation [This is how is it working now]: 1. Have a global landing page [say when user types in www.mysite.com - takes user to the global landing page: [www.mysite.com/global/en.html]](http://www.mysite.com/global/en.html] ) 2. Users from this landing page can select a country on his/her choice and get redirected say: [www.mysite.com/us/en.html] Would like to change the functionality as below: 1. When user types in www.mysite.com 1a. Would find the location of the request based on GEO IP and if the request is coming from North America region then would redirect the users to: www.mysite.com/us/en.html 1b. If the request is from any other location/region then it would continue to work as it is currently working: take the user to the global landing page: www.mysite.com/global/en.html Would this change have any negative impact or not found by search engines from SEO perspective? If it does then what are the impacts and if does not then why not. If it does then what is the best possible way to address this request. Appriciate your help. Thanks, Koushik Roy
Technical SEO | | KoushikRoy0 -
Name Servers & SEO
We have decided to create a few blogs and will eventually be linking to some of our clients. I have domain privacy and different class C addresses for each of my domains. But the name servers area all the same. Ex: If we create an article for one client on all 5 blogs, will the name servers be a problem?
Technical SEO | | waqid0 -
Pintrest SEO
Has any testing been done to determine if Pintrest helps a website ranking?
Technical SEO | | StreetwiseReports0 -
Robots.txt versus sitemap
Hi everyone, Lets say we have a robots.txt that disallows specific folders on our website, but a sitemap submitted in Google Webmaster Tools that lists content in those folders. Who wins? Will the sitemap content get indexed even if it's blocked by robots.txt? I know content that is blocked by robot.txt can still get indexed and display a URL if Google discovers it via a link so I'm wondering if that would happen in this scenario too. Thanks!
Technical SEO | | anthematic0 -
If multiple links on a page point to the same URL, and one of them is no-followed, does that impact the one that isn't?
Page A has two links on it that both point to Page B. Link 1 isn't no-follow, but Link 2 is. Will Page A pass any juice to Page B?
Technical SEO | | Jay.Neely0 -
Is blocking RSS Feeds with robots.txt necessary?
Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | | nicole.healthline0