Robots.txt | any SEO advantage to having one vs not having one?
-
Neither of my sites has a robots.txt file. I guess I have never been bothered by any particular bot enough to exclude it.
Is there any SEO advantage to having one anyways?
-
It's good practice, especially if you are operating a CMS that can create accessible URLs that cause duplicate content problems, create "junk" pages, etc. For example: http://www.asos.com/robots.txt
Google dislikes search results pages being indexed, so you can block those off, e.g. http://moz.com/robots.txt
You can disallow the archive.org bot if you don't want old versions of your site appearing in its search engine, and as others have said you can point to your xml sitemap.
It's not a bad resource to have at your disposal for site hygiene / maintenance reasons, but it's not an absolute necessity either.
-
There are actually a couple good reasons but in short, it's "best practice" so it won't hurt by adding it in. It wont take more than a couple minutes.
-
Just good practice. One SEO advantage would be to include a reference to your sitemap within the robots.txt file.
Aside from that, if you want all of your pages crawled and don't have a sitemap (although you should), no need for a robots.txt file.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two websites, one company, one physical address - how to make the best of it in terms of local visibility?
Hello! I have one company which will be operating in two markets, printing and website design / development. I’m planning on building two websites, each for every market. But I’m a bit confused about how to optimize these websites locally. My thought is to use my physical address for one website (build citations, get listed in directories, etc. ) and PO Box for another. Do you think there is a better idea?
Technical SEO | | VELV1 -
Robots.txt error
Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.
Technical SEO | | Shanidel0 -
Negative SEO
I have got some idiot bombarding my site with spam links, most likely a negative SEO attempt. It started off very small and has ramped up to between five and 10 spam links per day. I think it may be having a slightly negative affect although difficult to tell. I can do a link cleanup, but I'm not sure that is a long term solution, I'm just going have to do it again in a few weeks time. Does anyone have any experience?
Technical SEO | | seoman100 -
YouTube vs. LimeLight - What are the SEO pros and cons of each platform for on-site video viewing?
Are there any SEO benefits of hosting on-site videos using LimeLight? I know the various benefits of using YouTube but before going forward with a site redesign I want to hear what others have to say. Thanks, Jake
Technical SEO | | JakeMatulewicz1 -
I've consolidated other domains to a single one with 301 redirects, yet the new domain authority in MOZ is much less that the redirected ones. Is that right?
I'm trying to increase the domain authority of my main site, so decided to consolidate other sites. One of the other sites has a much higher domain authority, but I don't know why after a 301 redirect, the new site's domain authority hasn't changed on over a month. Does MOZ take account of thes types of things?
Technical SEO | | bytecgroup2 -
Robots.txt best practices & tips
Hey, I was wondering if someone could give me some advice on whether I should block the robots.txt file from the average user (not from googlebot, yandex, etc)? If so, how would I go about doing this? With .htaccess I'm guessing - but not an expert. What can people do with the information in the file? Maybe someone can give me some "best practices"? (I have a wordpress based website) Thanks in advance!
Technical SEO | | JonathanRolande0 -
SEO LINKS
New to S.E.O. so excuse my naivety. I have made lots of new links some of them paid for e.g. Best of the Web but I don’t see any change in the latest competitive link analysis. Some of the links we have been accepted for just do not show. Also the keywords we are trying to promote the most have disappeared off the radar for over 2 weeks now. I think we have followed the optimization suggestions correctly. Please could you enlighten me. Regards Paul www.curtainpolesemporium.co.uk
Technical SEO | | CPE0 -
Need Help With Robots.txt on Magento eCommerce Site
Hello, I am having difficulty getting my robots.txt file to be configured properly. I am getting error emails from Google products stating they can't view our products because they are being blocked, and this past week, in my SEO dashboard, the URL's receiving search traffic dropped by almost 40%. Is there anyone that can offer assistance on a good template robots.txt file I can use for a Magento eCommerce website? The one I am currently using was found at this site here: e-commercewebdesign.co.uk/blog/magento-seo/magento-robots-txt-seo.php - However, I am getting problems from Google now because of it. I searched and found this thread here: http://www.magentocommerce.com/wiki/multi-store_set_up/multiple_website_setup_with_different_document_roots#the_root_folder_robots.txt_file - But I felt like maybe I should get some additional help on properly configuring a robots for a Magento site. Thanks in advance for any help. Please, let me know if you need more info to provide assistance.
Technical SEO | | JerDoggMckoy0