Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Correct robots.txt for WordPress
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong? I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right. Thank you so much. FzSQkqB.jpg
On-Page Optimization | | AslanBarselinov1 -
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
SVG image files causing multiple title tags on page - SEO issue?
Does anyone have any experience with SVG image files and on-page SEO? A client is using them and it seems they use the title tag in the same way a regular image (JPG/PNG) would use an image ALT tag. I'm concerned that search engines will see the multiple title tags on the page and that this will cause SEO issues. Regular crawlers like Moz flag it as a second title tag, however it's outside the header and in a SVG wrap so the crawlers really should understand that this is a SVG title rather than a second page title. But is this the case? If anyone has experience with this, I'd love to hear about it.
On-Page Optimization | | mrdavidingram2 -
Is it better to put all your CSS in 1 file or is it no problem to use 10 files or more like on most frameworks?
Is it better to put all your CSS in 1 file or is it no problem to use 10 files or more like on most frameworks?
On-Page Optimization | | conversal0 -
Problemas with my htaccess file
Hi all, I have two doamins actived which content is the same. Regarding Google some of you guys told me that I wouldn't be penalized but If I wanted to do a redirection 301 it should be done from one domain to the domain which is my main market. And I just did it. But the problem is that although I have made the corresponding modifications in the htaccess file it doesn' work. When I write www.piensapiensa.com goes to piensapiensa.com (as I configured the WMT) and not to www.piensapiensa.es, in which my market is mainly present. Heres the code for the HTACCESS file: Options +FollowSymLinks +ExecCGI
On-Page Optimization | | juanmiguelcr
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.piensapiensa.es [NC]
RewriteRule (.*) http://www.piensapiensa.es/$1 [R=301,L]
<ifmodule mod_rewrite.c="">RewriteEngine On # uncomment the following line, if you are having trouble
# getting no_script_name to work
#RewriteBase / # we skip all files with .something
#RewriteCond %{REQUEST_URI} ..+$
#RewriteCond %{REQUEST_URI} !.html$
#RewriteRule .* - [L] # we check if the .html version is here (caching)
RewriteRule ^$ index.html [QSA]
RewriteRule ^([^.]+)$ $1.html [QSA]
RewriteCond %{REQUEST_FILENAME} !-f # no, so we redirect to our front web controller
RewriteRule ^(.*)$ index.php [QSA,L]</ifmodule> Thanks in advanced.0 -
Two sites, one with a ccTLD domain, the other with TLD domain, same content
Hi there! I have a site which can be accessed with two different domains: one ccTLD for Spain: www.piensapiensa.es one TLD www.piensapiensa.com Should I take care of something regarding SEO? I have also a redirection from www.piensapiensa.com to piensapiensa.com. I have set up them in webmasters tools individually, with the same sitemap obviously. Thanks in advanced.
On-Page Optimization | | juanmiguelcr0 -
One or two keywords/pages
Hi, I have a question about good keyword practice. I have a page: http://www.holdnyt.dk/skader-karantaener/fodbold/england-premier-league It lists all injuries ("skader" in danish) and suspensions ("karantaener" in danish) for the english premier league in football/soccer. On the page one can choose to show only injuries OR suspensions, which have their own URLs: http://www.holdnyt.dk/skader-karantaener/fodbold/england-premier-league/skader http://www.holdnyt.dk/skader-karantaener/fodbold/england-premier-league/karantaener My question is - what is best. To optimize the first URL (the more general one) to fit both of the following keywords:
On-Page Optimization | | rasmusbang
"skader premier league" and "karantaener premier league" OR should i focus on optimizing the two latter URLs, the more specfiic to target the two keywords. Regards, Rasmus0 -
I have two pages ranking for the same keyword.
The index page and the targeted landing page for that keyword. They have different content, title, meta but I am competing with myself for the main keyword in the industry. What is the best way to fix this? 301 the keyword page to the index page?
On-Page Optimization | | Aftermath_SEO0