Two Robots.txt files
-
Hi there
Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one.
Thanks in advance!
Cheers,
Satla -
Thank Riera
-
Hi Satia,
You mentioned that one robots.txt file placed in inside the server and search engine unable to see the file. If search engine won't see robots.txt file then what is the use of that robots.txt file?
AFAIK it must be placed under root directory an there is no way to keep two files with the same name. So you must have only one robots.txt and that should be placed under root directory.
Hope this helps.
Thanks
-
Hi Satla,
You're going to need to get rid of that 2nd version ASAP. The official standard for a robots.txt file is all lower case in the file name, so that's most likely what bots are seeing. But to err on the side of caution, I'd remove any possibilities of a "disallow: /" and remove that Robots.txt version.
Some servers are case sensitive, so you could run into issues here as well.
-
Hi,
I don't find any good reason why there are two files. There should be just one, where you specify everything you'd like to done.
If the tech team doesn't want to correct and leave just one file, may be because they are lazy or there might be some other issue that if they delete one file, the hole site blows up.Here, I leave you 2 Moz's articles about the robots.txt file.
What is Robots.txt? - Moz Learn
Learn About Robots.txt with Interactive Examples - Moz BlogTake into account that the name of the file must be in lower case. I've never seen any different and the servers are usually case sensitive with the filenames.
Hope its helpful.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Two Day After Starting Moz Pro campaign i experienced Sudden Huge Traffic Drop and My site ranki Drops in my site..Please I need your help
Two Day After Starting Moz Pro campaign i experienced Sudden Huge Traffic Drop and Drops in my site..Please I need your help
On-Page Optimization | | zizutz0 -
Robot.txt file issue on wordpress site.
I m facing the issue with robot.txt file on my blog. Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue. The search result shows that "A description for this result is not available because of this site's robots.txt – learn more." Any suggestion to over come with this issue
On-Page Optimization | | Mustansar0 -
Two sites into one
I have two sites owned by one client, he wants to merge them into one keeping one website, but which one? I've been using the Moz Pro to look at the stats for both sites; page authority, inbound links etc, but they're both fairly close in results. The client wants to know what would be the best course to take with these two sites, what site should he keep and which should he merge? Any advice?
On-Page Optimization | | barrowr0 -
Internal Linking : File Name or URI/filename.html
Hi, For crawlability, what would be the best way to do internal linking. /aboutus.html or www.xyz.com/aboutus.com Would this make any difference to the website in terms of SEO or bot crawling the website? Regards, Sree
On-Page Optimization | | jungleegames0 -
How important are image file names
Hi, How important do you think the image file names are for image search?
On-Page Optimization | | jjtech
I know it used to be the best practice a while ago but is it still important? Thanks in advance, JJ0 -
Two keyowrds for one page
Hi there! I just optimize two pages for the same keyword as I didn't find a especific keyword for each one independently. On the other hand, these keyword was "suitalbe" for the same two pages. Obviously Google will have to "make a decision" regarding what page should it be indexed in the first postion? In what aspects or elements of the page should I incide in order to give priority to one page more than the other one? Thnaks
On-Page Optimization | | juanmiguelcr0 -
Positioning one site in two languages
Hi all, I have used the plugin wpml in order to have one site in different languages (Spanish and English). The problem is that this plugin doesn't work with all in one SEO plugin so that the spanish home and the english home have the same title. Google has positionated in the first position the english one in google.es. How can I revert the situation if I fix the problem with other plugin? Thanks.
On-Page Optimization | | juanmiguelcr0 -
Image files names - should each be different ?
Hi SEOs I have created a page on my website with a certain tutorial - let's say "How to change a tyre in your car". I have Grade A from On-page report using SEOmoz tool. I have put a relevant alt description for each photo I have put in that tutorial and I have used relevant keyword as a name for each image. So for example this would look like that: changing-tyre-1.jpg changing-tyre-2.jpg My question is should I name these file differently according to their content ? For example if on my photo there is some tool I have to use to change the tyre should the file have the name relevant to the subject of the article/tutorial on the page or to the content of the image ? Is that a stupid question ? Or am I getting too fussy ? Thanks
On-Page Optimization | | lolskizz0