Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt on subdomains
Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
Technical SEO | | Whittie0 -
Website Pop up
I want to install a pop up on my site for collecting email addresses. I use nopcommerce as my platform. I was quoted a price to custom develop it but thought there might just be a simple plugin. I can find them for wordpress but that is all. Does anybody have experience with this?
Technical SEO | | katazoom0 -
Is this normal on my website speed tool
Hi, i would like to know if this is normal as i have never come across it before. i have just checked my speed which needs a lot of improving. I use joomla 3.0 and recently had a developer upgrade it from 1.5, but now i am seeing under the speed test my website showing twice which looks like it is causing a time delay. the tool is http://tools.pingdom.com/fpt/#!/dTjwDM/www.in2town.co.uk can someone please look and let me know if this is normal. my site www.in2town.co.uk is coming up twice which seems to be slowing the site down and i have checked this tool with other sites and they are fine many thanks
Technical SEO | | ClaireH-1848860 -
Duplicate Title - Shopping Website
My website is showing with errors due to duplicate titles, these errors add up to over 150 errors. The problem is a lot of my products are similar, just in different lengths, colors etc. For example
Technical SEO | | adelemaree
http://www.ugg.org.au/en/classic-candy-short-australia-ugg-boots/48-classic-candy-short-australia-ugg-boots-blue.html
http://www.ugg.org.au/en/classic-candy-button-short-australia-ugg-boots/31-classic-candy-button-short-chocolate.html
http://www.ugg.org.au/en/classic-candy-button-short-australia-ugg-boots/28-classic-candy-button-short-black.html How can I avoid this error without having to merge all my products and colors into one page? As that would make my catalogue look too small! Thanks in advance0 -
Is my robots.txt file working?
Greetings from medieval York UK 🙂 Everytime to you enter my name & Liz this page is returned in Google:
Technical SEO | | Nightwing
http://www.davidclick.com/web_page/al_liz.htm But i have the following robots txt file which has been in place a few weeks User-agent: * Disallow: /york_wedding_photographer_advice_pre_wedding_photoshoot.htm Disallow: /york_wedding_photographer_advice.htm Disallow: /york_wedding_photographer_advice_copyright_free_wedding_photography.htm Disallow: /web_page/prices.htm Disallow: /web_page/about_me.htm Disallow: /web_page/thumbnails4.htm Disallow: /web_page/thumbnails.html Disallow: /web_page/al_liz.htm Disallow: /web_page/york_wedding_photographer_advice.htm Allow: / So my question is please... "Why is this page appearing in the SERPS when its blocked in the robots txt file e.g.: Disallow: /web_page/al_liz.htm" ANy insights welcome 🙂0 -
Wrong page ranking even after fixes. Why?
Howdy, I've read a number of posts on this issue and followed all the advice I could find. My issue is this: several of our targeted keywords have the wrong page ranking which results in a loss of conversion and high bounce rate. I use the site:mydomain.com operator to see how the pages are ranking which I find really useful. One of our targeted keywords is 'wedding stationery' and the page which should rank is shop.confetti.co.uk/branch/wedding-stationery however this page ranks 3rd on our domain. I've done the following: Optimised the correct page De-optimised the ranking pages Added links from the incorrectly ranking page to the page I want to rank using keyword anchor text Added internal links to the page I want to rank from high Page Authority pages using keyword anchor text Here is a link to the SERP which shows the ranking for our domain for the 'wedding stationery' search term. I made the changes a couple of weeks ago. Any advice or explanations as to what is causing this would be greatly received. Thanks in advance, Brendan.
Technical SEO | | Confetti_Wedding0 -
Is it terrible to not have robots.txt ?
I was under the impression that you really should have a robots.txt page, and not having one is pretty bad. However, hubspot (which I'm not impressed with) does not have the capability of properly implementing one. Will this hurt the site?
Technical SEO | | StandUpCubicles1 -
Do you validate you websites?
Do you consider the guidelines from http://validator.w3.org/ when setting up a new website? As far as I know they don't influence rankings ... What is your opinion about that topix?
Technical SEO | | petrakraft0