Do robot.txts permanently affect websites even after they have been removed?
-
A client has a Wordpress blog to sit alongside their company website. They kept it hidden whilst they were developing what it looked like, keeping it un-searchable by Search Engines. It was still live, but Wordpress put a robots.txt in place. When they were ready they removed the robots.txt by clicking the "allow Search Engines to crawl this site" button.
It took a month and a half for their blog to show in Search Engines once the robot.txt was removed.
Google is now recognising the site (as a "site:" test has shown) however, it doesn't rank well for anything. This is despite the fact they are targeting keywords with very little organic competition.
My question is - could the fact that they developed the site behind a robot.txt (rather than offline) mean the site is permanently affected by the robot.txt in the eyes of the Search Engines, even after that robot.txt has been removed?
Thanks in advance for any light you can shed on the situation.
-
No problem! Good Luck!
-
That is a very fair point. It is a completely new site and I hadn't even thought about things like the domain age. It does show up under a "site:http://www.____.com" search, I was just wondering if this is one of those things Google keeps a memory of, if that makes sense.
Thanks for your response Mike.
-
That is a very good suggestion. I'll try it (a useful URL also so thanks for sharing).
Thanks for the response Matthew.
-
I think the much more likely culprit is that it is a new site. What do you get when you enter "site:http://www._____.com" in google? If the pages are indexed, one can't blame for the robots file for lack of rank.
Good luck!
Mike
-
Have you submitted the updated robots.txt to google? This is separate from updating the sitemap. Here is a google page to help you do this.
https://support.google.com/webmasters/answer/6078399?hl=en
Best!
Matthew
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt Syntax for Dynamic URLs
I want to Disallow certain dynamic pages in robots.txt and am unsure of the proper syntax. The pages I want to disallow all include the string ?Page= Which is the proper syntax?
Technical SEO | | btreloar
Disallow: ?Page=
Disallow: ?Page=*
Disallow: ?Page=
Or something else?0 -
Robots file set up
The robots file looks like it has been set up in a very messy way.
Technical SEO | | mcwork
I understand the # will comment out a line, does this mean the sitemap would
not be picked up?
Disallow: /js/ should this be allowed like /*.js$
Disallow: /media/wysiwyg/ - this seems to be causing alerts in webmaster tools as it can not access
the images within.
Can anyone help me clean this up please #Sitemap: https://examplesite.com/sitemap.xml Crawlers Setup User-agent: *
Crawl-delay: 10 Allowable Index Mind that Allow is not an official standard Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/ Allow: /catalogsearch/result/ Allow: /media/catalog/ Directories Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/ Disallow: /media/ Disallow: /media/captcha/ Disallow: /media/catalog/ #Disallow: /media/css/
#Disallow: /media/css_secure/
Disallow: /media/customer/
Disallow: /media/dhl/
Disallow: /media/downloadable/
Disallow: /media/import/
#Disallow: /media/js/
Disallow: /media/pdf/
Disallow: /media/sales/
Disallow: /media/tmp/
Disallow: /media/wysiwyg/
Disallow: /media/xmlconnect/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
#Disallow: /skin/
Disallow: /stats/
Disallow: /var/ Paths (clean URLs) Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product/gallery/
Disallow: */catalog/product/upload/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Disallow: /get.php # Magento 1.5+ Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID=
Disallow: /rss*
Disallow: /*PHPSESSID Disallow: /:
Disallow: /😘 User-agent: Fatbot
Disallow: / User-agent: TwengaBot-2.0
Disallow: /0 -
Is my website is over optimized for ON page SEO?
The keyword for the page is “locksmith Logan” based in: Brisbane, Queensland, Australia Is webpage over used main keyword 'Logan locksmith' and what other areas are for improvement.
Technical SEO | | bondhoward0 -
Duplicate website with http & https
I have a website that only in a specific state in the USA we had to add a certificate for it to appear with https. my question is how to prevent from the website to be penalized on duplicate content with the http version on that specific state. please advise. thanks!
Technical SEO | | taly0 -
How to remove a thin site penalty
Wondering if anyone could help out. A while back I made an affiliate store using wordpress and merchants products feeds. I didn't get found to adding any unique content to the site and, as was to be expected, I gained a penalty and my search traffic died. A few months back I redesigned the store, still using merchant csv but now with 98% unique content on each page. However, try as I may I still cannot get anywhere in the engines. The domain doesn't even rank for it's own name!! I have submitted reconsideration request but they have replied saying no penalty on the site. The domain is www.digitalcatwalk.co.uk. While the domain isn't massively strong I would prefer not to have to start again as I feel it is a very good domain name. Any advise would be most gratefully received. Thanks Carl
Technical SEO | | GrumpyCarl0 -
How To Proceed When A Portion of Website has been hijacked.
Hi - I've recently learn that a site I manage: http://www.hhisland.com has somehow been hijacked by other sites (examples below): http://wlwhost.info/schering-07-nissan-altima-air-conditioner-drain-clogged/ (460 links)
Technical SEO | | hhdentist
http://abhinav.co.uk/Mary-motorcycles-for-sale-in-iasi/ (440 links)
http://www.turetzky.net/pmr/knifty-knitter-hobo-glove-pattern.html (374 links)
http://safeimail.com/banker-cold-cstrike-16/ (233 links) Just wanted to find out what my best course of action might be? Would changing hosts (or IP address) help in this situation? Thanks!0 -
Removing 301 Redirects
Is it safe to remove old 301 Redirects from an SEO standpoint and can 301s dramatically affect seo? Prior to switching our old domain over to our new domain, we had (and currently still do) tons of 301 redirects, because of optimizing our file names and structure. Then our old domain was redirected to our new domain in the same redirect file. So that being said, now that our new domain has been up and running for about 3 months, would it be safe for me to get rid of the old 301 redirects and redirect anything that was on our old domain to our new domains home page? This would clean up our redirects tremendously and I hope would help with SEO.
Technical SEO | | hfranz0 -
Why would you remove a canonical link?
Currently, my client's blog makes a duplicate page every time someone comments on a post. The previous SEO consultant told the developer to not put a canonical link directing it to the main blog post. Did taking out the canonical link result in these duplicate pages? My question is why would she recommend this action? Is it best to now add in the canonical link in or should we implement a 301 redirect or insert a index: no follow? Would adding a canonical link keep duplicate pages from happening in the future?
Technical SEO | | Scratch_MM0
