Correct use for Robots.txt
-
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
-
Lewis,
Thank you for the clarification!
-
Hi Eric
The guidance above means that Google when it looks to crawl your site won't its not a message to Google telling it never to come back.
Once everything is sorted, remove whichever approach you took to block the search engines and supply a sitemap to Google via the Webmaster tools. Your site should be crawled in no time after that.
Hope this helps.
-
Damian,
Thanks for your answer, that helps. If I add either one of the above items to my web page, and then remove it at a later date, will the search engines crawl and rank my site (at sometime after they are removed)? In other words, and I know this sounds stupid, but does a search engine see a Robots.txt file and never visit it again?
-
Hey Eric,
If you want to create and work on pages but you don't want them indexed you can add the following to the page in the section (the pages will still be crawled):
If you want NONE of your pages to be crawled (I.E the whole website) you can add the following to your robots.txt file:
User-agent: * Disallow: /
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving servers which means moving ip address but using the same URL. Would it harm the website's SEO?
Hello everyone, The server (in-house) which we use to host our website is a bit old. We are using CDN77 for our static content. What if I move all our website to the CDN service? meaning I use their storage capability and just have our url point to the IP address they provide. Would that hurt our rankings?
Web Design | | Edgar-Cerecerez0 -
What do you use for test rendering your dev site?
I'm redesigning our company ecommerce site and need to test render an infinite scroller to ensure that it is as SEO friendly as possible. My problem is that I cannot view it in Webmaster Tools since I am blocking the site from crawlers using robots.txt. I know I could simply unblock Google temporarily but I really would rather not make my dev site available to search engine crawlers.
Web Design | | bearpaw0 -
Does Using Magento With Multi Sites Affect SEO
We have a client who has 3 separate websites targeting the US, Australia, and the UK. Each of them has relevant ccTLD's such as: .com .com.au and .co.uk. Our client wants to use the Magento multi-site function so it combines all the stores (which are the exact same products) and merge it into one through Magento. These sites are all hosted in the US and had nothing to do with me haha! I understand Rand has mentioned on a video it would be best having the websites with ccTLD's hosted in that country (if budget permits), however in this case the budget doesn't permit us to go down that road. Has anyone any advice on this matter, has anyone did this before and had a lot of success with the SEO? At present there doesn't seem to be a lot of information about it and opinions are varied and sometimes divided. Any help would be very much appreciated guys Thanks, Matt
Web Design | | HigherthanSEO0 -
Should the primary navigation use keyword rich menu items?
There doesn't seem to be a definitive (or even authoritative) answer to this. The quandry is primary top navigation - the client wants to use 2-3 keyword length nav items, and I'm politely forcing him to use single word nav. The problem is that I have nothing to back up that assertion other than "this way looks better". Examples - Site is www.buybluewidgets.com. The client wants: About Our Blue Widgets Buy a Blue Widget Azure Widget Information Sell Your Blue Thingamajig My suggestion is: About Us Buy Learn Sell
Web Design | | AMSVansSEOTeam0 -
Basekit.com anybody using it?
I need help to built a mulitlanguage website. After posting questions, I most admit Wordpress with a premium template might be the options for all the features I need. Thoses are: SEO Frendly (custom links...html...Alt Tag...Metadata...)
Web Design | | BigBlaze205
Mobile Version
Great Template
Multilanguage Complete web site
Google Analytics
Blog
301 redirect Some one suggest basekit.com Do you thing it might be my solution? Thank you for your help. BigBlaze0 -
Are there any studies, statistics or measurable impact of using mixed fonts on landing pages?
Are there any studies, statistics or measurable impact of using mixed fonts on landing pages? One of our landing pages is using five variations of the Arial font where size, strength (bold, italics) and color all vary. One camp internally believes that this okay, whereas another camp wants to standardize the presentation where there's less variance (such as a heading as one and the body copy as another). Have you been through a similar trial or test in the past? I've seen some instances of a Marketing Sherpa study on the topic, but no real numbers to support one thing or another. I've attached an example image of our current LP. I have a lot of strong opinions on a number of items - but we're looking to have an immediate internal discussion on the font issue first. Thanks! GIzA8.jpg
Web Design | | eMagineSEO0 -
The use of foreign characters and capital letters in URL's?
Hello all, We have 4 language domains for our website, and a number of our Spanish landing pages are written using Spanish characters - most notably: ñ and ó. We have done our research around the web and realised that many of the top competitors for keywords such as Diseño Web (web design) and Aplicaión iPhone (iphone application) DO NOT use these special chacracters in their URL structure. Here is an example of our URL's EX: http://www.twago.es/expert/Diseño-Web/Diseño-Web However when I simply copy paste a URL that contains a special character it is automatically translated and encoded. EX: http://www.twago.es/expert/Aplicación-iPhone/Aplicación-iPhone (When written out long had it appears: http://www.twago.es/expert/Aplicación-iPhone/Aplicación-iPhone My first question is, seeing how the overwhelming majority of website URL's DO NOT contain special characters (and even for Spanish/German characters these are simply written using the standard English latin alphabet) is there a negative effect on our SEO rankings/efforts because we are using special characters? When we write anchor text for backlinks to these pages we USE the special characteristics in the anchor text (so does most other competitors). Does the anchor text have to exactly I know most webbrowsers can understand the special characters, especially when returning search results to users that either type the special characters within their search query (or not). But we seem to think that if we were doing the right thing, then why does everyone else do it differently? My second question is the same, but focusing on the use of Capital letters in our URL structure. NOTE: When we do a broken link check with some link tools (such as xenu) the URL's that contain the special characters in Spanish are marked as "broken". Is this a related issue? Any help anyone could give us would be greatly appreciated! Thanks, David from twago
Web Design | | wdziedzic0 -
Is there any difference in using an underscore vs. a dash in the directory portion of the url?
A friend who is a software developer asked this question regarding the directory portion of the url: Is it better to use dashes or underscores? I know in the domain name Matt Cutts recommends dashes, but what about the directory extension?
Web Design | | RobertFisher0