Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
Hi all, We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed. And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time? Thanks
Algorithm Updates | | vtmoz0 -
Do pages with canonicals need meta data?
Page A has a canonical to Page B. Should Page A have meta data values such as description, keywords, dublin core values, etc.? If yes, should the meta data values be different on Page A and Page B?
Algorithm Updates | | Shirley.Fenlason1 -
How do I figure out what's wrong with my site?
I'm fairly new to SEO and can't pinpoint what's wrong with my site...I feel so lost. I am working on revamping www.RiverValleyGroup.com and can't figure out why it's not ranking for keywords. These keywords include 'Louisville homes', 'Homes for sale in Louisville KY', etc. Any suggestions? I write new blog posts everyday so I feel there's no shortage of fresh content. I'm signed up with Moz Analytics and Google analytics
Algorithm Updates | | gohawks77900 -
Website dropping from page 1google uk
Hi all, Firstly let me stress I am not really SEO minded, I know the very basics and that is about it. I am a driving instructor in the UK and have had my website (Wordpress) on page 1 for about 3 years now round the position 3 mark but for the last few months it has been dropping and is now right at the bottom of page 1 so no doubt a few more days and it will vanish completely from page 1 of google.co.uk . I was wondering if someone could just have a quick look at the page to see if they can see anything obvious that wouldn't be seen by me! The search term is " Driving Lessons Worcester " and the page that has always show on page 1 of google is .. http://www.passlee.com/driving-lessons-worcester.html I also had another site on page 1 for about 2 years which was www.drivinglessonsworcester.com this has also vanished to page 5 over the last 3 to 4 months. What really hurts is I made a website for another local instructor using WP and with similar setup to mine and that is now showing as No.1 on page 1 of google.co.uk!! So how is it the website I did for him is doing amazing yet mine is dieing a death when they are setup the same way but obviously different content! As I said I barely know the basics , I am sure a lot of you are thinking " Just go research " which I know I should and no doubt will, but I just wanted someone to have a very quick look to see if there was anything obvious! Kind Regards Lee Francis
Algorithm Updates | | germinus0 -
AS we using the keyword related to our link but we are not listed in first page of Google search
AS we using the keyword related to our link but we are not listed in first page of Google search, but our competitors using the same keyword , they are listing in first page. how we can short this problem and get into first page on search
Algorithm Updates | | krisanantha0 -
Organic listing & map listing on 1st page of Google
Hi, Back then, a company could get multiple listings in SERP, one in Google Maps area and a homepage or internal pages from organic search results. But lately, I've noticed that Google are now putting together the maps & organic listings. This observation has been confirmed by a couple of SEO people and I thought it made sense, but one day I stumble with this KWP "bmw dealership phoenix" and saw that www.bmwnorthscottsdale.com has separate listing for google places and organic results. Any idea how this company did this? Please see the attached image
Algorithm Updates | | ao5000000 -
Which website? It's hard to get enough resources to develop both
I started my Whitby cottage website for years ago www.endeavourcottages.co.uk not knowing anything about SEO, but I now have very good rankings for the keywords on Google. I'm ranking quite well for Whitby holiday cottages on Google and I now list to other cottages in the area for my friends. The point is that I'm now starting a small letting agency and hoping to expand it now so I bought the domain http://www.whitby-holiday-cottages.co.uk/ and set up a WordPress site but it's not ranking are getting anywhere near the same amount of traffic. This domain name and website structure would be much more applicable now that there are more than one cottage to let. Whenever I get some free time I spend it doing SEO on my first website to improve its rankings and so which is the best strategy from here on. One try and develop the new cottages letting website at the expense of my main website, or forget about the new website are together and continue with my first website which was designed initially for just one cottage. It's hard to have enough resources to get both websites ranking high. Thank you very useful tips Alan Davidson
Algorithm Updates | | thepersuader0 -
Has anyone starting using schema.org?
On the 3rd June 2011 Google announced that they are going to start using Schema. Do you think this will change the way search engines find content, from briefly looking at Schema I'm concerned that the proposed tags could just turn into another keyword meta tag and be abused. Have you started using this tags yet and have you noticed a difference?
Algorithm Updates | | Seaward-Group0