Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do pages with canonicals need meta data?
Page A has a canonical to Page B. Should Page A have meta data values such as description, keywords, dublin core values, etc.? If yes, should the meta data values be different on Page A and Page B?
Algorithm Updates | | Shirley.Fenlason1 -
Is there a we to get Google to index our site quicker?
I have updated some pages on a website, is there a way to get Google to index the page quicker?
Algorithm Updates | | webguru20140 -
SERP's & Search Engine Differences
Hey, I recently modified my pages to conform more closely to the "A" page rankings for MOZ's on-page report card but saw declines in my keyword rankings. They keywords in question appear in my title tag, description, one image alt tag, either an h1 or h2 tag, and 4 times throughout the text of the document. I don't think MOZ would recommend these changes if it was seen as stuffing - is there any other reason why my rankings might have dropped by 1-4 positions? Also, does anyone know of a good article/book for Yahoo/Bing SEO? My Yahoo & Bing rankings are far below Google's in most cases. Any help would be appreciated! -Michael
Algorithm Updates | | Stew2220 -
Best Moz article on landing pages?
From what I understand, building landing pages to link back to sites is a thing of the past. I am looking for a good article that explains best current landing page practices (post Panda and Penquin). Any suggestions?
Algorithm Updates | | cschwartzel0 -
Should social widgets be the kind that shares/likes a page, or the kind that adds followers to a brand social page?
I'm wondering if the social widgets on my blog should create a share/like referencing the page or should the social widget create a follower to my brands page on a particular social network? Any ideas?
Algorithm Updates | | salesduke0 -
Why has my homepage been replaced in Google by my Facebook page?
Hi. I was wondering if others have had this happen to them. Lately, I've noticed that on a couple of my sites the homepage no longer appears in the Google SERP. Instead, a Facebook page I've created appears in the position the homepage used to get. My subpages still get listed in Google--just not the homepage. Obviously, I'd prefer that both the homepage and Facebook page appear. Any thoughts on what's going on? Thanks for your help!
Algorithm Updates | | TuxedoCat0 -
Gifts.com - Multiple domain pages in SERPs
One of our big natural search competitors for gift keywords is Gifts.com. We are competing for many keywords like "teen gifts", "gifts for him", "gifts for her". For many of these, the Google SERP has multiple Gifts.com pages on the first page. I have never seen more than one of our pages (uncommongoods.com) on a SERP page. Any clue how/why Gifts.com has multiple pages in search results ? Thanks!
Algorithm Updates | | znotes0 -
First page slot 1 spot doesn't equal global monthly traffic
We have a client who has occupied the top spot on Google for the past several months. According to the Google Adwords keyword suggestion tool, this keyword should generate around 5,000 Global and Local Monthly average visits. Trends show this keyword has consistent month over month traffic. The keyword search type is broad match. When we look at analytics, they're only getting 5 visits per month. Shouldn't the top spot get the lion's share of traffic? We've noticed this trend on several of our clients whose traffic doesn't really increase proportionate to the estimated search volume that Google returns in the Adwords tool. Ideas? We see the estimated traffic and tell clients, "Once we get you in that top organic slot, you'll get most of that traffic," but it's not correlating. Thanks so much.
Algorithm Updates | | GravitateOnline0