Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL in SERP: Google's stand
Months back, we can notice "keyword" will be bold and highlighted if its in the SERP URL. Now Google no more highlights any URLs even with exact match of keyword we search. Beside UI, Does this mean Google might devalued or reduced the importance of URL as ranking factor? We can see many search results match partially or completely in URL with search keywords.
Algorithm Updates | | vtmoz0 -
Can a page be 100% topically relevant to a search query?
Today's YouMoz post, Accidental SEO Tests: When On-Page Optimization Ceases to Matter, explores the theory that there is an on-page optimization saturation point, "beyond which further on-page optimization no longer improves your ability to rank" for the keywords/keyword topics you are targeting. In other words, you can optimize your page for search to the point that it is 100% topically relevant to query and intent. Do you believe there exists such a thing as a page that is 100% topically relevant? What are your thoughts regarding there being an on-page optimization saturation point, beyond which further on-page optimization no longer improves your ability to rank? Let's discuss!
Algorithm Updates | | Christy-Correll1 -
Ranking dropped with no page changes
My rank for a keyword went from ranking #1 to #22. The page grade for this keyword is A, there was no site structure changes. The only thing I can see is that tumblr and reddit and other sources are now listed for this keyword and it's difficulty went from the mid-low teens to 28%. However, even given that, I do not a see a reason for this keyword alone to fall so far. It was giving us a ton of traffic, in fact, most of our organic search results came from this term for nearly two months. And 2 weeks ago for no reason, we were pushed to page 3. Has anyone else had similar experiences how do you counter it, and what can we do?
Algorithm Updates | | mozmemberanon0 -
Reasons for a sharp decline in pages crawled
Hello! I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week. So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks. Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue? FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
Algorithm Updates | | Symmetri0 -
Ideas on why Pages Per Visit Dropped?
Week over week our pages per visit continue to drop. Any ideas on where to look to diagnose?
Algorithm Updates | | Aggie0 -
Is it allowed to put a word in all domains URLs to get higher in SERP?
Hello, What good or bad could happen if someone put the same keyword in all site's URL's? (i.e. I would be selling cars and my domain isn't included any word cars, so i put all of my pages in one folder like domain.com/cheap-cars/etc)
Algorithm Updates | | komeksimas0 -
Same page but appearing in Google with different titles
I have a page ranking on position 1 for a key phrase. The key phrase is the title of the page as well. I'll use a mock key phrase to aid my question - "Teeth and Gums" So the page is ranking number 1 for "Teeth and Gums" and "Teeth and Gums" is the meta title. However, I went ahead and did a new search adding an additional keyword to the original search. When I did a new search adding an additional keyword to the original search, Google has done something weird.. Let's say the search is "Dentistry - Teeth and Gums", Google has ranked my page again as number 1 but changed the title. The title in the search result is now "Dentistry - Teeth and Gums" How and why? It's kinda like Google PPC's keyword insertion but the title hasn't got anything weird like {KeyWord: Dentistry}. It's just "Teeth and Gums" Has this happened to you guys? Any ideas?
Algorithm Updates | | Bio-RadAbs0 -
Double Listings On Page One
I've been noticing a trend over the past month and a half. My sites that use to get more than one page listed in certain SERPs are now being adjusted. It almost looks manual but I know it is most likely a change in the algorithm. Let's say I had a SERP where my site was showing two different sub-pages in a single SERP at #4 and #6 are now having one page being pushed up to #3 but the other page is being pushed back past the first page. I'm not worried about penalizations or loss of value. I have been seeing this accross many of my client's sites. I just wanted to confirm that others were seeing it as well (so I'm not going crazy) and/or if Google has made any announcements or leaks regarding this shift. Maybe it's just my sites coming of age or something but I would love to be able to explain it more knowledgeably than with a "Google might be doing this". BTW - This is not effecting any of my Brand SERPs.
Algorithm Updates | | BenRWoodard0