Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
-
Hi all,
We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed.
And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time?
Thanks
-
Hi vtmoz,
The most mandatory way to prevent any page to be indexed is by using a meta robots tag with a _noindex _parameter.
Then using robots.txt will help to optimize your server resources and is a way that prevent google to crawl any new page that do not have the meta robots tag.And yeah, its very common to have indexed pages even the robots.txt file blocks the entire website.
If what you are looking for is to remove from index the pages, follow this steps:
- Allow the whole website to be crawable (or at least that specific pages/section) in the robots.txt
- add the robots meta tag with "noindex,follow" parametres
- wait several weeks, 6 to 8 weeks is a fairly good time. Or just do a followup on those pages
- when you got the results (all your desired pages to be de-indexed) re-block with robots.txt those pages
- DO NOT erase the meta robots tag.
Hope it helps.
Best luck.
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How often should I update the content on my pages?
I have started dropping on my rankings - due to lack of time after having a baby. I'm still managing to blog but I'm wondering if I update the content on my pages will that help? All my Meta tags and page descriptions were updated over a year ago - do I need to update these too? We were ranking in the top spots for a good few years, but we're slowly falling 😞 Please give me any advice to keep us from falling even further. I have claimed all my listings, and try to add new links once a month. I share my blog to all social sites and work hard to get Google reviews, we have 53 which is higher than any of our competitors. Any other ideas? Have I missed something that Google is looking for nowadays? Many thanks 🙂
Algorithm Updates | | Lauren16890 -
Delay between being indexed and ranking for new pages.
I've noticed with the last few pages i've built that there's a delay between them being indexed and them actually ranking. Anyone else finding that? And why is it like that? Not much of an issue as they tend to pop up after a week or so, but I am curious. Isaac.
Algorithm Updates | | isaac6630 -
Ranking dropped with no page changes
My rank for a keyword went from ranking #1 to #22. The page grade for this keyword is A, there was no site structure changes. The only thing I can see is that tumblr and reddit and other sources are now listed for this keyword and it's difficulty went from the mid-low teens to 28%. However, even given that, I do not a see a reason for this keyword alone to fall so far. It was giving us a ton of traffic, in fact, most of our organic search results came from this term for nearly two months. And 2 weeks ago for no reason, we were pushed to page 3. Has anyone else had similar experiences how do you counter it, and what can we do?
Algorithm Updates | | mozmemberanon0 -
Website dropping from Page 1 pos 5 to no ranking and then back again?
Hi all, We have a very odd occurrence with a client of ours. It should be noted that they had a penalty recently removed about 2 months ago after much work from our company. Recently they started appearing back on Page 1 Google for a semi competitive keyword term. We were very happy with this and so was the client. The the ranking improved with our work to position 5, which was excellent. Unfortunately what has been happening is they have been dropping out of the rankings completely for this semi competitive keyword for a few days and then reappearing in the same position. The client is checking daily and has noticed. I thought this is just a 'hangover' from the Google penalty and perhaps a one off occurrence, but it has happened about 3 or 4 times now and seems to be happening every couple of weeks. Can anyone shed some light on this behavior? I have checked Webmasters Tools and everything is fine. Thanks Jon
Algorithm Updates | | Jon_bangonline0 -
Privacy page ranking above home page in serps
I'm using OSE to try and get some clues as to why my privacy page would rank higher than my home page. Could anyone help me figure out which metrics to review to rectify the issue? My key word is: Mardi Gras Parade Tickets The url that is ranking is <cite>www.mardigrasparadetickets.com/pages/privacy</cite> I'm happy to be ranking in the top 3 for the keyword, but I'd rather hoped it wouldn't be my privacy page. Any help would be awesome, Cy
Algorithm Updates | | Nola5040 -
Is using WPML (WordPress Multilingual Plugin) ok for On-Page SEO?
Hi Mozzers, I'm investigating multilingual site setup and translating content for a small website for 15-20 pages and came accross WPML (WordPress Multilingual Plugin) which looks like it could help, but I am curious as to whether it has any major international SEO limitations before trialing/buying. It seems to allow the option to automatically setup language folder structures as www.domain.com/it/ or www.domain.com/es/ etc which is great and seems to offer easy way of linking out to translators (for extra fee), which could be convenient. However what about the on-page optimization - url names, title tags and other onpage elements - I wonder if anyone has any experiences with using this plugin or any alternatives for it. Hoping for your valued advice!
Algorithm Updates | | emerald0 -
Has anyone notice their website page rankings in Bing and Yahoo slipping lately?
My websites for the most part rank higher in Yahoo and Bing than in Google. Now, ranking within those search engines (Yahoo/Bing) are settling in to where my websites rank in Google. Any thoughts? Anyone else experiencing this? Thank you for your time!
Algorithm Updates | | APICDA0