Indexed, though blocked by robots.txt: Need to bother?
-
Hi,
We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files.
Thanks
-
Hi there!
What Google is telling you is that you are indexing URLs that you probably are not wanting to be indexed, or the other way around, that important pages are being blocked but indexed for other reasons.
If I might ask, why did you blocked through robots.txt those files?
There most 2 answers are:
1- Wanted to remove those from search results. If this is your case, you've solved only a part of the problem. What you should have done is (previously allowing robots to crawl those urls) apply noindex rules (keep in mind that can be set up in the HTTP header, as long as not html files cant have meta robots tag), then after a sufficient time block them in robots.txt.
_2- Optimize how GoogleBot (crawiling) time. _Being this case, then you've done it correctly and there is nothing to worry.Hope this help.
Best luck.
GR
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help Needed regarding DA and PA..
Hey there guys, I need help regarding the DA & PA on my site Criccraze From the previous few weeks, the DA PA of my website has gone down. Any fruitful suggestions from anyone, please? I would be heartily thankful.
Algorithm Updates | | Jashii80 -
Website have Caching/Indexing / Ranking Issue
Hi, My Website (https://www.v3cars.com) is not cached or indexed on regular basic from last 15 days. before this it was cached or indexed on regular basic. We are uploading fresh content on daily basic. Currently my new content is not ranked anywhere in Google even after cached or indexed. Please help and suggest. Sandeep - Love to Cars
Algorithm Updates | | onlinesandeep0 -
Training - How Google Crawls & Indexes Websites
Hi Does anyone know of any online training resources/webinars/training UK based that will cover the following for SEO: Why monitoring how search engines crawl and index content is important and how this can improve your SEO performance Using Google advanced operators to evaluate website indexation How to use log file data to gain insight into how search engines crawl and index content Techniques to control how search engines crawl and index content How search engines deal with JavaScript, common frameworks and SEO considerations I'm trying to develop my technical knowledge - I have always been more focused on content/KWD research/optimisation. Thank you
Algorithm Updates | | BeckyKey0 -
If we have all products on-site for indexing, do we get dinged by Google for not transacting on-site?
I am trying to do research on the SEO impact of having an off-site transactional website. For example, Pepsi.com lists all product information on their site but guides visitors to transact on Amazon or Walmart. What impact, if any, does guiding the customer to a separate transactional site have on SEO? In short, if we have all products on-site for indexing, do we get dinged by Google for not transacting on-site?
Algorithm Updates | | KaylaV0 -
Panda, Negative SEO and now Penguin - help needed
Hi,
Algorithm Updates | | mlm12
We are small business owners who've been running a website for 5 years that provides our income. We've done very little backlinking ourselves, and never did paid directories or anything like that - usually just occasional forum or blog responses. A few articles here and there with some of our keyword phrases for internal pages. Of course I admit we've done some kwp backlinks on some blogs, but our anchor text profile is largely brand names and our domain name and non keywords (excepting for some "bad" backlinks). Our DA is 34, PA 45 for our home page. We were doing great until last Sept 27 when we got hit by Panda and have been working on deoptimizing our site for keywords, we made a new site in Wordpress for good architecture and ease of use for our customers, and we're deleting/repurposing low quality pages and making our content more robust. We haven't yet recovered from this and now it appears we got hit May 22 for Penguin...ARGH! I recently discovered (hard to have time to devote to everything with just two of us) that others can "negative seo" a site now and I feel this has happened based upon results below... I signed up for linkdetox.com yesterday and it gives a grim picture of our backlinks (says we are in "deadly risk" territory). We have 83 "toxic" links and 600 some "suspicious" links (many are in malware/malicious listed sites, many are .pl domains from Poland, others are I believe foreign domains, or domains that are a bunch or letters that make no sense, or spammy sounding emd domains), - this makes up 80% of our links. As this is our only business, our income is now 1/3 of what it has been, even with PPC ads going as we've been hit hard by all of this and are wondering if we can survive fixing this. We do have an SEO firm minimally helping us along with guidance on recovering, but with income so low, we are doing the work ourselves and can't afford much. Needless to say, we are quite distressed and from reading around, not sure if we'll be able to recover and that is deeply saddening, especially from Negative SEO. We want to make sure we are on the right path for recovery if possible, hence my questions. We haven't been in contact with Google for reconsideration, again, no penalty messages from them. First of all, if we don't have a manual penalty, would you still contact all the toxic/malicious/possible porn looking sites and ask for a link removal, wait, ask for link removal, wait then disavow? Or just go straight to Google disavow? For backlinks coming from sites that are "gone" (like a message saying the account has been suspended), or there is no website there anymore, do I try and contact them too? Or go direct to disavow? Or do nothing? For the sites flagged as malicious (by linkdetox, my browser, or by Google), I don't want to try and open them on my browser to see if this site is legitimate. If linkdetox doesn't have the contact info for these - what are we supposed to do? For "suspicious" foreign sites that I can't read the webpage -would you still disavow them (I've seen many here say links from foreign sites should be disavowed). How do you keep up with all this is someone is negative SEOing you? We're really frustrated that Google's change has made it possible for competitors to tank your business (arguably though, if we had a stronger backlink profile this may not have hurt, or not as much - not sure). When you are small biz owners and can't hire a group to constantly monitor backlinks, get quality backlinks, content, site optimization, etc - it seems an almost impossible task to do. Are wordpress left nav and footer link anchor text an issue for Penguin? I would think Google would realize these internal links will be repetitive for the same anchor text on Wordpress (I know Matt Cutts said to not use the same anchor text more than once for internal linking -but obviously nav and footer menus will do this). What would you do if this was you? Try and fix it all? Start over with a new domain and 301 it (some say this has been working)? Just start over with a new domain and don't redirect? Thanks for your input and advice. We appreciate it.0 -
How to speed up indexing of my site...
Only 4 out of the 12 pages of my blog/site have been indexed. How can I ensure all the pages get indexed? I'm using a wordpress site, and I also wondered how could I speed the indexing process up (I have submitted a site map) Thanks!
Algorithm Updates | | copywritingbuzz0 -
"No Follow", C Blocks and IP Addresses combined into one ultimate question?
I think the the theme of this question should be "Is this worth my time?" Hello, Mozcon readers and SEO gurus. I'm not sure how other hosting networks are set up, but I'm with Hostgator. I have a VPS level 5 which (I think) is like a mini personal server. I have 4 IP addresses, although it is a C block as each IP address is off by one number in the last digit of the address. I have used 3 out of the 4 IP addresses I have been given. I have added my own sites (some high traffic, some start-ups) and I've hosted a few websites that I have designed from high paying customers. -one man show, design them, host them and SEO them With the latest Penguin update, and with learning that linking between C Block sites is not a great idea, I have "No Followed" all of the footer links on client sites back to my portfolio site. I have also made sure that there are no links interlinking between any of my sites as I don't see them in the Site Explorer, and I figure if they aren't helping, they may be hurting the rankings of those keywords. Ok, so...my question is: "I have one IP address that I'm not using, and I have a popular high traffic site sharing it's IP with 5 other sites (all not related niches but high quality) Is it worth it to move the high traffic site to it's own IP address even though making the switch would take up to 48hrs for process to take affect? -My site would be down for, at the most 2 days (1 and a half if I switch the IP's at night) Is this really worth the stress of losing readers? Will moving a site on an IP with 5 other sites help the rankings if it was to be on it's own IP? Thank you very much ps- I can't make it to MOZcon this year, super bummed
Algorithm Updates | | MikePatch0