What to do about "blocked by meta-robots"?
-
The crawl report tells me "Notices are interesting facts about your pages we found while crawling". One of these interesting facts is that my blog archives are "blocked by meta robots".
Articles are not blocked, just the archives.
What is a "meta" robot?
I think its just normal (since the article need only be crawled once) but want a second opinion. Should I care about this?
-
Meta robots refers to the < meta name="robots" > tag at the page header level. This is usually the case when a blog is set up with an SEO program like All In One SEO for example, where you can manually set which content is blocked. It's common to block archives, tags, and other sections, in the theory that allowing these to be crawled could either cause duplicate content issues, or drain link value from the primary category navigation.
-
In general, there are two ways you can block crawlers from indexing your content.
-
You can add a Disallow entry to your robots.txt file
-
You can add a meta tag to your pages:
What you are saying in either case is "please do not list this content in your search engine".
In general, you would not want to block your archives. There certainly can be specific cases where you only want the public to see your most current content, in which case you can block it.
-
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Organic search traffic stats "leaking" into other channels?
Hi Everyone I have a website and am slowly getting to grips with SEO. Last week I enabled a new channel in google analytics which was "email" so I could track effectiveness of the weekly emails we send out. The good news is that a ton of traffic is now being assigned to the email "channel" in GA but my organic search traffic in channels is now down week on week. That feels odd as my overall traffic to the site is up, week on week. Does anyone have any experience of new channels coming on stream and canniballising old ones? Could it be that some of the traffic associated to organic search previously was actually coming from my email, I just didn't know it? thanks all!
Technical SEO | | NappyValleyNet1 -
Duplication in Meta Titles
Hi,
Technical SEO | | ChrisHolgate
In order to appease the Moz crawler we recently changed over 10,000 URL's in order to make our Meta Page Title less than 55 characters as it suggested. Unfortunately our rankings dropped dramatically pretty much overnight so I am getting the feeling that perhaps our titles are now just a little too concise and need elaborating on just a touch. Our competitors that rank well seem to use a small amount of keyword repetition. For example, whereas we may have:
Brother DCP-197C Inkjet Cartridges They will have:
Brother DCP-197C Inkjet Cartridges. Cheap Brother DCP-197C Ink. What are your opinions of the fact that: a) Their Title is over the 55 character figure that is suggested for displaying correctly in the SERPs.
b) The words Brother and DCP-197C are repeated in the title. The fact their title appears to be working better is almost enough to sway me but the competitors title just looks a little too spammy for me to make a sitewide change without asking some second opinions first. Cheers all!0 -
Meta tags
Hello, Does anyone know how long it takes for the meta descriptions to show up in Google? This because I just updated my meta descriptions for the whole website, but while moz toolbar is showing it correctly, google is still showing the old ones, even if i used the see as googlebot tool from webmaster tools. Thanks for a reply
Technical SEO | | socialengaged
Eugenio | Social Engagement0 -
Meta description and Meta Keywords
Hi, We are new to SEO and have some meta Q's Should Meta descriptions and meta keywords be different on every page? Is it bad to have the same meta data repeated on the site? If it has to be different does it have to be totally different per page of just slightly different? Should the description contain keywords is there an advantage to that? Thanks Andrew
Technical SEO | | Studio330 -
How do you mark a quote HTML wise?
Hi, As far as I know, in the past Italic was used to emphasize (similar use to Bold). Now I've seen people use Italic for quotations. Is that the correct thing to do for an entire paragraph or is it a problem for Google wise? Thanks
Technical SEO | | BeytzNet0 -
Robots.txt
should I add anything else besides User-Agent: * to my robots.txt file? http://melo4.melotec.com:4010/
Technical SEO | | Romancing0 -
Duplicate META Description
Two of my urls point to the same site. For example: 123.com and OneTwoThree.com. Because the pages have two separate url's and the same content, my Google Webmaster Tools account is showing this as Duplicate Meta Descriptions. Is there a way around this?
Technical SEO | | BradBorst0 -
Robots.txt and robots meta
I have an odd situation. I have a CMS that has a global robots.txt which has the generic User-Agent: *
Technical SEO | | Highland
Allow: / I also have one CMS site that needs to not be indexed ever. I've read in various pages (like http://www.jesterwebster.com/robots-txt-vs-meta-tag-which-has-precedence/22 ) that robots.txt always wins over meta, but I have also read that robots.txt indicates spiderability whereas meta can control indexation. I just want the site to not be indexed. Can I leave the robots.txt as is and still put NOINDEX in the robots meta?0