Meta Robots Noindex and Robots.txt File
-
Can we use meta robots noindex and robots.txt together?
-
Hi, let us take a scenario. test.html has been blocked using robots.txt file. In this scenario, the bots that respect the robots exclusion protocol will not crawl the page to encounter the page level robots noindex attribute. So does not make any sense to use both the page level robots exclusion method and server side robots.txt file.
Here you go for more:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93708
And here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
Best regards,
Devanur Rafi.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTML entity characters in meta descriptions
Is it okay to leave HTML entity characters, such as " in meta descriptions? Will search engines translate these appropriately?
Technical SEO | | ellenu0 -
Open Graph Meta Description...
Does my html meta description tag have to be the same as my Open Graph meta description? I'm having problems pulling through my meta description into Google SERPs and I wondered if its because my 'OG' data is not consistent? Thanks Guys, Kay
Technical SEO | | eLab_London0 -
Htaccess file help
Hi, thanks for looking i am trying (and failing) to write a htaccess for the following scenario <colgroup><col width="679"></colgroup> http://www.gardening-services-edinburgh.com/index.html http://www.gardening-services-edinburgh.com http://www.gardening-services-edinburgh.com/ | so that all of these destinations goto the one resource any ideas? thanks andy
Technical SEO | | McSEO0 -
Brand page meta tag
I have around 100 brands on my website (with 100 different pages.) Please suggest me best way to create meta tags for all brand pages.
Technical SEO | | Obbserv0 -
Robots txt
We have a development site that we want google and other bots to stay out of but we want roger to have access. Currently our robots.txt looks like this: User-agent: *
Technical SEO | | LadyApollo
Disallow: /cgi-bin/
Disallow: /development/ What would i need to addd or change to let him through? Thank you.0 -
Un-Indexing a Page without robots.txt or access to HEAD
I am in a situation where a page was pushed live (Went live for an hour and then taken down) before it was supposed to go live. Now normally I would utilize the robots.txt or but I do not have access to either and putting a request in will not suffice as it is against protocol with the CMS. So basically I am left to just utilizing the and I cannot seem to find a nice way to play with the SE to get this un-indexed. I know for this instance I could go to GWT and do it but for clients that do not have GWT and for all the other SE's how could I do this? Here is the big question here: What if I have a promotional page that I don't want indexed and am met with these same limitations? Is there anything to do here?
Technical SEO | | DRSearchEngOpt0