Meta robots or robot.txt file?
-
Hi Mozzers!
For parametric URL's would you recommend meta robot or robot.txt file?
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs.And what's the real difference between the two?
Thanks again!
Kay
-
No problem at all
-Andy
-
Thanks Andy!!!
-
Hi Kay,
If you want to disallow access to a page, then add the following to the Robots.txt file:
Disallow: /quickView
Then test this in Webmaster Tools.
If you want to tell Google not to index a page, then you need to do this at the page level using Meta Robots. However, don't do both (at least not at the same time). If you disallow access to a set of pages via Robots.txt and then at a later stage you Meta Noindex, Google won't see this because of the Disallow in the Robots.txt.
It really depends what you are trying to achieve, but it sounds like the Meta Robots is the way to go for you.
-Edit... here is an interesting read for you.
-Andy
-
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disavow File
After uploading a Google disavow file how long does it take to be processed? Before any trolls get going, not been doing anything dogy, looks like someone has been trying some negative seo on us.
Intermediate & Advanced SEO | | seoman100 -
SEO Friendly Files Redirected From Images
I have images (.jpg's) of products that when you click them redirect you to a .pdf's containing all the products' specs, patterns, colors, etc. These are 302 redirects that open on a different window when clicked on. Is there a way to keep these redirects and maintain SEO optimization? Any advice is appreciated.
Intermediate & Advanced SEO | | SuperiorPavers0 -
Google not displaying meta description
Hi, one of my clients is receiving the following error in SERP - "A description of the page is not available because of this site's robots.txt". The site is built on WordPress and I realized that by default, the settings were checked to blocks bots from crawling the site. So, I turned it off, fixed robots.txt and submitted the sitemap again. Since, then it's been almost 10 days, the problem still exists. Can anyone tell me what should be done to fix it or if there's a way to get Google to recrawl the pages again.
Intermediate & Advanced SEO | | mayanksaxena0 -
Noindex a meta refresh site
I have a client's site that is a vanity URL, i.e. www.example.com, that is setup as a meta refresh to the client's flagship site: www22.example.com, however we have been seeing Google include the Vanity URL in the index, in some cases ahead of the flagship site. What we'd like to do is to de-index that vanity URL. We have included a no-index meta tag to the vanity URL, however we noticed within 24 hours, actually less, the flagship site also went away as well. When we removed the noindex, both vanity and flagship sites came back. We noticed in Google Webmaster that the flagship site's robots.txt file was corrupt and was also in need of fixing, and we are in process of fixing that - Question: Is there a way to noindex vanity URL and NOT flagship site? Was it due to meta refresh redirect that the noindex moved out the flagship as well? Was it maybe due to my conducting a google fetch and then submitting the flagship home page that the site reappeared? The robots.txt is still not corrected, so we don't believe that's tied in here. To add to the additional complexity, the client is UNABLE to employ a 301 redirect, which was what I recommended initially. Anyone have any thoughts at all, MUCH appreciated!
Intermediate & Advanced SEO | | ACNINTERACTIVE0 -
Files blocked in robot.txt and seo
I use joomla and I have blocked the following in my robots.txt is there anything that is bad for seo ? User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/ Disallow: /mailto:[email protected]/ Disallow: /javascript:void(0) Disallow: /.pdf
Intermediate & Advanced SEO | | seoanalytics0 -
We recently fixed a Meta Refresh that was affecting our home page - But something still seems wrong. Any suggestions?
We recently fixed a meta refresh issue on our home page. Our store URL: http://www.ccisolutions.com had a meta refresh on it that was going to: www.ccisolutions.com/StoreFront/IAFDispatcher?iafAction=showMain The meta refresh is now gone, however there still seem to be some problems: Our IT Director has not been successful in trying to make www.ccisolutions.com/StoreFront/IAFDispatcher?iafAction=showMain 301 redirect to http://www.ccisolutions.com - so I believe we now have a duplicate content issue If you look at both of these URLs in OSE, you will see that www.ccisolutions.com/StoreFront/IAFDispatcher?iafAction=showMain is getting credit for almost all of the Internal Followed Links, while http://www.ccisolutions.com is getting all the credit for External Followed links. Why doesn't http://www.ccisolutions.com show the same number of Internal Followed Links? I realize this is more of a developer/webmaster question and would be very appreciative of any suggestions or advice. Thanks!
Intermediate & Advanced SEO | | danatanseo0 -
Blog not showing up when searching for exact post/META titles
I am working on a blog http://www.possessionista.com which is a very popular fashion blog. It is very well established with a 100% natural link profile and zero spammy stuff. The blog ranks #1 for random fashion terms like "kourtney kardashian cat eye sunglasses" and "emily maynard boots". The problem I am experiencing is that none of the actual titles of her posts or any of the content in the post results in her blog showing up if searched. EX: http://www.possessionista.com/2011/10/pippa-middletons-zip-jacket.html When you search "Pippa Middleton's Zip Jacket" on google her blog is nowhere to be found. Try searching allintitle:"Pippa Middleton's Zip Jacket" and she's nowhere to be found either. Even search "The other day, I met with my friend Kiran for our monthly mutual admiration society" on google and she's nowhere to be found even thoguh this is a unique snippet from her post. This post is already indexed and cached with the above mentioned details. i've also tested dozens of older posts as well. Same issue. You can actually do this to see a more clear picture: Do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown site:possessionista.com That will bring up her blog which means google recognizes that the phrase is in her META title. Now do a google search for: allintitle:Bachelorette Fashion: Episode 2 - Ashley Hebert Brown without the site: included. She does not pop up but other people do. I did find that she had a duplicate title tag for a few weeks, but I've fixed that. Her posts used to pop up #1 when you search the title, but now obviously not. I am kind of at a loss and have tried a bunch of options with no success. Oh, one other thing is that some people do scrape her content, but only a few like maybe 10 and they've always been doing it even when she used to rank for her own post titles. Have you guys experienced this issue? Do you have any ideas of how to fix it?
Intermediate & Advanced SEO | | modparent0 -
Best SEO META Description for forum topics
What would be the best SEO META Description tag for forum topics on a forum type website? I can think of a few options so far Snippet of first post. Title of the topic with templated trailing text Remove description tag completely Your thoughts and suggestions are greatly appreciated.
Intermediate & Advanced SEO | | Peter2640
