Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should you aim for Google to use your meta tags?
When updating meta titles and descriptions, I'm taking note of whether Google is displaying the set tag or changing it to copy from the page. Does this affect the ranking position if Google is having to change the tag? How much should I worry if Google is choosing to change every other page? Thanks!
On-Page Optimization | | Omar_aw0 -
The meta tags: Title and Description, showing unexpected results on google
When I type my company name on google "Navneet Gems", it shows a very different meta tag then what it actually is. How do I change this meta descrption when its non-existent on my homepage? The worst is, it is having a spelling mistake. We want to correct this.
On-Page Optimization | | Navneet.Agarwal20160 -
Google news rejection
Google News is always rejecting my application. I feel as if my site strongly fits the requirements yet they reject it all the time. My url is hiddentriforce.com Any thoughts?
On-Page Optimization | | Atomicx0 -
What does Google consider a "Duplicate Title Tag?"
Do the title tags have to be exactly the same, or can they have some of the same keywords but different context? Hypothetical example: Home Page = Raising a Kitten, Tips & Tricks for a Healthy Cat Sub-Page = How to Cat-Proof your Home when Raising a Kitten Since both title tags has "raising a kitten," "cat" and "tips" would this be considered a "Duplicate Title Tag" even though the pages have completely different content in them? Thanks in advance!
On-Page Optimization | | Scratch_MM0 -
Does Google penalize a page with the image tag with alt and without src?
Hi, I am curious whether Google penalizes a page with the image tag with a value in the "alt" attribute and without one in the "src" attribute? Would this count as stuffing? Sometimes you cannot put an image but you would like to get SEO benefit by having a keyword in an image?
On-Page Optimization | | Plivo0 -
Google Maps API as primary navigation
Is it okay for SEO to have a google maps api as the primary source of navigation? For example, have people find locations on a map instead of links to them. I'm wondering how/if Google views this method, kinda like how Google can't read images. Will Google realize that these pages are linked to from the homepage gmaps API?
On-Page Optimization | | terran0 -
Blog Comment IPs Seen By Google?
I have a page on a client's site for testimonials (a dental practice). The page is actually a post on a Wordpress install where customers can enter their testimonials as WP comments. In an effort to encourage more clients to give more testimonials I was considering setting up an iPad or other tablet at the receptionist's desk where patients would be able to enter their successes as comments on the page. If I made sure the patients all used unique names and emails in the Wordpress comments, would Google still see all the comments are from the same IP and view this as suspicious?
On-Page Optimization | | jargomang0