Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Site's Name Not Ranking in Google
Hey all, I've seen a few posts like this. But I wanted to start a new thread in hopes I may find the underlying issue. I've had my site: http://www.ctrl-alt-success.com for about 2 years. Recently I've started really adding a lot of content to it. (about 2-3 posts a week). I get zero organic views which is fine as I know it's still in the beginning. But here's my main question. If I type "ctrl-alt-success" into google. I get some site that shows up. "ctrlaltsuccess.com" I've been looking at this issue forever. That site has been "coming soon" for nearly 2 years. lol My site doesn't even show up on the first 10 pages of google. However in Bing and Yahoo it ranks on the first page. What could my site be doing wrong that it's not even ranking for the exact domain name? Keep in mind, if I google "ctrl-alt-success.com" my site comes up fine. Any help would be appreciated, thanks!
On-Page Optimization | | Ctrl-Alt-Success0 -
Does Google penalize a page with the image tag with alt and without src?
Hi, I am curious whether Google penalizes a page with the image tag with a value in the "alt" attribute and without one in the "src" attribute? Would this count as stuffing? Sometimes you cannot put an image but you would like to get SEO benefit by having a keyword in an image?
On-Page Optimization | | Plivo0 -
I've just manually edited all the page titles and meta descriptions on a site, when will this show in Google results?
I've just manually edited all of the page titles, meta descriptions and optimised the copy on a client's site. I submitted this for a new crawl on Google via Webmaster Tools but when I do a Google search the old versions are still showing. Will it still take a few weeks for the new versions to show even though Google has crawled it via Webmaster?
On-Page Optimization | | aoifep0 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | | bricerhodes0 -
Adding Google Authorship To Wordpress Category Pages
How can I add google authorship to my wordpress category pages? I use the 'google author link' plugin which adds it to posts and pages but not categories?
On-Page Optimization | | SamCUK0 -
Can Rankings in Google differ so much from computer to computer.
I was telling my friend via facebook to go on my website, I told him to search 'nightlife forum' in google. To which, I believed it was 11th, top of second page. On his computer, its currently ranking at 1st place is it possible to have a difference of 10 places? even though he lives in the same city as me. Would be good to see what it ranks on your computers too google "nightlife forum" look for www.talknightlife.co.uk (don't get confused with the .COM one out there) Cheers Guys
On-Page Optimization | | Lukescotty0