Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Console returning 0 pages as being indexed
HI there, I submitted my site notebuster.net to Search Console over a month ago and it is showing 0 pages as being indexed under the index status report. I know this isn't right as I can see that in google alone by typing in (site:notebusters.net) there are 113 pages indexed. Any idea why this might be? Thanks
On-Page Optimization | | CosiCrawley0 -
Page title in Google search is defferent
Hello, Google changes the title of the main page only for my sites in this way: What I configured: My page title | my site name How it shows in Google: My site name: My page title If I checked some meta tags analyzer it will show my configured page title and also in Bing.com So what do you thing about it. Best Regards, Housam
On-Page Optimization | | anubis20 -
My main domain is missing in google, subdomain appears instead.
I have two SEO optimised pages in my website targeting different keywords www.example.com <-- main selling page (Pocket Guitar | Guitar Instruments)
On-Page Optimization | | kevinbp
www.example.com/index/ <-- 2nd selling page (Guitar Australia | Guitar Perth) Q: At first my website "www.example.com" is ranking on google first page. Suddenly it disappears and the link "www.example.com/index/" appears instead. No matter what i search, "Pocket Guitar | Guitar Instruments | Guitar Australia | Guitar Perth", the link www.example.com/index/ appears on the front page instead of www.example.com. What is happening to my main domain? Should i be worried?0 -
How can I make google display rich snippets when searching for my domain?
My domain is www.phraseexpander.com If I search for phraseexpander in google, I'm in the first position (as it should be) but google is not giving extra space to my result as it happens, for example, if you search for "fogbugz". Is there any way I can hep google index my contents (I currently have a site index that is created by Yoast SEO). Thanks a lot. Andrea
On-Page Optimization | | nagar0 -
What is the rule of thumb for adding links to your blog posts?
I have started keeping detailed records of all my blog postings. Is it ok to link to my own url? I make sure to link to another blog posting in each post, and link to sources as well. Thanks in advance for the advice!
On-Page Optimization | | rivercityransom0 -
Stumped on why Google is not showing main site pages anymore
Recently had sites homepage listing taken off first page for brand name search even though search term is not competitive. Does anyone have any ideas?
On-Page Optimization | | Luia0 -
Meta Description not displaying in Google
Hi Mozzers, I have a client that wants to change the way the meta description for some of his pages is being displayed. I've tried using the NOOPD and NOYDIR tags and its not worked. This isn't the client but perform this search in Google.ie - "accommodation newry daft" you get this result - http://www.google.ie/#hl=en&sclient=psy-ab&q=accommodation+newry+daft&pbx=1&oq=accommodation+newry+daft&aq=f&aqi=&aql=&gs_sm=e&gs_upl=11197l11712l2l12016l5l5l0l0l0l0l186l851l0.5l5l0&bav=on.2,or.r_gc.r_pw.r_qf.,cf.osb&fp=f5c640577bb5a285&biw=1600&bih=775 See how Daft.com (2nd results down) has the text "10+ items" in the description- my client has this as well as do many other competitors but its not present in the meta description tag. Anyone know how to get rid of this and get the good old meta descrition in the SERPs? Thanks BUsh
On-Page Optimization | | Bush_JSM0 -
Original content and the Google Panda Update
We are an online furniture store with about 1300 products on the site, and we mostly use the catalogue descriptions for the product. Recently I have been reading about One Way Furniture: http://ecommerceprnews.com/e-commerce_articles/2011/03/one-way-furniture-shifts-toward-quality-content-after-google-panda-update-201928.htm They are a big american online furniture which seemed to have lost about a 3rd of there traffic due to being punished in the panda update. Now it seems they are blaming the fact they use they use catalogue descriptions for the product (like us), and now they are going to rewrite all their product descriptions. We are a small company and rewriting 1300 products (meaningfully) is no small task. Looking at our own traffic we have taken a small slump since feb after about 18 months of general increased month on month traffic ( bar seasonal dips and boost), but we didn't have a "fall of the cliff" like One Way Furniture. But have been expanding into other areas (and there for new keywords), so we had expected to be increasing our traffic. So the question is, how important is unique content for all our products? is it worth all the time and money to fix all the pages? Our plan is to make sure our category pages (and there for landing pages) have unique content, would that be enough on its own, or are the product pages damaging the site over all?
On-Page Optimization | | eunaneunan0