Does Google respect User-agent rules in robots.txt?
-
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site.
LinkSmart uses a bot to establish the linking.
The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking.
LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent.
I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore?
Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
-
Does Google respect User-agent rules in robots.txt?
Yes
I've seen googlebot ignore nofollow rules set at the page level.
Google honors the nofollow rules set at the page level. The issue is there may be other links on your site or elsewhere on the web that Google will find and follow those links.
Robots.txt is the absolute last means to use for blocking pages. You should not block a page with robots.txt unless you have exhausted all other options. A more appropriate method of keeping a page out of the index is the noindex tag. If you use the tag appropriately, Google will honor the tag.
-
Hi,
I would advise to block the directories which the files sit in in robots.txt, over adding no index tags to specific pages.
Yet then this would also leave these pages to not be indexed by Google, other search engines and also this Link Smart software you are referring to.
The thing is if you add a no index tag or if you add a robots .txt block to pages it will also block all search engines too.
So yes their is some risk involved, you have to do things carefully around this area.
Kind Regards,
James.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have updated title 4 days ago but still still showing old title and description on Google serps, How to resolve it?
I have updated the title tag but not showing, Please have look at the view source for this website- https://m.yolobus.in/ I want to show this title and description- <title>Online Bus Ticket Booking | YoloBus India</title> But showing the wrong title and description on google SERP- Title - YoloBus :: Home Description - Delhi Lucknow; Lucknow Delhi; Delhi Gorakhpur; Varanasi Lucknow; Gorakhpur Delhi; Delhi Delhi; Bangalore Bangalore; Manali Manali; Chennai Chennai 7mHsdmu
On-Page Optimization | | AnkitS.19900 -
Metadescription not being pulled by Google? Yoast v SmartCrawl?
Hey guys, For whatever reason, Google isn't pulling the metadescriptions I've provided for a wordpress site I'm working on. We had both Yoast and SmartCrawl installed, so I thought maybe they were confusing Google and deactivated Yoast. Unfortunately, that didn't fix the issue. Instead of using the text I've plugged into SmartCrawl, Google is just using snippets from the blog posts... And it's happening for every single post, leading to a huge uptick in metadata issues in moz. Any idea how to fix this?? Thank you!
On-Page Optimization | | laurendavidson0 -
Google rendering mobile to the right
So the website is displaying correctly in mobile devices and online tools. However, when using Google Page Speed Insights mobile or fetch as mobile in Search Console the page always loads as if it has moved 50% of the screen to the right - so whitespace on the left then 50% of the page in on the right side. I've been ignoring this. The site loads fine in devices and i've put it down to a glitch in Google, but at the same time it's been bugging me. Has anyone else seen this and should I ignore?
On-Page Optimization | | MickEdwards0 -
Do these items affect Google ranking or Quality Score?
Hi community, After my first crawl of a site that I'm working to improve the SEO on, I find that I have about 500 issues regarding missing Alt Text and Title text for images on my site, as well as about 175 issues with regard to duplicate meta description, missing meta descriptions, and too short/too long meta descriptions. My client is not sure it matters to fix these items and only wants to do so if they have an affect on Google ranking or Quality Score. Does anyone know? Thanks!
On-Page Optimization | | gataninc0 -
Can I just replay a backup within wordpress to regain my google ranking from yesterday?
Hi, yesterday I had a wonderful ranking for the website I do SEO for. The site is pretty still new (online since about one week). Around 11 am yesterday we ranked pretty good and challenged some local sites within the insurance business. Then I made a mistake and stuffed a part of a mail (full of keywords and comments) that was sent to me into an alt-tag of an image. Unfortunatelly it took me around 2 or 3 hours to notice. Before fixing the issue I made some other SEO changes and the site dropped on different keywords on google. Now it practically disappeared from the results… 😞 Does it make sense to just play back a backup from around 11 am yesterday or will google penalize so much going back and forth… (Meanwhile title-tag has been changed, an some alt-attributes, one of three H1s on the page to make it fit more to the title) Best regards Marc
On-Page Optimization | | RWW0 -
Ajax url returns an error by google. Is there another way besides creating a HTML version?
We trying to find out if there is anything to make it so google does not keep returning errors cuase of our ajax urls. Is there any other option besides creating it all in a HTML format for google to read? Any tips or help would be great!
On-Page Optimization | | DoRM0 -
Wrong sitelinks & landing pages in Google
I've recently launched a well-optimized website with good-content category landing pages and then I've added a blog to the website (as supporting content to the landing pages, the only links pointing to the blog are from the category landing pages) What happened is that Google is now using the Blog pages as the site - sitelinks and also as the landing pages for most keywords I only have inbound links to the reg. landing pages and none to the blog, how do I get Google to change that? I know I can demote sitelink URL's in webmaster tools, but would that help me with getting the right sitelinks, it sure wont help much with the landing pages Thanks
On-Page Optimization | | Plorex
-J0 -
Paid CTR Vs Organic CTR for high ranked terms - say average position 2 in Adwords and average position of 4 in Google SERP
Consider a situation where we are getting 5000 impressions for a term in Adwords and Bing/Yahoo and the same term with the same landing page is ranked within top 5 positions of Google and Bing search. If we get 2.00% CTR in Paid - Adwords and MSN average, What will be the acceptable Organic CTR - which is available in webmaster tool? Apart from Title and Description, what are the other areas need improvement to increase Organic CTR?
On-Page Optimization | | gmk15670