Blocking external links in Robots.txt - need advice on Best Practice
-
I look after an affiliate site that is doing quite well in the search rankings. We've been doing a review of our practices and one thing that has cropped up is our robots.txt.
In it, we disallow Google from crawling external links. This used to be best practice in the affiliate industry a couple of years ago, but I wanted to know if this is still the case, and what the implications are if we were to:
a/ leave it as is?
b/ allow crawling?
Thanks in advance.
-
I'm not sure how you're blocking Google from crawling external links in the robots.txt file--typically you only block them from crawling internal pages on your site.
If you're using a script, though, to track the clicks on external links and that script is running your site (and you're blocking that script in robot.txt), then that still should be fine. You may want to add a "nofollow" tag on those links, though, so you don't end up passing link "credit" or "link juice" to those affiliates (unless you want to do that?).
As far as external links go, though, it's typically okay (and expected) that you link out to other sites.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My affiliate links are being crawled by Google. Is this a bad thing? Also, should I add rel: nofollow to affiliate links?
Hi All, I have an issue on Google Search Console. It seems all my affiliate links are being crawled. Is this a bad thing? The reason I found out is that all but one link was flagged with a warming. It goes on to say that it was blocked by robots.txt. The strange thing is that I've not set any links to be blocked in robots.txt. Also, should affiliate links be nofollow? I'm asking because I was recently sent this article by Yoast on how to properly cloak my affiliate links without using a plugin. https://yoast.com/cloak-affiliate-links/ In the method, they apply the nofollow and noindex rules to the links. I'm just a little confused on how to go about affiliate links the SEO friendly way.
Affiliate Marketing | | nomad_blogger0 -
Person using expired domain and its links to drive traffic
Hi, I know about people using expired domains to drive juice to their primary site but what about people using AN expired domain as their primary site (totally changing that site into a trashy affiliate-marketing vehicle)? The site I'm looking at is thegunzone.com. It has, according to Semrush, almost 38K links. It used to be a legit 17-year-old firearms hobby site, and this is what it originally looked like: http://web.archive.org/web/20120213184627/http://thegunzone.com:80/ Here is its last page before it closed and the domain purchased by the affiliate marketer: http://web.archive.org/web/20170315084035/http://www.thegunzone.com/ It closed around February of 2017, and some affiliate marketer bought it and all its backlinks. However, all those backlinks, which were previously to various articles, are now directed back to those articles (which don't exist anymore) but the homepage, including Wikipedia links. Here's an example: https://en.wikipedia.org/wiki/Polygonal_rifling At the bottom, in the 7th Reference, there's a link to an article called " "Learning About Shooting . . ." but if you click on the original link, it just goes to thegunzone.com homepage. Again, the site's totally different. And there are just thousands of such backlinks to former articles that don't exist anymore but are redirected to this schlocky site's homepage (and it's passing its juice through too). My question is this: this cannot be kosher with Google backlinking policies, right? Is this prevalent on the internet? Why hasn't thegunzone.com been found out and its rankings penalized yet? And how do I report him? I see tons of other sites using this basic strategy too on search results with various hunting keywords. (Disclosure: I do own a hunting/firearms blog, but I don't do any backlinking at all.) Any help would be sincerely appreciated.
Affiliate Marketing | | HandyWoman1 -
Affiliates website, I need webpages linsk to study the topic
Kindly Provide me the best links where I can study about Affiliates websites, How to create them? there SEO? Content etc.
Affiliate Marketing | | bondhoward0 -
Low value link building to sitemap.xml
During some competitive research recently I discovered one of my clients competitors sites had an interesting backlink profile. Looking at the top-pages report in Open Site Explorer the home page was the #1 page (as you'd expect) with 2.5k links from about 500 linking root domains The second page was the sitemap.xml (~1.5k links, 400 linking root domains) and the third was their /feed page (again, ~1.5k links, 350 linking root domains). Links to these two pages aren't something that would happen naturally (particularly the sitemap.xml). There's a whole load of evidence for nasty low quality link building such as over-optimised keyword rich anchor text, comment spam, and even some blog/article based link networks. It's a pretty nasty niche with lots of cut-throat affiliate marketing. My guess here is that someone may have made a mistake using an automated link building too, but I'd be interested in what you might think? Have you seen this before? (Sorry, I can't reveal the domains in question as I'm bound by an NDA.)
Affiliate Marketing | | DougRoberts0 -
How to flag inbound affiliate links to Search Engines
Afternoon Mozzers, I was chatting to our VP of marketing earlier about one of our sites with a somewhat unhealthy looking link profile. This is primarily caused by the sector its in and the fact that lots of low quality lead generators / affiliates operate there, sending traffic to us in return for payment on accepted leads. Now, he recalled that there is a way to mark these inbound links, through a URL parameter he thought, as being affiliates and therefore should be ignored as we don’t want them misinterpreted as an attempt to manipulate our rankings. I have been doing a spot of research but can't find a straight answer. heres a few articles i looked at: blogaid, Yoast, Webmaster world and the Moz Q&A.The problem is all these are from the perspective of a site linking out and acting as an affiliate, so all deal with that page not losing PR. The two methods i have derived from this boil down to: No follow the links Use an intermediary redirect page and script which you can block from robots.txt So, back to our case - Are there any ways we can signal to Search engines, as the destination site, that these links are from affiliates (such as this URL parameter our VP had vague recollections off) or should i just get in touch and ask the sites to make them Nofollow? Thanks, Tom.
Affiliate Marketing | | Sarbs0 -
Need help about facebook fanepage!
I have a fanepage on facebook , some time have other fanpage go to my fanepage and spam..., now i want protect for they can't look at my fanpage :(( Who can help me thank very much! [link removed by staff]
Affiliate Marketing | | ITVSEO.COM0 -
SEO and Affiliate Links
Hello, We run a travel related website and we started to run our own affilate newtwork to promote the sales of our products. At the moment the affilate links pont to a spacific affilate url in order to tack conversions : abcweb.com/affiliate/nameqaz I'm wondering how is the best way to run a private affilate programm considering SEO: Is there a way yo benefit from those links ? What are the best strategies to do this? If yes, Is there any benefit from redirecting 301 those links to the original page (the one accesible to google and the one we want to rank for) or is it better to use a king of canonical method. Thanks a lot for sharing you experiences , giving your opinion and indicate resources. Best Regards Daria
Affiliate Marketing | | stereo690 -
Passing link juice via aff links?
Hi All, I know there was a recent post on this subject but I'm wondering if someone could take a look at these links and tell me if there is any SEO value in them at all and if not, what would be a way to improve them that might not be too much trouble for the affiliate? This URL: http://www.premiermodelskin.com/the-products/blemish-treatment has a Purchase button that passes product data (price, quantity, etc) directly to the basket of the host site (the site we want SEO benefit to). Using a form method to this URL: <form method="GET" action="<strong>http://www.monushop.co.uk/products/premier/blemish-treatment.html</strong>"> Qty <select id="add" name="add"> <option value="1" selected="selected">1</option> <option value="2">2</option> <option value="3">3</option> <option value="4">4</option> <option value="5">5</option> <option value="6">6</option> <option value="7">7</option> <option value="8">8</option> <option value="9">9</option> <option value="10">10</option> </select> 15ml £16.25 </form> My question is, does G see that form GET action as a followable link? If not what would be a better method? Any feedback much appreciated.
Affiliate Marketing | | lovealbatross
Cheers
J0