Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Affiliate Marketing in Australia
Hello Guys My website is BannerBuzz.com.au I'm Looking for Best Affiliate Marketing Websites In Australia and I've Researched and found the following Websites for the same. http://www.clixgalore.com.au/
Affiliate Marketing | | CommercePundit
http://www.jackmedia.com/
http://www.commissionfactory.com.au/ So I Need your valuable opinions on which website I can get maximum exposure in Affiliate Marketing. Please Suggest. Thanks in Advance. Thanks Prabhakar0 -
Can 302 chains (affiliate links) from "toxic" sources hurt you? Or are you "shielded"?
Hi, I'm going through some affiliate links, which send visitors to our website via a chain of several 302 redirects Some of them are relevant links, others perhaps not so much. I know that Google doesn't pass PageRank on 302s... But are they still considered valid links that pass, let's say, "reputation", "relevance", "link neighbourhood" kind of signals? Otherwise put, is a 302 similar to adding the "nofollow" attribute on a link? Sort of? Not at all? More succinctly put, should I be worried about "toxic" sources separated from us by 302 redirect chains? By the way, yes I recognize that (Google's 302 redirect chain handling aside) associating our brand with perhaps what some might consider spammy websites is not in general a good move; I'm concerned with the technical SEO implications here however. In fact, this technical information may very well help drive decisions/policies on where we allow our affiliate advertising to appear. Thanks 🙂 PS - The affiliate company by the way is cj.com if that helps
Affiliate Marketing | | ntcma0 -
How do you find Affiliate Links on your site that have not been nofollowed?
We've just signed up to an affiliate scheme because we were sending links to them because we thought their product was valuable to our users. So we now have to go through and nofollow all of these links over 100's of pages. Is there any way that do a crawl of the site to identify all links to a particular site and tell me what page they are on and whether they are nofollow/follow?
Affiliate Marketing | | Zippy-Bungle0 -
Any Ideas for SEO opportunities at an electronics convention (CES)?
Hello All, I'm currently helping someone that sells a unique 35mm slide scanner with his digital presence. For a quick explanation, it scans much faster than it's competition and targets photography/scanning businesses. The owner will have part of a booth at CES (large electronics convention) to display the product, and will also have time to network with others within this industry and many others. At last years convention, there was a good in-person response and a few articles written about the product, but that is about it. I would like to send him to the show with a more digital mindset, so I am wondering if anyone has any ideas for link-bait, promotions, or really anything that will help the website rank higher/get more traffic. We have tons of content ideas and other ideas that are "online" to drive traffic, but when it comes to a in-person event I am drawing a big old blank (but I feel there must be some opportunities that I am missing). If you have any ideas and would like to share, it would be greatly appreciated!!! Thanks! Eric
Affiliate Marketing | | EHuhn30 -
Is it a good idea to make HubPages? Or Squidoo lens?
Hello, I would like to promote the products in http://www.giftbasket4kids.com with Hubpage and or Squidoo.com. we are starting with Christmas Items. Please share your ideas as Should we use Hubs or Squidoo Presently he website is under reconsideration with Google for Penguin hit. We have lost most of the keywords on three landing pages. We do not have a lot of time before Christmas sales. How fast can we publish the pages? What are to do and what not to do list. I appreciate all your comments. Thanks in advance.
Affiliate Marketing | | giftbasket4kids0 -
Affiliate program in the education niche
Just curious if anyone knew of some good ones out there. Looking to try a couple out. Thanks!
Affiliate Marketing | | astahl110 -
URL Tags / Affiliate tracking help
1. If I give a sales rep a unique tag to put at the end of a URL for tracking purposes, will Google index the "www.mysite.com/service.html?unique-tag" version of a page and display it in search results? 2. Would a tool like Xenu be able to find that version of a URL? Reason for questions: If I give my sales people tags with their names in them, I don't want competitors poaching my sales people by search for all URLs on my domain. Follow up: are there any other potential issues that I should think about? Thanks!
Affiliate Marketing | | pbhatt0 -
Does an affiliate link bring the same SEO juice as a standard link?
I wonder if affiliate links, like the ones offered by Amazon Associates program bring the same SEO as would a link to the same page without this additional "ref=..."?
Affiliate Marketing | | maciek-0
