Looking for a way to crawl and test validity of affiliate links at scale. Ideas?
-
Hey all,
I'm on the hunt for a service that will crawl our affiliate links and let us know when they return an error. I need to know that the last URL in the chain is returning a 200 over thousands of pages and links on a continual basis. The hitch is that most crawlers like Screaming Frog will return all of our links as working because it's only testing the first step, and this really requires a cloud solution anyway. Anyone happen to know of something?
Edit for clarity's sake: I need something to check entire redirect chains in bulk that isn't a Wordpress plugin, isn't a website where you plug in a URL and it cuts you off after the first 100 results, and has the ability to crawl the site and provide reporting on a continual basis.
-
So, digging deeper into this issue, it turns out that we can't rely on status code as a reliable indicator of page validity for affiliate network URLs. Most of them turn up with a 200. Looking at possible custom solutions from affiliate compliance monitoring services now. No one seems to be doing this thing that I need to do, but it sounds like a great business idea for someone with coding experience and more entrepreneurial spirit than I've got. Just gonna throw that out there.
-
Oh, nice idea, running Screaming Frog on a server.. I passed your notes along to our Dev team, we'll see what happens.
-
Hello Rebecca,
Screaming Frog is capable of a lot when set up properly. Mike King has a great post about running it on Amazon Web Services (AWS) so it could fit your cloud solution request. Have you checked out Seer's guide to doing almost anything with Screaming Frog? Do you have "Check External Links" checked? Also "Always Follow Redirects" should be checked. And you can set the "Max Redirects to Follow" to whatever you like.
If that doesn't work, have you tried Deep Crawl?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Affiliate website redirection concern
First of all i want to say hi to everyone here since im a new guy! The issue: My website is a price comparison website. I create products and inside these products i list several shops with their priced offers from the lowest to the highest. When a customer clicks the "shop now" button to go to the desired shop a new window pops up telling the user that he is being redirected to that shop. Check the website here and click "shop now" to understand what i mean: https://www.pccdkeys.com/product/buy-world-of-warcraft-legion-cd-key-for-battlenet/ This pop up page is a custom .php called redirect.php. In order to create a unique url for each shop, we add parameters in every URL like this: /redirect.php?id=5030917189616&merchant=CDKEYS&game=4117. The redirect.php right now is under this setting: custom header 301 without destination URL (which probably is wrong) it is noindex, follow in meta robots the parameters: **id, merchant, game **are set to NO URL's in URL parameters in webmaster tools. If i remove the 301 redirect from the header it will lead to 200 OK but then its a soft 404 error because the page is small and for sure i dont want 8000 of soft 404 errors because google phantom will destroy me. Another setting was: No custom header so it was 200 OK noindex, nofollow (which was for extra protection from the disallow) disallowed in robots.txt Thi setting kept giving soft 404 errors in audits. I also had it with this setting: custom header 301 without destination URL meta robots in index, follow without disallow in robots This setting gave me Not followed errors in google webmaster tools. The question: Which is the proper setting for this php page? Maybe a combination of the above? Thanks in advance and sorry for the long post.
Affiliate Marketing | | dos06590 -
Setting up external link goals in GA
I have recently switched all my travel affiliate links to TripAdvisor using their javascript code that converts the natural links (direct links to TA URLs) to affiliate links. I have set up goal tracking in GA to try to put a monetary value on clicks through to these links (based on the known conversion rate to commissionable bookings) but GA is over-reporting the goals by a very large margin. I'm wondering what I've done wrong. The first screen shot below is of the goal settings I used in GA. The second screenshot shows that GA is reporting 7125 goal completions since I set up the goal on 7 August. The third screenshot shows the actual number of GA events recorded on outbound clicks to TA URLs in the same period, which is just 97. So the goal completions being recorded far exceed the actual goal completions, but I'm not sure why. I know I've done something wrong but as this is the first time I've used goals, I'd appreciate some help to see what I've done wrong. Thank you. 3f7dc3de6f9f1373d47994144a5cb016 56aa39130a7d54b07d8f2ce9cf1d7fb0 35ee827275928ca8c4d4c941f0672ef8
Affiliate Marketing | | Gavin.Atkinson0 -
What is the right way to link to your main site?
Hi, we have a system for tracking the leads that comes from a specific affiliate website. each affiliate has a unique tracking code. not only for affiliates that work with us but also for the SEO team has an affiliate tracking code so our bosses can track the leads and traffic that come from the activity of the SEO team. this means, links towards our website look like: www.mydomain.com/?t_src=campaign&t=AFF&t_cre=links&A=371 www.mydomain.com/?A=371 I have few questions about this: 1. This is the right way to link to my company site? How dose google crawl those link? It's can harm the link value? 2. What is my option to show my boss all the traffic / leads that our seo team brings to the site trough Google Analytics? Hope to get your support. Thanks in advance.
Affiliate Marketing | | JonsonSwartz0 -
Tweeting affiliate links...?
I work in the music business & am published music producer. I'm an authority in my niche. Yada yada. I have a website that sells products to other music producers. I recently set up an affiliate partnership with someone who is also in the music business but on the management side. She is not an authority to my customers as far as music making goes. But she does have a huge following of people who are EXACTLY who I am targeting. She has been tweeting my affiliate links. The traffic has been great, but the purchase response has not been. I've had much better responses from reviewers and other music producers who have tweeted my links. I was just wondering if anyone had any tips for the most effective ways of tweeting affiliate links? Thanks!
Affiliate Marketing | | PedroAndJobu0 -
In what ways would an affiliate site be okay with Google?
I know Google has slammed affiliate sites hard. But affiliate marketing can be a real business and had an idea for one, but I would like to know if ALL affiliate sites are looked at as evil in Google's eyes, or if it's just those with thin and duplicate content in the product descriptions. I know they want brands to come up ahead of affiliates of that brand, and that makes sense. If I sold Acme Widgets products, Acme Widgets itself should come up ahead of my site. But what if the site sold widgets in general, with the term widgets not a branded keyword? If all product descriptions are unique and well-written content, on a site that is high quality, would it still be downgraded by Google just because there are affilate links in it? I guess overall the question could be boiled down to, are affiliate sites hit because they have horrible content or because they have links to affiliate programs in them?
Affiliate Marketing | | bizzer0 -
Affiliate System for my Website
So, I like to implement one affiliate system on my website and to generate URL for every online marketing campaign. because i like to know when someone has ordered it from Organic, and when from any Google Adwords Campaigns etc.. What do you suggest me to implement?
Affiliate Marketing | | leadsprofi0 -
Gaining juice from affiliate links
Afternoon Mozzers, We run a basic affiliate program whereby we distribute banners and tools to affiliate sites who then link back to our homepage via these, often through a tool that incorporates an HTML link with our primary keyword term embedded. This banner/tool/link heads to our homepage, and will then incorporate that particular affiliate's unique code in the URL, eg. www.example.com/1A-265-GNH Seems pretty standard practice so far? So, considering this, that we include our primary keyword term in HTML in some banners, some also link to our homepage via an embedded image (and others then just work off cookies as per usual), can anyone hazard a guess, or know for certain, the beneficence of this as a link building means? I ask in terms of trying to determine whether to continue punting our tools and banners as a decent link building opportunity, or concentrate more on other link building means. Hope I've been clear enough, but am happy to elaborate if need be. Cheers
Affiliate Marketing | | Martin_S0
