Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing links from rubbishy 'blog' sites
I need to remove around 800 bad links, probably about 500 domains as a very rough estimate. These were built by a previous link building company. Here some example domains: http://globalweddingblog.com
Link Building | | Coraltoes77
http://theweddinginsider.net
http://www.couturefashionissues.com
http://www.topfashionlabels.com
http://weddingworldnews.com
http://www.savingsdistrict.com
http://bestfemalesblog.com
http://mylatestfashion.com
http://lastfashion.net
http://womansonlineblog.org I have already tried emailing a hundred or so with a manual link request - with zero outcome. Hardly surprising when you consider the types of sites they are. I've had a quote for a link removal service, but I'm not sure if it's wise to pay someone to do this work - not sure what resources/tools they would have above and beyond what I can access and there could be increased risk. Any advice?0 -
Back linking to t foreign sites
One of our major competitors seems to be linking to an Asian speaking website that has a blog/product review formant. I was wondering how they are achieving this. Are there any non English /Asian sites worth submitting to and what is the best way to go about this if English is the only language you speak! And of course is it worth while doing this from an SEO perspective.
Link Building | | Hardley10 -
Why Breadcrumbs don't work on my web page?
I tried 2 types of breadcrumbs plugins. Last - Yoast
Link Building | | NadiaFL
Breadcrumbs. But result is the same - they don't show up in the footer. I
followed direction and made settings - result is the same. Any ideas? Thank you! http://oasisoftheseasallureoftheseas.com/0 -
Links aren't showing up in SEOMOZ resports
Hi, I've been building links to my client's website for the past 3 weeks. I know that there are several sites that link to my client's website now but SEOMOZ's link analyses says there aren't any sites linking to my client's website. Anybody know what's up with that? Sincerely, Rex
Link Building | | Rex0 -
New website, small business, niche market --- what's my best link building strategy?
Hi everyone, We are a small company manufacturing a niche product (indoor playground equipment), our new English website (www.funlandiaplaygrounds.com) has just been launched 2 months ago, before that we didn't even have a website in English. As the international sales manager of such a small company, I have to do all the international marketing jobs including SEO, but before this I'm almost a noob on SEO. I've just started the linking building work for our website, after a research on the links of our highest ranked competitors, I have found out that almost ALL of the external links of them come from directories and purchased links, many links are very dubious, please see the open explorer results below: http://www.opensiteexplorer.org/links?page=1&site=www.spiplay.co.uk&sort=page_authority&filter=&source=&target=page&group=1 http://www.opensiteexplorer.org/links?site=www.softplay.com%2F http://www.opensiteexplorer.org/links.html?page=1&site=www.china-cheer.com&sort=page_authority&filter=&source=&target=page&group=1 http://www.opensiteexplorer.org/links?site=http%3A%2F%2Fwww.aileplay.com%2F http://www.opensiteexplorer.org/links?site=internationalplayco.com%2F The search keywords is: indoor playground equipment. According to the latest SEO theory and numerous posts I've read here, links from these directories carry very low value, and solely relying on these links may even cause penalty to the website, but the reality is, all these competitors rank on the top as a result of these "spammy" links. For example this website www.aileplay.com that has the highest PA of 64 and rank on the first page on the search result of indoor playground equipment, has tons of spammy links. That is the situation we are facing now, then my questions is: As a small business in such a niche market, what is our best strategy to rank well in a reasonable time, say 3 months to 6 months? I do not think our competitors are very strong and hard to beat, I believe we will beat them in content creation for sure, but what should we do in link building? should we start to get directory links now, as it obviously works for them? Or should we first create more attractive content, then use these content to get natural links BEFORE we submit for directory, as recommended by most link experts here? If so should we just sit back doing nothing before the link worthy content is created and natural links starts to come in? I highly appreciate any comments! DSG_clan
Link Building | | DSG_clan0 -
Backlink reports in OSE, the good and the bad!
Hi all Mozers, I have a couple of questions re the backlink reports in Open Site Explorer. In the introductory video Rand suggests that you can indentify backlinks that are a) Having a positive effect, and b) Having a negative effect on SEO campaigns. Do you identify such links using the domain/page authority of the linking page? Also, we know we have more links than OSE is reporting, does this mean that the links that are not reported are not helping our SEO campaign? Many thanks in advance, much appreciated. Lee
Link Building | | Webpresence0 -
301 to main page or deeper page for best 'link juice'.
I have two rafting websites. One ranks high for one set of keywords. The other ranks okay for a different set. I want to redirect the whole second site to the first to give the first more link juice and hopefully lift the first site even more in the rankings. Would it be better to link to a deeper page of the 1st site. I have a page that does well for some long tail keywords or would it be better to link to the main page. Or is it a bad idea over all.
Link Building | | tkobrien0 -
Ecommerce websites' SEO strategy
Hi, Can you please make a short list how to make ecommerce websites SEO strategy?
Link Building | | Netkreativ
I am working on ecommerce websites and I curiuos how You professional think this. Now I am: Optimizing the website Build few relavant links Regards, Misi0