Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I've watched a few backlinks disappear from our inbound link search.
After inquiring to Moz.org about this, they suggested this might be due to a new feature that was rolled out, however, this 2 year old backlink still hasn't shown up and I'm a little concerned. Has anyone else experienced this?
Link Building | | Deacyde0 -
I have listed my business in a lot of directories and my authority hasn't changed why??
I have listed my business in a lot of directories recently and my domain authority dropped 1 point...why? second I would like to know what is the most effective way to increase my website domain authority?
Link Building | | VanityCosmetic0 -
How can I tell if a site is trustworthy and is not / hasn't been penalized by Google?
Hello Moz Community, I'm looking to do some link building for a client and I would like to know if there's a way to find out if a website is trustworthy and is not or hasn't been penalised by Google. Thank you.
Link Building | | CosminC0 -
I'm thinking about buying a competitor and 301 redirecting? How much SEO value?
I'm thinking about buying a competitor and 301 redirecting their site to mine. The high level stats are as follows. My site has a DA of 46 and the homepage has a PA of 55. 375 root domains, 170,000 links. The site sells millions of dollars worth of product each year. The competitor (who I had never heard of) has a DA of 58 and a homepage PA of 64. They have 634 root domains and 260,000 links. They aren't selling much of anything (less than 100,000 per year). We might be able to operate their site but I'm concerned about maintaining 2 platforms. My question is about the value of buying this site and 301 redirecting it to my site. Would this create long term SEO value or not? Any examples that have been documented are greatly appreciated.
Link Building | | bradwayland0 -
Anyone have Free Directories with High Domain Authority they'd like to share?
I was just curious if anyone had any directories they'd like to share that carry high Domain Authority(imo: 70+)? I know about dmoz.org and Pegasus but other than that, none. Thanks.
Link Building | | Modbargains0 -
Ecommerce websites' SEO strategy
Hi, Can you please make a short list how to make ecommerce websites SEO strategy?
Link Building | | Netkreativ
I am working on ecommerce websites and I curiuos how You professional think this. Now I am: Optimizing the website Build few relavant links Regards, Misi0 -
My customer needs to change hosts! What part of the SEO process should I worry about most? Backlinks?
I just hired an agency to build quality backlinks and articles....should I worry about the new webhost? Should I wait? Ill assume if the page names are the same, all will be good. What if they change? Newbie Help! Thanks Mozzers.
Link Building | | Giggy0 -
What's your best link, and where is it from?
I'll start with ours, it's a BBC article which listed our website in the resources box, keyword rich 🙂
Link Building | | tomcraig860