Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why moz does not see all incoming links
Hi friends. I noticed that moz.com does not see some important incoming links. I added them to link tracking list but no result? for example from domain delfi.lv we receive two types of links One no follow and moz seas it. and other do follow and moz does not counts it and do not see it. https://www.delfi.lv/bizness/komerczinas/kravu-parvadajumi-ar-temperaturas-rezimu-lielakie-izaicinajumi.d?id=52924741 and there is few more like this but important and strong. kravu pārvadājumi Could you explain why is this happening. This link comes from bigest newsportal in my country, And it is indexed by google and serchable.
Link Building | | netcomsia0 -
Should I change a domain name with a good score for a newly registered one
Hi I have inherited a website http://www.hanmersprings.net.nz/ Domain sore 17 The website promotes Rippinvale so I want to register a domain rippinvale.com and direct the old domain to the new one. Is this the best way to do this to maximise the link authority?
Link Building | | VelocityWebsites0 -
Requesting udpate's from old domain to new domain
Hey guys, We had a domain change three years ago now which was redirected. Many of the old domains have not been updated to the new domain. The redirect is in place. I'm just wondering if there would it be a worthwhile exercise to reach out to sites to request an update of the URL link from an SEO perspective? Many thanks Rob
Link Building | | Griffith0 -
Moz authority. Should this be taken as just a guideline.
Hi Guys. We have an opportunity to get a back link from a charity site...the site of "Friends of Richmond park'. I've just checked it out in open site explorer and Moz grades it's domain authority as 25/100. However, looking in more detail the site has gained quite a few high authority links, both in terms of DA and PA. Many of these links are reciprocal i.e) Friends of Richmond park have links to sites which also link back to them. So i'm wondering... should the 25/100 DA be taken as a basic guideline? Or is this quite a sophisticated measure that has taken into consideration the reciprocal links and 'downgraded' them as Google might? Isaac.
Link Building | | isaac6630 -
Do the bots crawl the top of the page first - directory listing question
Hi all, I've purchased a link from a good PR4 website, but they added my link to the bottom of a lengthy page. They offered to up-sell me to get to the top of the page, where I'll be 10 or 12 links down. One of my direct competitors is at the top, so I'm happy with the directory choice (they spank me in rankings). The only reason I'm on this directory is to purely capture link juice. So knowing all of that, my question is, should I go ahead and pay to be on the top, assuming that Google bots crawl a page from the top down? It's important to me for the bot to log this link quickly. Or is paying to be at the top pointless, because the bot will just as quickly crawl to the bottom of the page and pick up the link anyway? Thanks! T
Link Building | | 800GoldLaw0 -
I use to have very good ranking for my website, now I'm struggling
I use to have very good ranking for my website, now I'm struggling with ranking, I've been trying almost everything, I purchased an advance link building, a contextual link building campaign with Submit, and doesn't seem to be working. Any recommendations!
Link Building | | desdetj0 -
How do paid directories like thomasnet.com do so well in the serps? Aren't the Panda updates supposed to be moving us away from this?
With all of the updates/changes to Google's algo, I assumed that paid listings & links like those on thomasnet.com would have less merit. Is this an incorrect assumption?
Link Building | | PropelMike0 -
Impact of Panda Algorithm Change on Articles Base.
I am a published author and researcher. All my articles are original content and high quality. If I am looking for backlinks by submitting high quality original articles to Articles Base, how will the panda algorithm change affect me? Should I be submitting articles somewhere else? If so, where? I greatly appreciate any suggestions you may have.
Link Building | | DanManCastro0