Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do i have too many 'follow' backlinks and am i being penalised by Google for it?
Hi all. I read on Moz recently that if a website has too large a percentage of 'follow' backlinks, that Google penalise the website because that is unnatural. IS this correct please? I ask because i have recently found that our own website, according to Moz, has 16,500 inbound links and they are ALL 'follow' links. These are all from independent 3rd parties and we havent commissioned any of them, so it is completely natural. URL if anyone cares is www.themosquito.co.uk Any advice would be appreciated. Cheers
Link Building | | TheMozzy0 -
Backlink not index on MOZ
Hi All, help me... why backlink for https://gobiz.co.id/pusat-pengetahuan/aplikasi-kasir/ at https://aldhifajar.com/spots-aplikasi-kasir-simple-untuk-setiap-jenis-usaha/ not index at Moz? how it problem on web?
Link Building | | masirwin9180 -
Post index of quality back links; How long should you leave it to see if it's had any impact?
The question is pretty much in the title. We had a link from a Charity site (high PA/ DA) and from Metro online recently. Indexed about 3 weeks ago and 5 days ago respectively. No movement to speak off. (Previously we've seen significant boosts from far lesser links). Should I assume they have had no effect? Or is too early to tell I know it takes up to 2 months for some of our new pages to rank, same with passing authority from links? (Note: I appreciate the competition will play a significant part in this, but my question is specifically about how long one should leave to know one way or another. )
Link Building | | isaac6630 -
Does paying a reviewer for an impartial review violate Google's guidelines?
When a company pays for an impartial review from a website, should these links be no-followed? I am confident that paid positive reviews are seen as a manipulation of search, but is paying for an impartial review okay?
Link Building | | RG_SEO0 -
Disavow Links - how do you know if it's worked?
I asked another SEO company to analysis my link structure (as I was too busy!) As I was flat lining on some work I was doing. They said I potentially had an algo penalty and that i need to do a disavow , even though I had no messages from Google saying I had unnatural links. stupidly I agreed to the disavow. Looking at Webmasters tools it seems they've submitted a bunch of links. Since they've done this traffic dropped by 60%, ranking dropped massively. In google Webmasters all the links which are meant to be removed are still showing. How do I know if the actual disavow has been done? And should I do a reconsideration request? Even though Google hadn't flagged an issue ??
Link Building | | Cocoonfxmedia0 -
Think I'm ready to do some link building. Couple questions.
Getting ready to do some link building. I've got several lists of competitors' links, including a bunch of sites with broken links that would be a great fit to link to us. I've got a capable VA to get started work on reaching out to people. Just curious if this is the right game plan, seems a little simple: For this round of link building I'm thinking all the links would point to my root domain. -Find quality sites/links to go after -Find an email to the owner/webmaster -Have the VA send them a value proposition email(i.e. why it's good fit for all)...or tell them about broken links etc. -Follow up myself when a response is generated. -Hope/verify they link to us. Thanks for the help with the newbie questions.
Link Building | | astahl110 -
Panda Update: Isn't a link still a link?
I was doing some link building and some SEO's said that the Panda update affected many websites. I am going to use eZineArticles.com as my example. EzineArticles was affected by the Panda update and does not show up in the SERPs as much as before. But they still have doFollow Links coming from the articles I am submitting. QUESTION: Regardless if EzineArticles was affected by the Panda Update, isn't a "Follow Link" still a "Follow Link" OR am I completely wasting my time on this devalued website? Edit: Yes I know a PR 0 page is not as valuable as a PR 9 page. I am asking from the standpoint of the affected Panda Update domains overall.
Link Building | | Francisco_Meza0 -
Why doesn't the Better Business Bureau show up in my link analysis
I've been working on SEO for one of the companies I've designed a website for and I'm confused by the company's lack of Better Business Bureau backlinks. The Company in question does have a BBB account and that account links back to the company's website. However, when I check in the link analysis for the site, the BBB link doesn't appear. My competitors, on the other hand, do have BBB links in their analyses. So, I'm wondering if I somehow don't have the right type of BBB account. The BBB seems to be a pretty good place to have a link from, and the company pays $300.00 per year for the membership, so I'd like to get the most out of it. Here's a link to the BBB page for the company http://www.bbb.org/utah/business-reviews/plumbers/platinum-plumbing-services-in-west-jordan-ut-22199778#bbblogo And here's the company's website www.slcplumbing.com Now, the company site I've just listed is 301 redirected to www.platinumplumbinginc.com, but even when www.slcplumbing.com was the main site, the BBB backlink didn't show up. Thank you Blake
Link Building | | BlakeMcGillis0