Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are there less backlink domains in Moz vs. Semrush?
For our domain studyville.com, Semrush is reporting 46 linking domains, and Moz is reporting 7. Does anyone know where there is such a large discrepancy?
Link Building | | shelbythomas0 -
Sitemap.xml change to anchor links (converter)
Hi all, I would like import the sitemap.xml to an anchorlinks list. List with links navigatetion html ^^. Is there a posibility to use a tool or in excel? Because I will use 5000 products and 1000 brands.
Link Building | | Dreamgame20160 -
Has anything changed www vs non www
Hi, I started to read different comments on www vs non www and I am a little confused. As far as i know from SEO point, either url is fine but can you please share your latest view ? I was just wondering if anything has changed since penguin on this? Thanks!
Link Building | | Rubix0 -
A link with "return false"- OSE sees as a No Followed Link
Hello, I couldn't find a clear answer to the impact on SEO for a link written in this way: [" class="expert_info" onclick="window.open(this.href);return false;">](w</span>ww.yourwebsite.com<span style=) [Does the "return false" act as a "no follow"? I came across this in our link data in Open Site Explorer which lists these links all as "no follows." However, an engineer I spoke to said that it shouldn't impact search engine behavior. Any ideas? Thank you in advance! -Sarah K.](w</span>ww.yourwebsite.com<span style=)
Link Building | | OneMedical0 -
Open Site Explorer Changed?
I am getting this at the bottom of Open Site Explorer. I have never seen this message before. "Why does my number of inbounds links listed not match my total links count? The listed number may be slightly different because we only show 25 links per domain (so that you see a variety). Additionally you have 18 links from no-indexed pages. Unfortunately, we can't get info for those links." Did SEOMoz change something? If I look at a site, SEOMOZ's OSE doesn't show me all the links it can find anymore? It only shows 25 max???
Link Building | | JML11790 -
SEOmoz's ranking of links on competitor question..
I am Perplexed about SEOmoz crawl for new links on a competitors website. When I looked at a company who received very high ranks for the links...on SEOMoz...such as. 95, 96, 97 etc..... I couldn't figure out how they ranked so high. One company bought 20 links for a couple hundred dollars and all of them were ranked very high on SEOmoz but when I used the page rank tool on them individually they were either "0" or "1" and none were over a 2. I was debating submitting for those since I am just starting out and wanted to get our name listed in the internet world. Are the high rankings by SEOmoz related to something else? Would I be better off buying a listing in some of the premium directories instead. Like Yahoo or BBB or Manta. ( After I get my site optimized first) Thank you, Greg
Link Building | | Boodreaux0 -
Changing backlinks anchor text
Hi, I've read a few blog post here that suggests the strength of building links using your brand as an anchor text. This supposedly gives the site authority. Currently a chunck of the back links to my homepage are on generic terms i'm trying to rank for which doesn't seem to be working very well. I was thinking of contacting the various webmasters to change the anchor text to that of the site brand name but wondering if this will signal a manipulation of links to the search engines and potentially could be flagged as paid links? Has anybody done this before and what is the danger of doing this? Thanks Duke
Link Building | | clickangel0 -
Backlinks I don't want
Hi all. According to google webmaster tools www.buywatchesin.net made 29,245 links to ONE category to my website. I went to this site and they copied my article with my links without my permission. Its the same one on all sites in that domain. Thats a lot of links and I thinks that bad for my site. I can't find any contact details to that site. What do you think, any advise?? Please help!
Link Building | | menswatchshop0