Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at [email protected].
Cheers!
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Inbound Link checking gets different answers
I've been comparing the inbound links report from Open Site Explorer against other backlink services and the results are very different. E.g. Compare the domain on http://backlinkwatch.com/ When I ran my test this other service found about 30% more active backlinks...why would Open Site Explorer skip those results?
Link Explorer | | rsnell0 -
Backlinks not showing on Open Site Explorer's Just-Discovered Links
Hi guys! My name is Edgar. I've recently been adding a bunch of backlinks to the site i work for, and most of them have a very high DA (DA > 80. Wohoo!). Nevertheless i'm often feel a bit frustrated when i check my site with the OSE, because just a few part of these links are showed in the just-discovered section, although i created them few days ago. So, i would like to know why is this happening, how long it take for my links to get crawled, and when can i stop getting frustrated on this? Than you very much! Edgar.
Link Explorer | | PCBU0 -
Inbound Links - How accurate or up-to-date is Open Site Explorer?
When I type my domain into OSE I get a list of linking domains. Most of them I have seen before and I know were active last year. However, when I click through, many of these links have been removed (which is what I wanted/requested) and some of the pages return 404. Google (via link:www.mydomain.com) doesn't show these links as active either. My question is why does OSE show them? Thanks
Link Explorer | | neilmac0 -
Why are none of the links we've received from press and other sources from the last two months showing on our Inbound Links in OSE???
It's really hard to judge if the DA of our site according to MOZ is 1) up to date and 2) accurate when for example there are SO many inbound links still not recorded, that were published months ago.
Link Explorer | | TVape0 -
Open Site Explorer Doesn't Discover Link for my website
Hi, I've been trying to use Open Site Explorer to track link to my own website but it always comes up with no data and shows me this message: It looks like we haven't discovered link data for this site or URL. However, this site has been up and running for more than 9 months and google webmaster can track all the links to my website. I've also followed MOZ's support instruction to tweet a bit.ly links with my website but still no luck at all. My website is www.carpetcleanerperthwa.com.au Has anyone seen the same problem before? Any help would be much appreaciated. Thanks Will
Link Explorer | | willwai0 -
Opensite explorer and Just discover links
Hi Not sure if it is just me, but I am seeing no data for Just discovered. I know there should be data in there. Is anyone else having any issues this morning. Thanks
Link Explorer | | Andy-Halliday0 -
Open Site Explorer not reporting all 301 redirected links
Our site had over 2,000 root domains linking to it as reported in MOZ Open Site Explorer (and Google Webmaster Tools). We then changed the domain and made sure that 301 redirects were set up for all pages across the site. That was about 1 month ago. Open Site Explorer is now reporting less than 300 linking root domains. For the links that it is reporting the majority of these are being 301 redirected to the new URLs (some of them we changed the links directly). However the majority of the links that are being 301 redirected are not being reported. It is reporting most of the links coming from the old to the new domain. Google Webmaster Tools is reporting over 1,000 linking root domains to the new domain (it has a max of 1,000). We did notice that MOZ took some time to update the domain authority of the new domain. It was 1 for along time and it has now jumped up to 46 ( it was previously at 74). Maybe this is a time delay thing and eventually Open Site Explorer will report all of the 301 redirected links? It is a bit frustrating at the moment as we can't fully analyse the links to the site to try to focus on the high domain linking sites to get them to change the link directly. Also, If all links are being 301 redirected to the new domain should the authority not be close to where it was previously after one month? Our search traffic has dropped considerably since the launch of the new site and hasn't returned yet, so just wondering if the 301 redirected links pass on as much value as thre original direct links. Thanks, Damien
Link Explorer | | james.harris0