Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at [email protected].
Cheers!
Erin
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What does "Number of Linking Root Domains" in Open Site Explorer actually mean?
I've read through the forum, but I'm confused as to what "Number of Linking Root Domains" means in the Linking Domains report. For example, if the root domain is Moz.com and the number is 135,000 does that mean there are 135,000 links from Moz pointing back to my site or does it mean there are 135,000 sites linking back to Moz? Any help is appreciated. Thank you!
Link Explorer | | karrabarron1 -
Why is Open Site Explorer Showing me Spammy Links That I Already Removed?
Every two months or so, I like to download the OSE list of all the backlinks passing equity to my site. I do this so I can see which spammy links I need to try and remove. However, I see that OSE is still showing me links that I already removed...Either A) the page that is being linked to is 404-ed B) the spammy website that was linking doesn't exist anymore or C) the spammy site still exists, but the link is GONE. Why is OSE still spidering these sites and links? Does that mean Google is still spidering these links? Should I disavow them to be safe? Thank you! P.S We are not experiencing any drastic drops in traffic or penalties at the moment.
Link Explorer | | TMI.com0 -
Do other sites linking to my page with a low domain authority affect my rankings?
Hello, my page authority 30, I see other page with an authority of 21 for example linking to us. Is this good or bad, does this affect our ranking? A lot of these don't have anchor text, I'm guessing we should reach out to these webmaster to include anchor text with the keywords we want to drive traffic to this page or make as descriptive as possible? Thanks! Laura
Link Explorer | | lauramrobinson320 -
OSE not picking up all internal links?
Hi! I've been checking a few pages in OSE. For example: https://www.holidayhypermarket.co.uk/holidays/last-minute This, according to the tool, has only 6 internal links. However, a Screaming Frog crawl picks up 700+, which is about right seeing as there is a link in the footer. What do you think could be behind the discrepancy?
Link Explorer | | Blink-SEO0 -
Open Site Explorer and Expired Links?
Hi guys, I've been checking a few domains for SEO purposes and it appears that OSE is indexing a lot of links which no longer exist, this has been the case for over 6 months - does OSE cleanse expired links? As I'm sure they will be contributing to authority in the eyes of Moz?
Link Explorer | | lotterybase0 -
Why won't Open Site Explorer detect inbound links from big name websites?
I understand smaller, spammy sites woudn't get indexed, but some of the websites linking to mine have high domain authorities. The website is a parked subdomain.
Link Explorer | | myers3680 -
Internal Equity Passing Links Showing 0
Hi There. I ran Open Site Explorer on a client's site and it's showing 0 for Internal Equity Passing Links and 0 Total Internal Links. Has anyone seen this before? If anyone knows what might be causing this I'd be very appreciative of the assistance. Thank you!
Link Explorer | | kwaterman0 -
Why Moz doesn't offer tool similar like SEOprofiler Link Disinfection?
Hi Moz, As I have been your member & using pro tool for more than an year, it's been amazing experience working and using your each tool's function tool, wehther to check on-page grader, rank tracking, on-page crawling issue, keyword difficulty and the one my best for link analysis is Open Site Explorer etc. I came here to ask about something which I didn't find here than other providers is most popular Bad Link Detection Tool. As **SEOprofiler Link Disinfection **offers their customers to use this to get all of their bad or spammy links harming them to rank high or getting away from ranking goal. Let me know, are you planning anything similar to this to bring soon in your tools and provide your members to take benefits of such type function as well OR should i go to SEOprofiler for this? Best,
Link Explorer | | Futura
Teginder Ravi0