What's the Story on Mozscape Updates?
-
Hey gang,
As you may be aware, we were considerably late with our last index release. You have my sincere apologies for that and the apologies of the entire team. In the interest of transparency, I want to try to explain what's been going on.
Since stepping down as CEO, I've been asked to take on a few roles in the company. One of those is product architect (basically the product owner) of our Big Data team, who produces the Mozscape link index. For several years, that team has been almost exclusively focused on getting us closer to a near real-time indexing system that does not have scalability issues. Mozscape is currently smaller than our major competitors, and we're also often slower. Our metrics (PA, DA, MozRank, MozTrust, Spam Score, Social Data, etc) have been the unique value we provide, but it's not enough. We need to be competitive on size and freshness.
Building a raw link index (without processed metrics like PA/DA et al) is hard, but it's possible. Building a link index with those metrics is really tricky, and requires computer science knowledge and skills far beyond the scope of my understanding. That's what our team's been working on, and they've made some progress, but it's been slow, hampered by unknown unknowns, and materially hurt by a lack of experienced talent we can hire to help (we've had open job posts for years now).
In the meantime, our historic Mozscape index structure keeps encountering challenges - this latest round is still somewhat unexplained (we believe there's hardware issues compounded by how the system is architected to handle large domains, but there may be other issues). The team's struggled to split time between keeping the old Mozscape running and hunkering down to finish the new system. I'm trying to help them balance things as best I can, and we're going to be putting effort toward making sure we get index releases out on time. However, to do that, we'll need to scale down size, and then rebuild back up. We think we can do this while also improving the prioritization of which links we crawl (e.g. deeper on important domains that link out, less so on deep pages that don't link anywhere) so the index overall improves.
However, I don't want to minimize the risks - we may have some slow updates, some smaller indices, and some less-than-ideal data in the next one or two indices while we work to remedy this issue. I HOPE we don't, and that things actually get better immediately, but we can't promise that until the work gets finished.
TL;DR - Mozscape V2 is in development and will let us as big and faster as any link index. In the meantime, current Mozscape's having issues & we're making smaller indices in an attempt to diagnose and repair.
As always, thanks for your understanding, continued support, and if you have any questions, feel free to leave them below. I realize that this level of service/product quality is NOT OK, and I'm doing everything in my power to fix it.
-
Thanks for the update, Rand, and good luck on getting Mozscape V2 up and running ASAP. I'm sure I'm not alone in wanting to see Moz emerge as the leader in this industry, bringing the tools up to par with the amazing community assets that you guys provide.
Looking forward to seeing the mustache disappear!
-Yair
-
Hi Ken,
It's normal to see fluctuations in link counts from update to update, especially if a large number of your links are coming from relatively few domains. Take a look at this Q&A - http://moz.com/community/q/why-did-my-da-and-links-go-down. See if it helps shed light on why this happens.
-
Yeah basically on the split index. I think it would give people a better view of how they are actually doing without exposing their data across the internet. It also seems like it could be a way of more accurately determining the DA, PA, and trust as well. And could also add more value to the product. Right now the link index only picks up a portion of the total links and then tried to determine a ranking method off of that. But in GWT there is a totally different set of links usually.
I think if you merged the GWT data, GA data, and the Moz data together things could be very powerful.
I realize a lot of people want to see just what they can do to improve their ranking and nothing more. But I tend to look at SEO as a whole picture and just want more traffic. I could really careless if it comes from Google, Bing, or some guys blog. So I think a link section would also help too. This is what I imagine in my head. Another section in our account that scrapes the referral pages out of our GA accounts and shows us how our backlinks are performing over time. That would be nice, because part of what a lot of us do is buy links in some way or another. It would let us know at a glance if a blog we want to target again is still pulling the traffic it was before. Things of that nature. But I think a real value is putting everything under one roof. GWT and GA are totally separate, having the information from both of those platforms merged into one would be awesome for me.
-
"our social channels and Q+A reaches the Moz audience, which is mostly marketers and very, very few software engineers (even fewer with the right big data skills)"
True. But we might know some. I know some. We're all well connected.
-
Unfortunately, our social channels and Q+A reaches the Moz audience, which is mostly marketers and very, very few software engineers (even fewer with the right big data skills). We've got a recruiting team that's going hard at this problem, though, and yes, we've just changed to accept remote candidates for technical roles, so hopefully that will help.
-
Well, there's probably only a few thousand people who've worked on big data issues of this scale and complexity, and hiring ex-search-engine employees who've already made their millions in stock is pretty hard
Thanks for the positive wishes - we'll keep working hard to try and get this to where it should be.
-
I'll ask Martin from the Big Data team to answer the first question - I believe we use a number of different languages across the various services that power Mozscape.
On the split-index, I don't think I quite understand? Do you mean there'd be a unique version of our index that crawls the links we see from GA referrers (via the connected profiles in Moz Analytics)? That would be possible, but a ton of infrastructure work, and we'd likely need to build a unique version of the index for each customer to keep that data private.
As for GWMT - we don't have access to those, as folks don't connect/oAuth their GWMT accounts to Moz.
-
So what are the accurate numbers, last time or this time? My total links went from 15,000 down to 3,000. I really depend on the consistency and whatever changes were made, problems encountered, have really screwed up my reporting to the executive team.
-
A giant game of Whack-a-Moz ... yeah, that's fun until it's not. I can feel the ... what is it? Distress? Worry? I know you want these things to come out on time & exactly how you want but that's rare even in the land of Google & Facebook.
You guys have great company benefits. It's amazing you find it so hard to hire the right people and as you said, have had openings for 2+ years.
Glad to hear the issues are at least unrelated. That gives us hope for the site crawls & other bugs to get fixed whatever happens to the link index.
Thanks again for the update & good luck getting through it all!
-
A suggestion.
Put out a call in Q&A and social channels. Be specific about the skill set needed. Link to job descriptions. Ask for referrals and shares. Use social channels to the do the same thing. Surely we can get a few viable candidates to step forward...
Edited for one final thought. Are you open to remote candidates?
-
I deal with a few enterprise grade clients on the server configuration end, just out of curiosity what language platform do you run on?
Also, while you are here I would like to make a feature request or inquire if it is in the pipeline.
I would like to see the index split into two subsets that are used to generate the rank / trust. I would like to see it use the GWT links and also the GA referrers. It could show a better scope and at the same time keep that data private.
-
Moz has a hybrid cloud we built ourselves on a datacenter with hardware we customized. We were previously using AWS, but moved to give ourselves greater control and huge cost savings.
-
My guess and hope right now is that we don't have another delayed index. However, based on the inconsistencies and not-totally-clear diagnosis of issues, I'd say there's some risk associated with the next index.
Re: crawl updates - those have been behind, but should be catching up. I think the Moz Analytics team said 2-3 more days for completion of all the lagging reports.
-
The good news is we do know what we need to do and we have the people at the architecture level to make smart decisions (IMO). However, finding folks to help us execute on this plan has been insanely hard. We've had open positions on this team for literally 2 years, so if you know anyone, send 'em our way! I'm certainly working my networks to try and help, too.
-
So Rand, does the engineering collective know what needs to be done or do you need a technical architect to offer an assessment of the situation and recommendations? I don't have that skill set, but perhaps there's someone in your vast audience that does and would be willing to help out.
-
Just out of curiosity are you running on your own network or off of a cloud like AWs or Google?
-
Hi Rand,
Appreciate the update. Are we probably seeing another delay this round? Ive usually had my crawl updates by now.
Cheers for the good service all round.
-
Unfortunately (or maybe fortunately), those issues are unrelated. The campaigns getting behind on updating was due to some server configuration challenges as I understand it - somehow our monitoring falsely told us things were fine, and thus, even after we fixed the issue, we've been trying to play catch up on thousands of weekly/monthly reports. I'd seen an email from the team lead last night that we should be nearly clear - maybe a day or two away.
Re: memberlist - that's a separate team, and they're aware of it, so should get a fix out soon. I think one of Moz's challenges is that we've bitten off a bit more than we can chew (that's my opinion, not necessarily representative of the whole company) and with 6+ engineering teams handling 100s of operations across a giant software suite, it feels like we're always playing whack-a-mole. We fix one thing, and the next week, 3 more things we never imagined could go wrong do. It's a frustrating problem, to be sure, and I honestly think the only way we'll break out of the cycle is to cut back to fewer projects and staff up teams, which will take time, discipline, dollars, and some good luck.
-
Great to read the update Rand! You know I'm passionate about Moz keeping up in index size so I'm happy to wait to get a better v2 as long as that wait is reasonable.
Is this causing other issues on the site? The memberlist isn't updating and it feels like bits of the site here & there are either very slow at updating at the moment or just not updating at all. There have been quite a few Q&A questions about campaigns not running/updating/performing alongside this. Are these related or should we be reporting them as they happen?
Thanks again for the update & good luck! It's a huge project but you have a great team to help accomplish it!
-
Yes - that should be possible and I will endeavor to make sure we're keeping the updates honest and correct for the future. Thanks Donna.
-
Very much appreciate this update. Thank you!
As you might expect, some of us rely heavily on the Moz API update. Consistency and quality is more important than frequency and volume. So while I now can better appreciate some of the difficulties and obstacles the Moz team is facing, I would love it if you could give us more than a few hours notice when an update is going to be delayed. Is that possible?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved when is the spam score updated?
when is the spam score updated?
Link Explorer | | Tv24.ro
we have been waiting since December to see an update of the spam score and moz is lying to us continuously. Shame, whenever we ask questions, they don't answer us, I think they almost make fun of people.
I requested an update to antena24, tv24, digi24, alba24, g4media, b1tv,newsela0 -
Moz isn't crawling all my backlinks.
Moz isn't crawling all my backlinks. It's showing only 29 referring domain when I have more than 200 referring domains linking to my website. My website URL is 360gisthub.com.ng 360gisthub.com.ng
Link Explorer | | Mustybay0 -
Moz's new Link Explorer, including our revamped index and DA/PA scores is now open to everyone!
Hey Moz Community, Link Explorer is now open to the public! Everyone can access it via a subscription or a free Moz ‘Community’ account. As you may know by now, the brand-new Link Explorer tool is primed to replace Open Site Explorer as Moz’s link building and analysis tool. The Link Explorer project is the result of an incredible amount of perseverance and hard work by the team, and we’re proud to be able to finally share it with you — we know it’s going to revolutionize how you approach link building and make your job easier. You can read more about the tool here in Sarah Bird’s announcement post. Because Link Explorer improves on almost every aspect of Open Site Explorer, the metrics have improved, too. That means you’re likely going to see some Domain Authority and Page Authority discrepancies between OSE’s index and Link Explorer’s index. We definitely suggest you use the new DA/PA from Link Explorer, as they’re more accurate and refresh daily rather than monthly, as was the case with OSE’s index. However, we also realize that many of you use these metrics to report to your clients and colleagues, and a sudden change or fluctuation could potentially make your job harder. Which DA is the real DA? The new DA is based on a much larger index that has many improvements, several of which are designed to make the index more like Google’s than ever before. You should consider moving towards the new DA (and the old DA won’t be updated after April 26th 2018, so the sooner the better). While there will be fluctuations as we improve the model and add features to the index, we expect it to remain largely stable and to be a far more accurate picture of a site’s authority according to how it’s seen by Google. Why is Link Explorer’s DA/PA considered better than OSE’s, and which should I trust? The larger link index with improved crawl selection allows us to produce a stronger model that includes a much larger proportion of the web. That being said, DA and PA should always be considered in the context of your competitors. A drop in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Is Domain Authority/Page Authority an absolute score or a relative one? Both DA and PA are relative to the Internet as a whole. If Facebook acquired a billion new links, everyone’s PA and DA would drop relative to Facebook. Because of this, it’s always best to look at PA and DA in comparison to your competitors. What does a drop/raise in DA mean in Link Explorer vs OSE? How can I explain this to my clients when I’m reporting it? DA and PA should always be considered in the context of your competitors. A drop or raise in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Reporting that your site has moved from a DA of 45 to a DA of 42 doesn’t tell the whole story, but reporting that your site has a DA of 42 while your main competitor moved from a 43 to a 37 shows that, relative to the sites you’re competing against in the SERPs, your site has significantly more authority and ranking power. What’s happening to MozTrust and MozRank and why, and what should I replace those with? The improvements to our DA/PA and Spam Score metrics now now account for more important nuances in helping you determine one site’s ability to rank higher than another. Because they no longer correlate with Google’s ranking model as well as they used to, MozRank and MozTrust are being deprecated for better metrics. Users should rely on Page Authority, Domain Authority, and Spam Score to determine the importance and quality of pages, domains, and links. I have historical data I use to help my clients benchmark their progress. What do I do now that DA is calculated differently? You should annotate any KPI changes referencing the change in DA and PA. However, most importantly, you should compare those changes to your competitors, as this will best show how strong your site’s authority is relative to the sites you’re competing against in the SERPs. We take updating our metrics very seriously, and our last major update to the model was 7 years ago. Users of Domain Authority and Page Authority can expect us to continue to produce steady, reliable metrics for the long haul, and only make changes to these metrics when we believe the benefits dramatically outweigh the stability of the metric. Do you have any questions about the new metrics? Anticipating a tough time reporting changes to clients or bosses? Metrics, features or functionality missing that you would want to see? Let us know in the thread, and we’ll work to find a good answer for you. Hope you enjoy the new Link Explorer product and the amazing new link index powering it. We are very excited to provide this valuable data to our community and customers.
Link Explorer | | IanWatson9 -
Learn how to use Moz's Link Intersect tool to build your link profile. Get your Daily SEO Fix!
The quantity and (more importantly) quality of backlinks to your website make up your link profile; one of the most important elements in SEO and an incredibly important factor in search engine rankings. In today's Daily SEO Fix, How to Use Link Intersect to Build Your Link Profile, Tori shows you how to use Moz's Link Intersect tool to analyze the competitions' backlinks plus, find opportunities to build links and strengthen your own link profile. This video is one of our last videos in The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, make sure to visit the Daily SEO Fix channel on YouTube. And, if you have suggestions for future tutorials, please let us know via the comments below. Thanks!
Link Explorer | | kellyjcoop2 -
Can't see the link I earned from Moz in OSE
Hi Mozzers, A month ago I got the nofollow tag removed from the first custom link on my Moz profile but I can't see this in our links on OSE? SEO Spyglass shows it just fine. Is this related to the delays in the back end of Moz or have I set up my profile wrong? Cheers! Jamie
Link Explorer | | SanjidaKazi0 -
How long will it take for the changes we've made to reflect in Moz OSE spam score data?
I signed up for Moz to see the spam flags our site had triggered. As soon as I found out, we worked on it and have been trying to correct our mistakes but it's been more than a month and we've managed to neutralise zero flags. I would appreciate if someone can clarify how long the OSE data takes to refresh. Also, how do you combat the following three specific flags: Ratio of Followed to Nofollowed Subdomains Ratio of Followed to Nofollowed Domains Low Number of Pages Found Crawl only gets a valid response to a small number of pages. Thanks.
Link Explorer | | Oziti0 -
Moz can't crawl domain due to IP Geo redirect loop
Hi, I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages. In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results. Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time. Does anyone have any idea how to solve this issue? At this point I've paid for a year's subscription to a product I can't use. Thanks, Mel
Link Explorer | | SalvationArmy0 -
Repeated mysterious 404's from ancient site structure killing my rankings
Several years ago I changed my site structure to go from a flash based site to a blog based wordpress site. After doing so I went from page 1 to page 30 for my relevant search terms. I have employed people to help me track down the problem and I believe that they have narroed it to the existance of 404's being created from some unknown internal source. I have been for years getting links like this... <colgroup><col width="792"></colgroup>
Link Explorer | | dfphotographer.com
| http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/?share=facebook http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/charisma-and-steve-301/?share=email http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-04-2/?share=email http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-12-2/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/?share=facebook http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/feed/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-16-2/?share=email | ......regularly showing in webmaster tools, (this is from a top pages report from MOZ where there are hundreds also shown). When I do a moz crawl of the site, none of these links show up. Therefore I have no way of finding the source of these links (they also do not show me the source in WMT as they should). We have completely cleared the site and rebuilt it and although it is still only a couple of weeks in it still does not appear to have stopped them. Does anyone have any way of helping me find the source of these mysterious 404's?0