Measuring the size of a competitors website?
-
I think our website is too big, far too many indexed pages. I'd like to do some research on how big our competitors' websites are (how many indexed pages). Is there a way to do this?
Cheers,
Rhys
-
Thanks, both!
-
Xenu's Link Sleuth is free so you may want to check that out (not sure just now whether there are any limits with regard to website size) but I also recommend Screaming Frog - it's money well-spent, such a feature rich tool!
-
I highly recommend buying the license for Screaming Frog, at $100/year, you won't find a more valuable SEO tool for the money. You won't find a free (and trustworthy) that will crawl a site that large.
-
Hi Alick,
I tried that but I only have the free version so we're capped at 500 URLs. Also, the site:search provided 50,000 results, but I know we don't have that many pages. Are there any other tools?
Cheers,
Rhys
-
Hi,
Do a site: search on Google itself - like "site:google.com" - to return pages of SERPs containing the pages from your competitors site which Google has indexed.
I would also suggest you use screaming frog tool you will get more accurate value here.
Hope this helps.
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What Tools Should I Use To Investigate Damage to my website
I would like to know what tools I should use and how to investigate damage to my website in2town.co.uk I hired a person to do some work to my website but they damaged it. That person was on a freelance platform and was removed because of all the complaints made about them. They also put in backdoors on websites including mine and added content. I also had a second problem where my content was being stolen. My site always did well and had lots of keywords in the top five and ten, but now they are not even in the top 200. This happened in January and feb. When I write unique articles, they are not showing in Google and need to find what the problem is and how to fix it. Can anyone please help
Technical SEO | | blogwoman10 -
How to decrease the spam score of website ?
Hello , Everyone I have my own website crawlmyline we are doing seo from last 2 years but didn't do any spammy work on my my site .Still I won't able to stop the spam score and now my website has 28 % spam score according to the moz.I have taken subscription of moz pro but still i can't understand the process. Can anyone please tell me how to decrease the spam score of website ?
Technical SEO | | kuldeep_chauhan0 -
Website is not indexing
Hi All, My website URL is https://thepeopeople.com and it is neither caching nor indexing in Google. Earlier the URL was https://peopeople.com. I have redirected it to https://thepeopeople.com by using 301 redirections. I have checked the redirection and everything else is fine and I have submitted all the URLs in search console also, still the website is not indexing. Its been more than 5 months now. Please suggest a solution for this. Thanks in Advance.
Technical SEO | | ResultfirstGA0 -
Duplicate content on charity website
Hi Mozers, We are working on a website for a UK charity – they are a hospice and have two distinct brands, one for their adult services and another for their children’s services. They currently have two different websites which have a large number of pages that contain identical text. We spoke with them and agreed that it would be better to combine the websites under one URL – that way a number of the duplicate pages could be reduced as they are relevant to both brands. What seamed like a good idea initially is beginning to not look so good now. We had planned to use CSS to load different style sheets for each brand – depending on the referring URL (adult / Child) the page would display the appropriate branding. This will will work well up to a point. What we can’t work out is how to style the page if it is the initial landing page – the brands are quite different and we need to get this right. It is not such an issue for the management type pages (board of trustees etc) as they govern both identities. The issue is the donation, fundraising pages – they need to be found, and we are concerned that users will be confused if one of those pages is the initial landing page and they are served the wrong brand. We have thought of making one page the main page and using rel canonical on the other one, but that will affect its ability to be found in the search engines. Really not sure what the best way to move forward would be, any suggestions / guidance would be much appreciated. Thanks Fraser .
Technical SEO | | fraserhannah0 -
Website not ranking but the blog is!
I am hoping someone might be able to help me, I am doing some work on a website. A new version of the site was recently launched and since then rankings have plummeted and the new blog pages are ranking better! When the new version of the site went live, the domain changed to the non-www version, plus an incorrect robots.txt file and we have never really been able to fully recover (both of these things were beyond my control!). The robots.txt file was corrected and some of the external links links changed to the non-www but there is a 301 redirect in place so changing to the non-www shouldn't have been the reason to drop the site out completely. Before the launch of the new website, the site was ranking on the front page of Google for a lot of relevant keywords such as outdoor blinds, outdoor blinds Perth, cafe blinds, patio blinds, etc. The quality of the links is pretty bad and I am attempting to remove them before doing a disavow of all the really bad quality links but unless we were really unlucky I don't think it's the links right now that are causing the problem. I have ran the site through numerous crawl tests, checked the robots.txt, there are no messages in GWMT, the pages are indexed but I have a feeling there is something wrong with the site that is stopping this site from ranking well. If anyone could give me any insights I would be really grateful. I know the site could be better structured from a keyword/ structure perspective but the site was ranking fine!
Technical SEO | | Karen_Dauncey0 -
Google not indexing my website
Hi guys, We have this website http://www.m-health-expo.nl/ but it is not indexed by google. In webmaster tools google says that it can not fetch the site due to the robots.txt but i do not see any faults in it. http://www.m-health-expo.nl/robots.txt Do you see something strange, it really bothers me.
Technical SEO | | RuudHeijnen0 -
Websites that scroll....forever....
I'm seeing more and more websites that have all their content on one page. I came across www.otbthink.com today and am wondering if I'm missing something here. I ask this because I am trying to figure out how to link to their copy writing portion of their website. I have a client that is in their area and needs someone to do some copy writing work for them. All the content there is being served through css. The search engines will see it but how do you optimize for this sort of site? How do you link to a particular section of this site? Am I missing something obvious here? Thanks.
Technical SEO | | DarinPirkey0 -
Best way to display maintenence mode on a website?
I have a website with lots of traffic and sometimes the backends fail. I want to use lighttpd to show that the website is under mantenence and should be back up shortly. I was thinking of using Soft 503 errors or doing a 302 for every page to /maintenance.html. What would you do (besides fixing the backends, we are already doing that :P) to avoid hurting your SEO efforts? Thanks in advance Mariano
Technical SEO | | marianoSoler980