Crawl Diagnostics: How many pages (deep) will it crawl for dup content
-
Does anyone know how deep the crawl diagnostics will crawl when searching for dup content? Will it crawl the entire site, or will it only crawl "x" amount of pages?
Thanks!
-
Hello!
The standard and medium plans will each have a set limit to crawl up to 50,000 pages. The higher plans have adjustable limits, 10,000, 20,000, etc.
Hope this helps
-
The number of pages crawled depends on which plan you have - https://mza.seotoolninja.com/products/pricing
- Standard = 250k
- Medium = 500k
- Large/Premium = 1.25m
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Top pages according Link Explorer shows 301 pages and images
I was looking at the top pages on my website using Link Explorer, it contains 301 pages and image URLs. Is this expected or I should be doing something to fix it? URL with second highest PA in site is a page already 301 redirected more than 6 months back. URL with highest PA in my blog is a random image in the blog, rather than any blog posts. Is this normal? Thanks -Aji
Moz Bar | | ajiabs1 -
Duplicate Content on Website with Multiple Locations
Hi there, I've spent hours reading posts on duplicate content and googling this but I'm still not sure what to do. We created a site that has two WP installs for a company with two different locations - the landing page is website.com and links to WP install 1 (website.com/city1), and WP install 2 (website.com/city2). They specifically wanted two different sites so they could be managed by staff at either location. However some of the pages have the same content - ie. services, policies, etc. so all of those are showing errors for duplicate content. All pages have different city-specific URL's and meta-descriptions but that clearly doesn't help. We can't redirect the "duplicate" pages because then it would take the user to the other city's specific site. Is there anything we can do?? Is this going to significantly damage rankings? Thanks kindly for any help you can provide.
Moz Bar | | charlie0071 -
Weird 404 in Crawl Diagnostics
I'am getting a lot of 404 errors (196 to be precise ) - but their pattern is weird.
Moz Bar | | oorbo
The page that the crawler is trying to find is (e.g):
http://www.oorbo.com/item/asufa-israeli-design-shop**/www.oorbo.com.
the linking page is** http://www.oorbo.com/item/asufa-israeli-design-shop meaning it adds to the end of the link the root URL - /www.oorbo.com. This happens in all 196 cases - trying to find a page http://www.oorbo.com/some-page/www.oorbo.com from a refferer page http://www.oorbo.com/some-page. Obviously this pages do not exist, and it's getting a 404. I've look into the pages themselves and digged into their code - It doesn't seem that the bad link is any where on the page. Did anyone came across this kind of issue? any one can point me to a solution ?0 -
Moz is finding phantom pages
I suddenly have 4xx errors in my crawl diagnostics because pages with “/%3C/div” added to the end of the URL that are linked from the normal page can't be found. I didn't create the pages, and they don't exist, but Moz thinks that they do. I went back through to see if any changes in WordPress, theme or plugins versions might be the cause, but this is the only site that I have this issue, so I don't think that is it. Does anyone have an idea what causes this?
Moz Bar | | samuelldrew0 -
Puzzled. High domain authority, but low page authority?
I have a question regarding how a page's authority can be low but the domain authority be high. I'm using the SERP report in moz to compare competing pages. The page I'm working on is an ecommerce category page. Moz says it has very low page authority, has been indexed for a little over a year, it's domain authority is a moderate 50, and has no root domains linking to it. It is more relevant than the other pages (has more products for sale that fit the keyword searched for), has a decent amount of copy on the page, but ranks around #10 in the search results. The competing pages are also ecommerce category pages. The 9 ranking higher have domain authority varying from 30-75, and 1 or 2 root domain links to them. Their number of products that match the keyword searched are all over the board, some have many matches, some have only 6 or 8 results. Some have a moderate amount of text on the page, some have very little text on the page. Yet they have page authority in the 30s-40s. What goes into page authority that I'm missing which would make these other pages have so much more authority when they are just slightly better in the other metrics listed above? What am I missing here?
Moz Bar | | K-WINTER0 -
Deleting old obsolete pages from the Page Grades section?
Last year we did a total site overhaul, moving from an old ASP site setup to a new WordPress set up. I notice that there are still a handful of old ASP pages being graded still - and turning up as F of course. I tried marking them and deleting them, but hitting delete makes the check box clear, but does not remove the page. How do I get them totally out of grading circulation? Also, some of the links are showing with session info and extra unnecessary characters - how can I clean those links up?
Moz Bar | | OBIAnalytics1 -
URLS appearing twice in Moz crawl
I have asked this question before and got a Moz response to which i replied but no reply after that. Hi, We have noticed in our moz crawl that urls are appearing twice so urls like this - http://www.recyclingbins.co.uk/about/ www.recyclingbins.co.uk/about/ Thought it may be possible rel=canonical issue as can find URL's but no linking URL's to the pages. Does anyone have any ideas? Thank you Jon I did the crawl test and they were not there
Moz Bar | | imrubbish0