What is the best tool for getting a SiteMap url for a website with over 4k pages?
-
I have just migrated my website from HUGO to Wordpress and I want to submit the sitemap to Google Search Console (because I haven't done so in a couple years). It looks like there are many tools for getting a sitemap file built. But I think they probably vary in quality. Especially considering the size of my site.
-
Screaming Frog is good for crawling an existing Sitemap.xml file and can indeed produce Sitemap.xml files, but if our site is medium-sized (thousands of URLs) then really you'd want a dynamic one. Pretty sure the Yoast SEO plugin for WordPress has this built in with some tweak options and variables, probably start there
With Screaming Frog you'd have to keep manually re-building your sitemap XML / XML index file. Sounds pointless, boring and tedious when relatively stable dynamic options exist
-
As Keen already stated, Screaming From is the best option
-
We use Screaming Frog and we are about 80k pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Add selective URLs to an XML Sitemap
Hi! Our website has a very large no of pages. I am looking to create an XML Sitemap that contains only the most important pages (category pages etc). However, on crawling the website in a tool like Xenu (the others have a 500 page limit), I am unable to control which pages get added to the XML Sitemap, and which ones get excluded. Essentially, I only want pages that are upto 4 clicks away from my homepage to show up in the XML Sitemap. How should I create an XML sitemap, and at the same time control which pages of my site I add to it (category pages), and which ones I remove (product pages etc). Thanks in advance! Apurv
Technical SEO | | AB_Newbie0 -
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Please advise.
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Are there any other precautions I should be taking? Please advise.
Technical SEO | | BVREID0 -
My website pages are not crawled, what to do?
Hi all. I have made some changes on the website so i like to crawled them by the search engines Google especially. I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites. Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml. Still my pages are not crawled your opinion please. Thanks
Technical SEO | | lucidsoftech0 -
Change in url structure - added category page
I have recently started an e-commerce website and have now changed the url structure and added another level to my category pages. So where it before was www.website.com/shirts it is now www.website.com/clothes/shirts. So I added the clothes category (just an example) before the shirt category and am now finding that the old url is still found in the search index and is still live on my site. How could this be? I use wordpress and simply change the urls in the backend. The products are still under www.website.com/product/blue-shirt-123 so they won't be affected but I suppose it now means I have duplicate category pages? So my question is: Should I 301 the the old category page (www.website.com/shirts)to the new url (www.website.com/clothes/shirts). And how can the old url still be live on my site? If this was a bit unclear, please let me know. Appreciate your replies!
Technical SEO | | bitte0 -
What is the best URL designed for a product page?
Should a product page URL include the category name and subcategory name in it? Most ecommerce platforms it seems are designed to do have the category and sub-category names included in the URL followed by the product name. If that is the case and the same product is listed in more then 1 category and sub-category then will that product have 2 unique urls and as a result be treated as 2 different product pages by google? And then since it is the same product in two places on the site won't google treat those 2 pages as having duplicate content? SO is it best to not have the category and sub-category names in the URL of a product page? And lastly, is there a preferred character limit for a URL to be less than in size? Thanks!
Technical SEO | | gallreddy0 -
Omitting URLs from XML Sitemap - Bad??
Hi all, We are working on an extremely large retail site with some major duplicate content issues that we are in the process of remedying. The site also does not currently have an XML sitemap. Would it be advisable to create a small XML sitemap with only the main category pages for the time being, and then after our duplicate content issues are resolved, uploading the complete sitemap? Or should we wait to upload anything until all work is complete down to the product page level and canonicals are in place? Will uploading a incomplete sitemap be fraudulent or misleading in the eyes of the search engines and prompt a penalty, or would having at least the main pages mapped while we continue work be okay? Please let me know if more info is needed to answer! Thanks in advance!
Technical SEO | | seo320 -
Best practice for rich snippet product data - which page shows up?
We have a website with thousands of pages that rank locally for a specific service we offer. What I'd like to do is add rich snippets to these pages. I'd like to setup the services we offer as 'products' in the rich snippets, so that our 2 services show up below the url as rich snippets. I guess I'm not sure if the markup is supposed to be on the product page itself, or if I should use the offerurl tag, to create a separate page on the site whose only purpose is to have a long list of the services we offer pointing to the local pages as the offer url's. What do I do with this page? what are best practices for this offer aggregator? Are there any resources I can look at? Am I even doing this right? I'm new to having markup pages, and I'm hoping that the markup code doesn't actually need to be on the product offer page itself, but that the product offer page is the one that shows up on the results - that is my last question actually - which page will show up? the offerurl link, or the actual markup page.
Technical SEO | | ilyaelbert0 -
Is there a great tool for URL mapping old to new web site?
We are implementing new design and removing some pages and adding new content. Task is to correctly map and redirect old pages that no longer exist.
Technical SEO | | KnutDSvendsen0