Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Advice for structuring hotel website
Hey guys, I am currently setting up a hotel booking website and I'm not so sure how to structure it. I have landing pages for: 1. Cities
Intermediate & Advanced SEO | | baresound
2. Sights
3. States The main keywords are mainly "Hotels in Cityname" or "Hotels near Sightname". What would be the best SEO friendly way of structuring the url? https://hotels-example.com/hotels/cities/cityname
https://hotels-example.com/hotels/sights/sightname
https://hotels-example.com/hotels/states/statename or https://hotels-example.com/hotels/cityname
https://hotels-example.com/hotels/sightname
https://hotels-example.com/hotels/statename or https://hotels-example.com/hotels-in-cityname
https://hotels-example.com/hotels-in-sightname
https://hotels-example.com/hotels-in-statename Or are there better ways of structuring it or am I just overthinking it? I would greatly appreciate any advice and suggestions 🙂 Best, Max0 -
Website not ranking
Firstly, apologies for the long winded question. I'm 'newish' to SEO We have a website built on Magento , www.excelclothing.com We have been online for 5 years and had reasonable success. Having used a few SEO companies in the past we found ourselves under a 'partial manual penalty' early last year. By July we were out of penalty. We have been gradually working our way through getting rid of 'spammy' links. Currently the website ranks for a handful of non competitive keywords looking at the domain on SEM RUSH. This has dropped drastically over the last 2 years. Our organic traffic over the last 2-3 years has seen no 'falling off a cliff' and has maintained a similar pattern. I've been told so many lies by SEO companies trying to get into my wallet I'm not sure who to believe. We have started to add content onto all our Category pages to make more unique although most of our Meta Descriptions are a 'boiler plate' template. I'm wondering.... Am I still suffering from Penquin ? Am I trapped by Panda and if so how can I know that? Do I need more links removed? How can I start to rank for more keywords I have a competitor online with the same DA, PA and virtually same number of links but they rank for 3500 keywords in the top 20. Would welcome any feedback. Many Thanks.
Intermediate & Advanced SEO | | wgilliland1 -
Keyword Stuffing - Ecommerce websites
Hey Mozzers, Im undertaking a content audit and its going very well, we have written some better content for the first set of pages, it still needs some improvement but we have a good base and starting point from which we can make an SEO log and work on it over time. For the content I used the following formula for how many times to include a keyword Word Count / Length of Keyword. (eg. 600 words / 3 word keyword = 200). Then 1-4% of this (2-8 times). This has worked well for me in the past and has been a good base guide. I have ran the pages through Moz optimiser and every single page hit an A for keyword page optimisation. However many of the pages failed on keyword stuffing, which obviously has high priority. My dilemma is that, moz counts 15 as the cut off for keyword stuffing with the written text we have done really well with using it a set number of times. But these pages are product category pages. The keyword in the extreme of cases is listed 7-9 times in the side nav menu. 7-9 times in the product category listings. Take for example *** it is optimised for thermometers (i know it a tough single word keyword, and we have fairly modest aims with it, im using it here for example purposes). The word is used a good number of times within the article but is sent through the roof with the links to the sub categories. This page for example mentions the keyword 30 times. Can anybody suggest any ways to improve on this? Is how we display the categories in the nav bar and in the page excessive? As always many thanks!
Intermediate & Advanced SEO | | ATP0 -
Website Suddenly dropping rank
Morning Moz Fans: My URL is: http://goo.gl/Dhbjwj According to MOZ, which we are tracking this URL with, somewhere between the 3rd Feb and 10th the domain went from being fairly well indexed to being dropped to pages further back than 6-7, for pretty much everything, even the company name was only registering at the bottom of page one. Around this time we were transferring the website from .php into wordpress, so we were creating new pages, called by the same names and all the same content but we created the wordpress area in a sub domain of the website. Again around this time we had an issue with the blog area and had to take it down for 4-5 weeks due to some errors which meant google wouldn't have been able to crawl these pages properly, but the rest of the website was up and running. We also discovered recently that the company have and use this domain http://goo.gl/5JvDUH So my question is, what do you think caused the problem? has it been premaritally penalised? is there a way I can get google to specifically look at it and is there any more i can do?Thank you for your help
Intermediate & Advanced SEO | | popcreativeltd0 -
Is CDN Good For International Website?
Hello - Which solution is better For International Website: 1) using a CDN, or 2) using some VPS's in each location?
Intermediate & Advanced SEO | | Vahid-af0 -
Moving half my website to a new website: 301?
Good Morning! We currently have two websites which are driving all of our traffic. Our end goal is to combine the two and fold them into each other. Can I redirect the duplicate content from one domain to our main domain even though the URL's are different. Ill give an example below. (The domains are not the real domains). The CEO does not want to remove the other website entirely yet, but is willing to begin some sort of consolidation process. ABCaddiction.com is the main domain which covers everything from drug addiction to dual diagnosis treatment. ABCdualdiagnosis.com is our secondary website which covers everything as well. Can I redirect the entire drug addiction half of the website to ABCaddiction.com? With the eventual goal of moving everything together.
Intermediate & Advanced SEO | | HashtagHustler0 -
Sitemap Folders on Search Results
Hello! We are managing SEO campaign of a video website. We have an issue about sitemap folders. I have sitemaps like ** /xml/sitemap-name.xml .** But Google is indexing my /xml/ folder and also sitemaps and they appear in search results. If i will add Disallow: /xml/ to my robots.txt and remove /xml/ folder from webmaster tools, Google could see my sitemaps? or it ignores them? Will my site effect negatively after remove /xml/ folder completely from search results? What should i do?
Intermediate & Advanced SEO | | roipublic0 -
Need a mobile XML Sitemap?
We're going to be running our mobile site on the same domain and generating content for users on mobile devices with style sheets (will not have m.domain). The content on our URLs will be the exact same. My question is if we need to create a mobile XML Sitemap to submit to the search engines. Do we need to create the Sitemap, that will contain the exact same URLs as our non-mobile Sitemap, and just include <mobile><mobile>tags around the URLs? Or do we need to create a mobile Sitemap at all to alert the search engines that we have mobile content?</mobile></mobile> Thanks!
Intermediate & Advanced SEO | | bonnierSEO0