For large sites, best practices for pages hidden behind internal search?
-
If a website has 1M+ pages, with most of them being hidden behind an internal search, what's the best way to get pages included in an engine's index?
Does a direct clickpath to those pages need to exist from the homepage or other major hub pages on the site?
Is submitting an XML sitemap enough?
-
Hello Vlevit,
You could do several things. I recommend giving Google your product feed, which should accomplish your goals. Another possible solution would be to make those search pages noindex,follow so they don't end up getting indexed, but Google can still use them for discovery.
Thanks for explaining the situation.
Below is more on submitting product feeds. It is for Google Product Search, but I would imagine the "link" field where you put the URL to your product detail page will help those pages get indexed in the standard results:
http://support.google.com/merchants/bin/answer.py?hl=en&answer=188494#USEverett
-
Everett, thanks for your reply. I understand the problems of showing internal search pages. I'm not looking to have internal search results being indexed, just the pages that the results link to. We're in eCommerce.
I was under the impression that there was a clever way to have the individual product pages indexed without establishing a direct click path, but best practices recommend otherwise.
Question answered. Thanks all for your help.
-
Hello Vlevit,
If you can be more specific we may be able to be of more help. Google doesn't want you to show internal search result pages, but if this is a different type of situation it there may be an exception. Are these search result pages, product pages, category pages, content pages.... is it an eCommerce site, community, content site... ?
Generally speaking, 1M+ pages with no links going into them and content that is either sparce/thin or partially/fully duplicated on other similar pages (like a search for widgets and a search for green widgets showing overlapping content) is exactly the type of thing that will get you in hot water that would affect even the rankings of your home page.
Do you feel like your question has been answered or would you like to be more specific about your site and goals?
Cheers,
Everett
-
This is what I was assuming, but was wondering if there was a clever way around creating direct click paths to those pages, while still maintaining their importance to the site. Thanks for the info.
-
Make sure they are part of the actual structure of your website, not just part of search. Meaning, you have to have links pointing at them. Also, you will also want to make sure that those pages have value.
-
Hi vlevit,
The best practice would be to exist a direct path of flow from index page. Something like: index -> category(filter) -> subcategory(filter) -> page/product. But in some cases xml sitemaps can also help you in indexing.
BUT, beware with to large XML sitemaps, try to create more then one sitemap, group them as possible.
A few very good resources can be found under the next links:
http://www.seomoz.org/ugc/solving-new-content-indexation-issues-for-large-b2b-websites
http://www.seomoz.org/qa/view/29009/sitemaps-management-for-big-sites-tens-of-millions-of-pages
I hope it helpes,
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical for duplicate pages in ecommerce site and the product out of stock
I’m an SEO for an ecommerce site that sells shoes I have duplicate pages for different colors of the same product (unique URL for each color), Conventionally I have added canonical tags for each page, which direct to a specific product URL My question is what happens when a product which the googlbot is direct to, is out of stock but is still listed in the canonical tag ?
Technical SEO | | shoesonline0 -
Redirecting a single page on a separate domain to a new site?
My client started a subdivision of their company, along with a new website. There was already an individual page about the new product/topic on the main site, but recognizing a growth area they wanted to devote an entire site to the product/topic. Can we/should we redirect that page on the old corporate/main site to the new domain, or just place a link or two? Thoughts?
Technical SEO | | VTDesignWorks0 -
My pages are not listed in search results
My URL is: puremobile.comI have two websites: puremobile.ca and puremobile.com : both same products, but different discription , but same title of productwhen i exact search a product for example :** "HTC 70H0029701M Smartphone Case Large"** , puremobile.ca shows up , but not puremobile.com I have no issues with indexing, webmaster tools is indexing normallywhen i search for: puremobile.com "HTC 70H0029701M Smartphone Case Large" , i get the puremobile.com product page.but when i search ANY product (no matter how unique its title or description is : google doesnt display puremobile.commy PR ( as i far as i can see was PR 5 last year, and today when i checked it was PR 0) .. I havent been doing any fishy Link building, some basic blogger outreach ( non paid), and social bookmarking. and my blog is very active and I have original content on my pages.what is causing this? and how can i resolve this issue.any help is greatly appreciated
Technical SEO | | puremobile0 -
"noindex" internal search result urls
Hi, Would applying "noindex" on any page (say internal search pages) or blocking via robots text, skew up the internal site search stats in Google Analytics? Thanks,
Technical SEO | | RaksG0 -
Should I index my search result pages?
I have a job site and I am planning to introduce a search feature. The question I have is, is it a good idea to index search results even if the query parameters are not there? Example: A user searches for "marketing jobs in New York that pay more than 50000$". A random page will be generated like example.com/job-result/marketing-jobs-in-new-york-that-pay-more-than-50000/ For any search that gets executed, the same procedure would be followed. This would result in a large number of search result pages automatically set up for long tail keywords. Do you think this is a good idea? Or is it a bad idea based on all the recent Google algorithm updates?
Technical SEO | | jombay0 -
Should I nofollow search results pages
I have a customer site where you can search for products they sell url format is: domainname/search/keywords/ keywords being what the user has searched for. This means the number of pages can be limitless as the client has over 7500 products. or should I simply rel canonical the search page or simply no follow it?
Technical SEO | | spiralsites0 -
Managing international sites, best practises
This question follows on from my earlier question http://www.seomoz.org/q/how-to-replace-my-co-uk-site-with-my-com-site-in-the-us-google-results My client owns www.blindbolt.co.uk for the UK site and www.blindboltusa.com for their US site. They will shortly be having a new site for Australia. They have just acquired www.blindbolt.com and have expressed an interest in using this as the main hub for all of their sites, i.e. http://uk.blindbolt.com, http://aus.blindbolt.com. The current, existing sites (e.g. www.blindbolt.co.uk) could be 301'd to the new locations. Could I have your thoughts please on whether to go down this route of having international subdomains , vs keeping the sites on separate top level domains? What should I take into consideration? Is google smart enough to return different subdomain results in different countries? Many thanks!
Technical SEO | | OffSightIT0 -
Mobile SEO Tips and Best Practices
Hi, Any advice on approaches for mobile SEO much appreciated. For example, what things do I need to do to optimise a mobile version of a desktop site? E.g. optimise titles, meta description, headings and copy, do I need to submit a mobile sitemap to Google? Do I need to link from the mobile page to the corresponding page on the desktop site and vice versa? Will Googlebot Mobile naturally find the site from the desktop link? What about link building for mobile sites, any thoughts on this, are there specific mobile sites that will link such as directories for a start? Any other tips or resources? Any SEOMoz resources on this? What about the same domain versus the subdomain debate about hosting the mobile site? Any thoughts? Many thanks
Technical SEO | | MarkChambers0