NoIndex user generated pages?
-
Hi,
I have a site, downorisitjustme (dot) com
It has over 30,000 pages in google which have been generated by people searching to check if a specific site is working or not and then possibly adding a link to a msg board to the deeplink of the results page or something which is why the pages have been picked up.
Am I best to noindex the res.php page where all the auto generated content is showing up and just have the main static pages as the only ones available to be indexed?
-
Yes, add it to the robots.txt (use a Disallow and a NoIndex statement). I did find that Bing for example has not reliably in the past honoured robots.txt (especially in the case where you have an explicit "index" tag on the page and a noindex for a URL path).
-
Thanks for the replies Gerg & Irving.
The robots.txt block/exclude I take it I can just do that to the res.php page and not have to individually for the 30k generated dynamic pages off it (probably a silly question I know but wanted to double check).
-
I'd noindex the page, block in robots.txt, make sure your sitemap.xml is not generating these URLs if automated, and if there is a main folder where all the user generated pages are then request removal of that content in Google WMT.
-
I would also exclude them via robots.txt and then push through a sitemap with your static content to "nudge" Google to recrawl your content (and hopefully drop the other pages off quickly over time).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Home Page Ranking Instead of Service Pages
Hi everyone! I've noticed that many of our clients have pages addressing specific queries related to specific services on their websites, but that the Home Page is increasingly showing as the "ranking" page. For example, a plastic surgeon we work with has a page specifically talking about his breast augmentation procedure for Miami, FL but instead of THAT page showing in the search results, Google is using his home page. Noticing this across the board. Any insights? Should we still be optimizing these specific service pages? Should I be spending time trying to make sure Google ranks the page specifically addressing that query because it SHOULD perform better? Thanks for the help. Confused SEO :/, Ricky Shockley
Technical SEO | | RickyShockley0 -
HTTP Vary:User-Agent Server or Page Level?
Looking for any insights regarding the usage of the Vary HTTP Header. Mainly around the idea that search engines will not like having a Vary HTTP Header on pages that don't have a mobile version, which means the header will be to be implemented on a page-by-page basis. Additionally, does anyone has experience with the usage of the Vary HTTP Header and CDNs like Akamai?Google still recommends using the header, even though it can present some challenges with CDNs. Thanks!
Technical SEO | | burnseo0 -
Rel canonical for partner sites - product pages only or also homepage and other key pages?
Hello there Our main site is www.arenaflowers.com. We also run a number of partner sites (eg: http://flowershop.cancerresearchuk.org/). We've relcanonical'd the products on the partner site back to the main (arenaflowers.com) site. eg: http://flowershop.cancerresearchuk.org/flowers/tutti_frutti_es_2013 rel canonicals back to: http://www.arenaflowers.com/flowers/tutti_frutti_es_2013). My question: Should we also relcanonical the homepage and other key pages on partner sites back to the main arenaflowers website too? The content is similar but not identical. We don't want our partner sites to be outranking the original (as is the case on kw flower delivery for example). (NB this situation may be complicated by the fact we appear to have an unnatural link penalty on af.com (and when we did an upgrade a while back, the af.com site fell out of the index altogether due to some issues with our move to AWS.) We're getting professional SEO advice on this but wondered what the Moz community's thoughts were.. Cheers, Will
Technical SEO | | ArenaFlowers.com0 -
Page feedback
We recently wrote a new website page to cover the direct mail services our organization offers. We kept the title tag to 70 characters, the meta description under 150 characters. H1 tag has what we feel is the most important term. If anyone out there has time to review & provide a little feedback, we'd really appreciate it. It would be great to know if it is built well and providing a solid end user experience. http://www.cushingco.com/print_products/additional_services/direct_mail.shtml At the moment, the only links pointing to this page are from our blog. One bit of content I am contemplating is a short paragraph - What is Direct Mail Marketing? Literally providing a short definition of it. The page was activated last Thursday and showing up in some Google results on the 4th/5th page but I am thinking this is probably just a temporary bump for now. Anyway, thanks in advance for any advice!!!
Technical SEO | | SEOSponge0 -
According to 1 of my PRO campaigns - I have 250+ pages with Duplicate Content - Could my empty 'tag' pages be to blame?
Like I said, my one of my moz reports is showing 250+ pages with duplicate content. should I just delete the tag pages? Is that worth my time? how do I alert SEOmoz that the changes have been made, so that they show up in my next report?
Technical SEO | | TylerAbernethy0 -
Too many on page links
Hi All, As we all know, having to much links on a page is an obstacle for search engine crawlers in terms of the crawl allowance. My category pages are labeled as pages with to many "one page" links by the SEOmoz crawler. This probably comes from the fact that each product on the category page has multiple links (on the image and model number). Now my question is, would it help to setup a text-link with a clickable area as big as the product area? This means every product gets just one link. Would this help get the crawlers deeper in these pages and distribute the link-juice better? Or is Google smart enough already to figure out that two links to the same product page shouldn't be counted as two? Thanks for your replies guys. Rich
Technical SEO | | Horlogeboetiek0 -
Product category paging
Hi, My product categories have 2-3 pages each. I have paging implemented with rel=next and rel=prev. from some reason Google GWT now reports the pages as having duplicate titles and description. Should I be worried? Should I set a different title like "blue category - page x" ? Thanx, Asaf
Technical SEO | | AsafY0