How to resolve - Googlebot found an extremely high number of URLs
-
Hi,
We got this message from Google Webmaster “Googlebot found an extremely high number of URLs on your site”. The sample URLs provided by Google are all either noindex or have a canonical.
- http://www.myntra.com/nike-stylish-show-caps-sweaters
- http://www.myntra.com/backpacks/f-gear/f-gear-unisex-black-&-purple-calvin-backpack/162453/buy?src=tn&nav_id=541
- http://www.myntra.com/kurtas/alma/alma-women-blue-floral-printed-kurta/85178/buy?nav_id=625
Also we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters.
Your comments on how to resolve this issue will be appreciated.
Thank You
Kaushal Thakkar
-
Hi Kaushal,
Thanks for the question.
There are a few ways to deal with this problem which are recommended by Google here. In summary, you can:
- Use parameter handling as you have done
- Add the nofollow attribute to problematic URLs
- Block problematic URLs in robots.txt
There is also a thread in the Google webmaster forums which may be useful to you:
Overall, it comes down to having a good site architecture and cutting down / removing / blocking URLs that you don't care about from a search perspective.
I hope that helps a bit!
Paddy
-
Thank you David, Its been more than 10 months since these parameters have been specified in webmaster. This and other activities like noindex and canonicals helped us to reduce the indexed URL count from 32 million to 1.2 million. As the url index reduced this warning from google stopped for 4 months. However we started receiving this message again from february 2014.
Thanks
Kaushal
-
"we have specified the parameters on these URLs as representative URL in Google Webmaster - URL parameters."
How long ago was this done? Since there are so many URL's, it may take a while for them to recrawl and index the representative URL's per your request.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
High ranking nationally but not locally via google
A website I am working on is ranked very well in all tracked keywords at a national level, but not from a local standpoint via google. I find it weird that the site is on the first page if you search from many other states/towns/locations but not locally. Looked on Google Search Console and couldn't see any link to why this is happening. Figured we would clear out the htaccess for any redirect issues and hope it fixes it. Suggestions please? Never seen google do this. It is strange.
White Hat / Black Hat SEO | | SeobyKP1 -
Best URL structure for SEO for Malaysian/Singapore site on .com.au domain
Hi there I know ideally i need a .my or .sg domain, however i dont have time to do this in the interim so what would be the best way to host Malaysian content on a www.domainname.com.au website? www.domainname.com.au/en-MY
White Hat / Black Hat SEO | | IsaCleanse
www.domainname.com.au/MY
domainname.com.au/malaysia
malaysia.domainname.com.au
my.domainname.com.au Im assuming this cant make the .com.au site look spammy but thought I'd ask just to be safe? Thanks in advance! 🙂0 -
Where is the rule here that forces any home page URLs that include several versions of explicit page name references, such as default.htm or index.html, to redirect to the canonical homepage?
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)/(1)?$ /search/$1/sort-$2 [R=301]
White Hat / Black Hat SEO | | esiow2013
RewriteRule ^search/([a-zA-Z0-9,-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$5&sort=$2 [L]
RewriteRule ^search/([a-zA-Z0-9,-]+)/(1)?$ /search/$1 [R=301]
RewriteRule ^search/([a-zA-Z0-9,-]+)((/)([0-9]+))?$ index.php?cmd=search&q=$1&page=$4 [L]
RewriteRule ^([r|R])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=1 [L]
RewriteRule ^([s|S])-([0-9]+)((-)([0-9a-zA-Z-]+))?$ index.php?cmd=propertyPage&id=$2&group=2 [L]
RewriteRule ^reset-password$ index.php?cmd=retrievePassword [L]
RewriteRule ^reset-password/([0-9]+)/([a-zA-Z0-9]+)$ index.php?cmd=retrievePassword&act=retrievePasswordConfirm&code=$1&key=$2 [L]
RewriteRule ^map-page.php$ index.php?cmd=mapPage [L]
RewriteRule ^advertise.php$ index.php?cmd=advertiseWithUs [L]
RewriteRule ^my-account.php$ index.php?cmd=protectedMyAccount [L]
RewriteRule ^bookmarked-properties(/((R|S)-([0-9]+)))?$ index.php?cmd=protectedBookmarkedProperties&id=$2 [L]
RewriteRule ^bookmarked-properties/([0-9]+)$ index.php?cmd=protectedBookmarkedProperties&page=$1 [L]
RewriteRule ^bookmarked-properties/delete/([0-9,]+)$ index.php?cmd=protectedBookmarkedProperties&act=protectedBookmarkedProperties_delete&id=$1 [L]
RewriteRule ^register.php$ index.php?cmd=register [L]
RewriteRule ^about-us.php$ index.php?cmd=aboutUs [L]
RewriteRule ^news(/([0-9]+)(.*))?$ index.php?cmd=news&id=$2&title=$3 [L]
RewriteRule ^registration/confirm/key=([a-zA-Z0-9]{150})/code=([a-zA-Z0-9]{5})$ index.php?cmd=register&act=registerConfirm&key=$1&code=$2 [L]
RewriteRule ^terms-and-conditions.php$ index.php?cmd=termsAndConditions [L]
RewriteRule ^privacy-policy.php$ index.php?cmd=privacyPolicy [L]
RewriteRule ^contact-us.php$ index.php?cmd=contactUs [L]
RewriteRule ^sitemap.php$ index.php?cmd=siteMap [L]
RewriteRule ^members/login.php$ login.php [L]
RewriteRule ^rss$ index.php?cmd=rss [L]
RewriteRule ^resources.php$ index.php?cmd=links [L]
RewriteRule ^links.php$ /resources.php [R=301]
RewriteRule ^resources2.php$ index.php?cmd=links2 [L]
RewriteRule ^results.php$ index.php?cmd=results [L]
RewriteRule ^residential-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&page=$3 [L]
RewriteRule ^commercial-properties((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=2&page=$3 [L]
RewriteRule ^short-term-rentals((/)([0-9]+))?$ index.php?cmd=divisionsRoot&mainDivision=1&str=1&page=$3 [L]
RewriteRule ^buy((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=2&page=$3 [L]
RewriteRule ^rent((/)([0-9]+))?$ index.php?cmd=divisionsRoot&group=1&page=$3 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&sort=$2&page=$5 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&sort=$4&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&sort=$5&page=$8 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&str=1&page=$7 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)/([a-z_-]+)/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&str=1&page=$6 [L]
RewriteRule ^short-term-rentals/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&str=1&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&sort=$5&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&sort=$2&group=2&page=$5 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=1&page=$7 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1®ion=$2&city=$3&area=$4&group=2&page=$7 [L]
RewriteRule ^rent/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/residential-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$1&group=2&page=$4 [L]
RewriteRule ^lease/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/commercial-properties/([a-z_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$1&group=2&page=$4 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=1&page=$8 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z,()0-9_-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2®ion=$3&city=$4&area=$5&category=$1&group=2&page=$8 [L]
RewriteRule ^rent/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/residential-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=1&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^lease/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=1&page=$5 [L]
RewriteRule ^buy/commercial-properties/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&mainDivision=2&country=$2&category=$1&group=2&page=$5 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()0-9_-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&sort=$6&page=$9 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&sort=$5&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&area=$5&category=$1&page=$8 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/([a-z()-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2®ion=$3&city=$4&category=$1&page=$7 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)/sort-([a-z0-9-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&sort=$3&page=$6 [L]
RewriteRule ^rent/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=1&country=$2&category=$1&page=$5 [L]
RewriteRule ^buy/([a-z,-]+)/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain&group=2&country=$2&category=$1&page=$5 [L]
RewriteRule ^rent/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=1&page=$4 [L]
RewriteRule ^buy/([a-z()-]+)((/)([0-9]+))?$ index.php?cmd=divisionsMain_categories&mainDivision=1&country=$1&group=2&page=$4 [L]0 -
Suggestions in redirecting an old URL to new URL with parenthesis () ?
What should I use in .htaccess if I will redirect an old URL with parentheses to a new URL like below? RedirectMatch 301 http://www.olddomain.com/buy/nike-shoes/kobe(7)/red http://www.newdomain.com/buy/nike-shoes/kobe(7)/red Or RedirectMatch 301 http://www.olddomain.com/buy/nike-shoes/kobe(7)/red http://www.newdomain.com/buy/nike-shoes/kobe(7)/red
White Hat / Black Hat SEO | | esiow20130 -
URL structure: 301 redirect or leave as is?
Hello, My website, www.coloringbookfun.com is very old and authoritative, but the URL structure is terrible. If you check out some of our subcategories such as http://www.coloringbookfun.com/Kung Fu Panda and individual printables such as http://www.coloringbookfun.com/Kung Fu Panda/imagepages/image2.html You can see that they aren't optimized. I am curious to know the pros and cons of fixing the URL structure and 301ing them to the new optimized url. Will 301ing lose authority and backlinks for the sites pages? Does optimizing the url structure outweigh losing the authority/backlinks?
White Hat / Black Hat SEO | | WebServiceConsulting.com0 -
Solved PayDay hack - but SERPs show URLs - what should I do?
We had the PayDay hack - and solved it completely. The problem is - the SERPs have over 3,000 URLs pointing to 404 on our website all of which have urls that are like this: <cite>www.onssi.com/2012/2/post1639/payday-loan-companies-us</cite> What should I do? Should I disavow every one of the 3,000? No Follow?
White Hat / Black Hat SEO | | Ocularis0 -
Starting fresh on a new url after serious Penguin update down rank
Hi friends My site www.acupunctureclinicvictoriabc.com was recently hit by the penguin update and i dropped to page 5 of local searchs for my key words. A while back I had some bad link building done and now paying for it:( I thought the disavow tool (used 4 months ago) would deal with this issue but apparently not The current url is feeling like a lost cause. My question is if I start fresh on a new url, can I use my old content (or even clone the site and move it to a new url) without being punished for duplicate content on the new site? Any recommendations for starting fresh? I really appreciate any thoughts on this matter, as I am feeling a bit lost and bummed about this issue thanks!
White Hat / Black Hat SEO | | Silasrose0 -
Penguin Update or URL Error - Rankings Tank
I just redid my site from Godaddy Quick Shopping Cart to Drupal. The site is much cleaner now. I transferred all the content. Now my site dropped from being in the top ten on almost every key word we were targeting to 35+. I "aliased" the urls so that they were the same as the Godaddy site. However when I look at our search results I notice that our URLs have extra wording at the end like this: ?categoryid=1 or some other number. Could this be the reason that our rankings tanked? Previously on the godaddy site the results didnt show this.
White Hat / Black Hat SEO | | chronicle0