Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google penalty removal expert questions
We have searched online for a Google penalty “expert” (individual or company) and have located what appear to be “experts”. Please provide feedback on the following 2 individuals/companies we have found that can help with penalty removal. Have you or one of your clients used either of the “experts” below? What were the results? How many disavows and reconsideration requests did you/they have to make? 1.www.penaltypros.com . To give a quote and to see what your links are they use links from Google Webmaster Tools only. Penaltypros.com disavows first and then removes bad links second. This is opposite of what Google and Seo’s recommend but penaltypros.com claims 100% success using this non-traditional approach. See imgur.com link for screenshot. 2.http://www.hiswebmarketing.com/ To give a quote and to see what your links are they use links from https://ahrefs.com/ only. Please provide any and all feedback on the above 2 “experts” and also post the websites, individual names, company names of those that you consider Google penalty removal “experts” so that we may obtain a quote from them. Lp9F3FI
Industry News | | RetractableAwnings.com1 -
Are large property portals going to continue to dominate Google's search results?
We are having a discussion (potential argument) in our office around whether large portals (namely property portals) have longevity in Google's search. Argument 1:
Industry News | | NeilPursey
Google's rise in local search and rewarding strong brand names rather than keyword driven domain names will devalue property portals with keyword rich domain names. Property portals are essentially duplicating content on smaller individually owned property websites, therefore in time Google will devalue property portals. **Argument 2: **
Property portals have more property stock listed on their websites so therefore Google will reward them by ranking these websites higher than the smaller real estate agencies with niche stock in their areas that they operate in. The property portals that already are already in a dominate position already carry authority and their own sense of branding, therefore it's difficult for Google to ignore them. If we assume that Google is looking into user behaviour as a ranking factor, then this will help portals as they have more stock which means higher engagement on the website. I'd love to read the moz community thoughts and opinions on this. I reckon it's a worthy debate..0 -
Best ranking Magento sites?
What would be your top 10 Magento sites in terms of SEO and ranking? I'm looking for examples of Magneto sites that have been set up correctly and are ranking well. Especially recently launched sites that have managed to get themselves ranking well in competitive markets! Cheers
Industry News | | OnlineAssetPartners0 -
How do I ascertain how/why a site appears higher/lower in search results, based on different search terms?
The site in question is www.bullethq.comHere are the search terms I used, and what position Bullethq.com appears in the SERP's: <colgroup><col width="188"><col width="58"><col width="95"></colgroup>
Industry News | | PeterConnor
| Search-term | Position | Page returned |
| irish payroll online | 5 | Home Page |
| irish payroll online software | 20 | Home Page |
| online irish payroll software | 20 | Home Page |
| online payroll | 75 | (Blog post) | Could someone be so kind as to help me figure this out?0 -
We noticed that goods offered in our email newsletters used to disappeared from first google search results page!?
We noticed that goods offered in our email newsletters used to disappeared from fisrt google search results page. Goods where in top 5 positions or even higher, but after email newsletters we didn't find them even in top 100. We suspect service provider of email sending is in blacklist? Could it be reason? If yes, how could we check that?
Industry News | | Patogupirkti0 -
What is triggering Google account suspensions?
Over the past 24 hours many of our clients have had their Google accounts suspended. The explanation has been: "After reviewing your profile, we determined that it has been used to impersonate another individual or mislead other users. This violates the Google+ User Content and Conduct Policy." We are NOT impersonating our clients, we have their permission. We are not misleading anyone, simply setting up profiles for our clients on Google+. This has not affected all of our clients, but a significant number of them. We cannot find a common variable between the clients that have been suspended, and those who have not. Some have had other Google+ profiles in the past, in another account, some have not. Some have been previously verified via SMS, others by phone. Some have posts in their profile, others have only the profile info filled out. Again, we are not trying to game Google, we are simply setting up authorship for them, with their permission. I have not seen much in the SEO community about this today, and this is NOT related to fake reviews. We do not partake in that kind of activity. We have written a post on the topic, and no matter how this shakes out, I think our take is solid. Authorship is changing the game, content is changing the game, trust is changing the game… and Google is getting serious about it. We have also seen this happen to our clients, to our competitors' clients, and to other marketing firms' clients, outside of our vertical. Does anyone know more about the topic, especially in regards to the suspensions over the past 24 hours?
Industry News | | Einstein-Industries0 -
Any pointers for my site would be greatly apprecieated and rewarded.
We just started changing all the meta data and adding relevant blog content. Anything I'm missing? Any advise or "after panda" link building strategies? inkfarm.com
Industry News | | ibex0 -
Google's New Release "What do you love"
Is this gonna be a game changer for SEO http://www.wdyl.com/ Regards, Shailendra Sial
Industry News | | IM_Learner0