Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Tag Manager
What are some of the best resources with learning and teaching other GTM and conversion tracking?
Industry News | | WebMarkets1 -
Yelp (recrawl Google/Bing)
If Google and Bing show an older version of a site's Yelp rating in the search results, what options are there to help ensure Google and Bing recrawl the Yelp page? Additionally, it appears third-party sites such as MapQuest show Yelp ratings and appear in Google search results; is it possible to request MapQuest to recrawl Yelp and then ask Google to recrawl MapQuest? Any advice would be much appreciated!
Industry News | | Mack_1 -
Spanish equivalent of MOZ? need help with understanding Google.es
I try to keep up to date with everything SEO to ensure my sites rank well.. marblerenovation.com & Trailersafe.co.uk I now have a spanish site that I am having trouble ranking well for clima2000egara.es I have recently found out (by asking our lovely moz community) that googles algorithms are not rolled our world wide. SO I have no idea where I am at with Google.es. Does anyone know how I can find out at what stage google.es is at? and what the last update was? and whether there is an equilalent community like Moz for Spain? I am fluent spanish so the language won't be a problem.. Thanks in advance for all your help
Industry News | | david.smith.segarra0 -
Google Penguin 2.0 - Coming soon
There is an interesting article on SEW that Google is going to update Penguin to the next major version - http://bit.ly/15Vkr6O So what do you think, what should we expect? And also, is there available updated webmaster guidelines?
Industry News | | ditoroin1 -
Using 'Pipes' for Page Titles
A couple of years ago I was told to take off the 'pipes' for my page titles. To be more organic ....now I am being told it is best to put them back. For example I just read it is a good idea to do the following: 'Using pipes to separate these words would look like: SEO Company | Franchise Services | Higher Visibility Although this might not be the creative title you’re used to seeing on blogs, it’s a title that is incredibly optimized for Google (yet it still lets readers know what they can expect).' Should I replace the titles of my poor performing pages, with the common format which I am told is: Page Branding | keyword 1 |keyword 2 or keyword 1 | keyword 2 | Page branding. And I should keep the piping. Any opinions?
Industry News | | Llanero0 -
I need some help with google please
Hello i am trying to remove google partial penalization to my website since july 2012 I have removed link, used disallow tools, wrote letters Last comunicacion was a letter send on 12/2012 that they responded 3/2013 web in cuestion is www.propdental.com I have not answered yet because i am looking for a good profesional help they responded saying that i still have unnuturall links wich is pretty probably but i dont nkow wich ones remove. Site authority has dropped from 43 to 32 with all links removed traffic dropped from 48000 to 15.000 Please profesional contacts needed
Industry News | | maestrosonrisas0 -
So, Google is the best site on the internet.. Right? Or is that just what most people tend to think off-face?
LOL woah, put the guns away. I'm not about to rant, I just have a question and wanted to present it well. Then again, I might have actually found some easy fixes to some of Google's tools that they could make. So here's the thing. I noticed how annoyed I always getting when I have to sign in every time I go to the adwords keyword tool, or analytics. Why do you have to sign in a million times? I think it is a problem that can be fixed because if you go to check your webmaster tools, you go straight into your account, where you can then select which site you want to explore. It knows that I am already signed in to Google Accounts when I go to webmaster tools, but it doesn't recognize that fact when I go to my Analytics account, or to use the Adwords Keyword Tool. Now, every site has things that they need to work on, but not necessarily that need to be 'fixed'. Google being so commonly accepted as the best site on the net, I thought it was funny/interesting at the least to point out the problem. Even funnier is the fact that I could submit it as a problem to see if they could fix it or not, but they do such a good job of making it hard for people to contact them, that A) I don't feel like wasting my time trying B) I don't even really know if it is possible to do that. Also, why is there no official Google Analytics App / Mobile site?? Google has been pushing how important mobile is to us webmasters, but then it doesn't seem to be very high on their priority list for the tools that we use. I mean you can't view graphs on phones / tablets (mine at least), in webmaster tools, OR google analytics. Also, its a pain in the but to click the sign in button on Google Analytics when using my phone / tablet, it disapears really fast for me (needs more research from others to see if everyone has the same problem) Thanks for the interest / answers everybody. Look forward to hearing from you guys. Also, tips and help would be nice if anybody knows a solution to my sign in issue
Industry News | | TylerAbernethy0 -
Does anyone have a copy of the 2011 Google Quality Raters Handbook that was recently leaked?
http://searchengineland.com/download-the-latest-google-search-quality-rating-guidelines-97391 Google has been on a conquest taking them down online but I would really like to take a look at it if you have a copy! [moderator note - please use the PM system and exchange email addresses there. We've removed emails from this thread before it gets indexed and exposed to the world]
Industry News | | altecdesign4