Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Other Category Pages Impression Down
My Domain is http://www.grandcrayon.ca/ We have more than 40,000 Office supplies products pages including 200+ categories and we get most of impression on only home page with keyword office supply, office supplies Canada, grand crayon office supplies Why our other category pages have no impressions, My targeted Keywords which have impression on homepage Cheapest Office Supplies, Stationary Supplies Canada Please suggest to enhance impression for other category pages impression.
Industry News | | CommercePundit0 -
Article marketing sites
Hi, I'm looking for article marketing sites in English. I have searched and analyzed over 50 sites but I have not find any with the specifics I'm searching to: high/average DA, dofollow, and the possibility to have an anchored text, any suggestions? Thanks!
Industry News | | eriksatie0 -
Best Site or Place to Hire Mid Level SEO Specialists
Hello Mozzers, Our digital agency has been continuously growing which is a good thing BUT recruiting for Junior or Mid Level SEO Specialists is getting harder in our area (we are in Orlando Florida). We post on Indeed.com but the level of knowledge and also the amount of candidates is not a lot. Do you know any other site that only pertains to our industry? Any suggestions? In fact, its been hard , we are even thinking of developing a job board site just for our niche. Your recommendation and thoughts?
Industry News | | ChatterBuzzMedia0 -
Google penalty removal expert questions
We have searched online for a Google penalty “expert” (individual or company) and have located what appear to be “experts”. Please provide feedback on the following 2 individuals/companies we have found that can help with penalty removal. Have you or one of your clients used either of the “experts” below? What were the results? How many disavows and reconsideration requests did you/they have to make? 1.www.penaltypros.com . To give a quote and to see what your links are they use links from Google Webmaster Tools only. Penaltypros.com disavows first and then removes bad links second. This is opposite of what Google and Seo’s recommend but penaltypros.com claims 100% success using this non-traditional approach. See imgur.com link for screenshot. 2.http://www.hiswebmarketing.com/ To give a quote and to see what your links are they use links from https://ahrefs.com/ only. Please provide any and all feedback on the above 2 “experts” and also post the websites, individual names, company names of those that you consider Google penalty removal “experts” so that we may obtain a quote from them. Lp9F3FI
Industry News | | RetractableAwnings.com1 -
What's an example of an SEO firm's site that lists their SEO packages excellently
Hello, I'd like to make a packages page on the site I'm still building bobweikelseo.com What's an example of a firm or SEO that lists their packages in a way that is appealing, useful, and all around excellent? I'm thinking of having a first package for analysis at $440, a second package for analysis and on-site SEO for $880, and a third package for analysis, on-site SEO, and link building for $2000-$4000/month depending on how fast they want the link building to go. Looking for some input on how to create a packages page, if my prices are appropriate (I want to be affordable), and an example of a website that does an excellent job at presenting their packages.
Industry News | | BobGW0 -
Can not get the ranking needle to move even after a year with a US based SEO company
Hello community, I have spent the last 16 months spending lots of money on our USA Based SEO company and on content. For some reason. The powers who may be at Google are not giving us any love. I am beginning to become very dissapointed after all of our efforts. We did have a link penalty a year ago however that has long ago been removed. We have been doing everything our SEO company has asked us but the needle is not moving. Any ideas for us? http://tinyurl.com/n2yoeyn Thank you in advance. Chris
Industry News | | scamper0 -
Do rankings return to previous rank after a site has been un-indexed?
My client has had a site since 1996 (the first one was a www subdomain, his current site has no www). He came to my company for SEO about a year ago, after having a WordPress site that was accidentally set to not be crawled/indexed by search engines. We set the site to be indexed and added a blog. Since then, he has had slow increase in keyword rankings (on page 3 or most globally searched terms) but is still nowhere near page 1 for most of his keywords. He thinks my company has done nothing, while I see the keyword improvements as a sign of success - slow, gradual - but going in the right direction. I don't know if he was ranking well before he had a site that was not set to be indexed - but if he was on page one, would he eventually return there? Or do we have to build everything over again?
Industry News | | Alex_Ratynski0 -
Am I getting traction, spinning my wheels, or just spinning in circles?
My story: recently "promoted" to creating our marketing department and efforts despite lack of expertise; new to SEO as a daily/weekly task; focusing a lot of effort polishing current webpage (redesigned last year); Hit most of the low hanging fruit in "Beginners Guide"; asked enough questions to know that now I need to focus on content at this point. My conundrum: There is **so **much information out there about SEO, that keeping focus on the tasks at hand is incredibly difficult. KPIs, Trends, Research everything I read serves to derail me from what might be my most critical tasks (content creation?) SEO is like my business (solar), you need to be as site specific in your design and install as possible, but only generalized information is available to the novice. My questions for you experienced SEOs: how do you keep focus? How do know that the project you're focusing on is the right one? Do y'all know of a resource for consultation/advising to look at my initial efforts so someone can help me navigate and gain perspective? I'm not looking for someone to solve my problems (although that would be nice), but more of a "grade" (ie-less of this, more of that, nice work here, try again over there) Thanks folks, any and all help is appreciated.
Industry News | | wreevesc1