Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
International pages - SEO - which metatags to use?
I'm trying to get my International pages set up correctly for SEO
Industry News | | MikeSEOTruven
Can you tell me which of the following meta-tags are the ones to use on the pages?
I've heard that some might be obsolete, so will it hurt if I throw on all 3 or just choose 1? Example: Italian language page0 -
Spanish equivalent of MOZ? need help with understanding Google.es
I try to keep up to date with everything SEO to ensure my sites rank well.. marblerenovation.com & Trailersafe.co.uk I now have a spanish site that I am having trouble ranking well for clima2000egara.es I have recently found out (by asking our lovely moz community) that googles algorithms are not rolled our world wide. SO I have no idea where I am at with Google.es. Does anyone know how I can find out at what stage google.es is at? and what the last update was? and whether there is an equilalent community like Moz for Spain? I am fluent spanish so the language won't be a problem.. Thanks in advance for all your help
Industry News | | david.smith.segarra0 -
Google Cached "Text Only" version
Is there a way to test what a page would look like in Google "Text Only" version before a page is indexed in Google? Is there a tool out there to help with this?
Industry News | | activejunky10 -
Google Search Quality Team - Commission Based Reviews
I have been busy this past week writing articles for various sources about the recent update on Google. A number of people contacted me about the analysis I was doing and the report. Some were members of the Google Search Quality Team. I knew manual reports were done before - but after the documents they showed me regarding the reports they do and the compensation for doing the reports - I am left in a state of being pretty shocked. May be I have been naive for all these years but I didn't realize that; Google outsourced the review and reconsideration requests to individual reviewers for a compensation Google's position in terms of checking qualification and experience of these "reviewers" was very insufficient at best, The three contacts I spoke to who had done reports had very little training or experience. I went through the GSQT REVIEWERS PDF (a very long and thorough document) that I was sent - with them. We went together through some sites I wanted them to review and their comments that came back were quite astounding to say the least and would have made many of you Mozzers laugh. Obviously I don't want to post said document online here.... BUT, I wanted to know if: a) any Mozzers had ever been part of such a group - the GSQT b) had any dealings with them - in terms of having your website reviewed and known about it. I knew about this group way back - like in 2005 or 2006 or sometime around then - I was told at time it was stopped and Google had stopped paying these sub contractor reviewers. Please don't get me wrong here... totally on board with manual reviews... I would just prefer them done by a trained team that possibly worked for either a professional company that maintain high quality review testing and standards - or for that matter GOOGLE employees that were trained. I just am a little unsure of them being done by individual subbies that get paid for the amount they do. What if that subbie has got some skin in the game for a particular keyword? What if their knowledge about certain aspects isn't up to par or not tested on a regular basis. This space is always changing and as you guys ./ girls on this forum know - it can change pretty quick. I just would want all websites to be judged fairly and equally by a group trained EQUALLY and to the same standards. I don't care if this is a G team or not - I just want it to be a team that is trained equally and trained continuously as opposed to paying outside people based on numbers of reviews done. When the livelihood of a small business is the balance I don't want a commission hungry toe rag with one years experience being the gate keeper for me or any of our clients. Carlos
Industry News | | CarlosFernandes0 -
Google Product Feeds - New Requirements
We are in the jewelry industry, and for Google product feeds, we list our products under "Apparel & Accessories > Jewelry". As of the new Google feed requirements, they are saying that we have to choose a gender and color for each product that is in the Apparel category. While this makes sense for clothes, it doesn't exactly for jewelry because many items are for both men and women, and there's not always a color associated with each product. I can enter some of these fields manually, but with 5,000+ products, it makes it difficult w/ each update. Anyone have solutions for this? Or a way around it? Can we just include those fields but leave them blank? Any other solutions?
Industry News | | applesofgold1 -
What is the best site-map software?
I am in the beginning stages of building a website and would like to know the site-mapping software for organizing my company's ideas. Any recommendations are greatly appreciated. Thanks!
Industry News | | canadianplum0 -
Rebranding Sites
Our company just went through a rebranding and this includes the web sites. We have a new website (same domain) hosted on a new sever. We implemented 301's for outdated domain names and urls. The content has been overhauled to be simpler and we are building this from ground up with SEO in mind.I have canonical tags, being mindful of follow and no follow, with out trying to page sculpt) and ensuring the urls are descriptive and free of auto generated cms trash. To prepare for the launch, I captured all the analytics and adword campaigns before the switch. and began making lists of where we need to change the name around the web. We were performing pretty well, on SERPS before hand and now we want to try to keep the momentum speeding up with a cleaner newer site. Does anyone have any more suggestions on what more I should be doing to start off on the right foot.
Industry News | | KJ-Rodgers0 -
Schedule for Panda / Farmer Update rollout for non English sites?
Hey Mozzers, I just read about the global rollout of googles Panda/Farmer Update for English queries, e.g. at searchengineland. Has anyone read something about when google is planning to roll out the update for non-English speaking countries, i.e. queries in other languages? I googled a bit but couldn't find anything, not even speculations. Cheers, Frank
Industry News | | FranktheTank-474970