Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Not Cached By Google
Hi My name is Apoorwa and i have my own website, My website is not cached by google, Why is this problem happening with my site.can somebody help me please? its urgent....this is my site - http://www.holifestival.org/Please assist me.......
Moz Pro | | Packersmove0 -
Listing of all Google Indexed Pages
I started managing a site that has about 391,000 indexed pages. I want to get to the bottom of why there are so many in preparation for a ecommerce Migration and improving SEO. Anyone know of a tool? Many tools I have came across can only take 100 at a time. I would love to get them in excel or a database. I look forward to the suggestions.
Moz Pro | | dhands170 -
Does SEOmoz tool provide me the option to track rankings for other search engines like google.co.uk?
I need to track rankings on search engines across different geographies. Does your tool provide the option to track rankings for search engines other than the ones used in US
Moz Pro | | zycus0 -
Google and Open Site Explorer not showing as many links
I've noticed this past week that when you search for the links pointing to a given site, by using the "link:" operator, that Google not showing as many links as they use to. I noticed this also with Open Site Explorer, it is not showing the detail link information as much as it did before. Is Google trying to mask what we can view now on competitors backlinks? If so, how can we see the backlink building that our competitors are doing?
Moz Pro | | tdawson090 -
SEOMOZ stopped counting local results as a rank?
I had a terrifying experience today upon receiving my weekly rankings report that stated I had dropped out of google for the majority of my keywords! However upon further inspection it would appear the report is not including local results in the ranking (my page shows up #1 but is a local result). This didn't happen before, did something change that I didn't know about?
Moz Pro | | r1200gsa0 -
Local search block seems to mess up ranking calculations!
Time and time again i see SEO moz come out with a new rankings report and tell me that I've gone up or down in the rankings by 9 and I get really excited. Then i go look at the search and once agian we are in the exact same position! What it seems like is sometimes SEO Moz decides to count the local search block, and other times it decides not too. Is there any way to fix this? To make it always count the local search block would be preferred.
Moz Pro | | adriandg0 -
Is there a beginners guide to using the tools and what do the results mean.
Great set of tools! But what would make them better is some type of guide to what the tools do (remember I am a new user) and how to maximize the results. Oh and while your at it what do the results mean.
Moz Pro | | marcreece0