Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Website Grader Export Omits Check Graphics
So, I love the Grader, but when I export to PDF the pretty little graphics go away. Is this correctable?
Moz Pro | | ClickMonsterIM0 -
Crawl results on keyword rankings differ from actual search rankings
For the second week in a row, the weekly keyword rankings in my MOZ account show "not in top 50" for most keywords. However, when I check some of my rankings I still broadly have the rankings that I used to have. It seems that the MOZ crawl is experiencing some difficulties. Does anyone have suggestions how to find out what's going on? If I can't rely on the crawl outcomes, then the tool is not very useful anymore!
Moz Pro | | matteis0 -
Google shutting down Rank Tracking Software? - Raven and AHREFS close down ranking results.
Hi, I was wondering if other people had noticed the (very sudden) closing down of the ranking tools for both Raven and Ahrefs - both of which were major parts of their software. Given the sudden and synced timing my guess is that Google is throwing its weight around and banking on the doors of companies offering ranking software with the threat of a fat lawsuite! What is everyone else using for their rank checking ? Will this impact SEOMOZ ranking results ?
Moz Pro | | James770 -
Google Analyitics Average Position for Queries
Why is the Average Position in the Queries view of Google Analytics SEO report so different on each query from the position shown by SEOmoz?
Moz Pro | | rfaulk0 -
Crawl Diagnostics - unexpected results
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site. It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server. It appears that the Crawler doesn't think that it's running on the live site. For example http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html will go to a Product Not Found page, and therefore Duplicate Content errors are produced. Running http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html produces the correct product page and not a Product Not Found page Any thoughts?
Moz Pro | | nordichouse0 -
Links counted in Google Webmaster Tools vs. OpenSiteExplorer
Hi all SEOmozzers, site: http://f8phototravel.pl I have (in my GMT profile) 43 domains linking. Among them one of the biggest Polish like: onet.pl (5/187), gazeta.pl (8/393), grono.net.pl, canon-board.info, eturystyka.org and other. They are not present in OpenSiteExplorer. Any ideas why? Thanks, Marek */ Alexa rank (Poland/Global)
Moz Pro | | mad2k0 -
Open Site Explorer only exporting 120 rows!
Has this happened to anyone else? I keep running reports in OSE and then exporting them, but I only get 120 rows exported to the csv when it should be several thousand. Thought I'd ask around before raising it to the Moz team!
Moz Pro | | PeterAlexLeigh0 -
Crawler reporting incorrect URLs, resulting in false errors...
The SEOmoz crawler is showing 236 Duplicate Page Titles. When I go in to see what page titles are duplicated I see that the URLs in question are incorrect and read "/about/about/..." instead of just "/about/" The shown page duplicates are the result of the crawler is ending up on the "Page not found" page. Could it be the result of using relative links on the site? Anything I can do to remedy? Thanks for your help! -Frank
Moz Pro | | Clements1