Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drop in Android organic search
Hay Moz Community, I have been analysing traffic fluctuations for a few websites and have found a noticeable drop in organic traffic from android devices. I initially thought it could be a generic mobile drop but iOS is holding its own? Can anyone share any insights they may have come across to explain the drop on android specifically. Thank you in advance
Search Behavior | | JBGlobalSEO
Alex0 -
Google web master tool - stats
Hello everyone! I have 2 questions : 1. does anyone come across this behavior ? (image 1). In the last week Page crawled per day behavior was quite wild and it wasn't affect directly on Time spent downloading a page. 2. I'm getting site map warning for high response time (image 2) i have 29M pages so I aware that 4 warning for 4 pages is almost negligible but still ? Thanks to all helpers 8OkgXp7 q3txRAb q3txRAb
Search Behavior | | Roi_Bar1 -
Why do all traffic curves show a "saw tooth" pattern in Google Analytics
Greetings Mozzers This sounds like a dumb question, but it's bothering me just the same and I would like to think what the fine community of SEO experts at MOZ thinks I have taken a look at the traffic curve for about 20 different sites in very different industries (home improvement, wedding party supplies, pet feeding systems) and it seems to me that they all show a saw tooth pattern, more or less like the one below.
Search Behavior | | Masoko-T0 -
Looking for something better than Googles in page analytics
Buongiorno from 12 degrees C wetherby UK 🙂 I'm interested in measuring content engagagement, things like what gets clicked on mouse tracking beahaviour etc. Google Analytics in page analytics does a good job but is there another product able to guage user interactive behaviour with content (Yes ive added event tracking). Thanks in advance,
Search Behavior | | Nightwing
David1 -
Google reconsideration nightmare
Hello and thanks in advance The website has had a penalty on it for a while now, around 10 months, it was worked on by an agency who bought bad links to it but before then it was worked on by other agencies that may have done the same. I cleaned up as many bad link (according to many posts read) and filled for reconsideration and was told to get rid of a whole bunch of links which i did not know existed. Downloaded WMT links as instructed by Google admin person and contacted a heap of people which took a lot of man hours and cost us a fortune. Resubmitted and again was shown a handful of links by the Google admin person and told to contact and remove. The funny thing is that a few of them I disavowed in my list so they should not have pointed these out. I emailed back and showed that everything I could do was done and am happy to disavow any other link which they though violated their terms. This was not enough and I was told to show more efforts in removing links and then resubmit for reconsideration. I have done as much as I can on the website, I cannot see any more links which show violation, if there are some I am happy to remove but am now at a stage where i need direction from others to tackle this matter. Any advice would be helpful; I cannot start over from scratch as it's a brand and not a small website.
Search Behavior | | Benbug0 -
Local vs Global Search Results Yield Very Different Rankings Lately
When I monitor my website's rankings, I always do it from Canada (direct connection) and from the USA (using a VPN in Arizona). I've been monitoring rankings this way for the last 3 years. Most of the time, I got similar results (-5 / +5) from both location. My website is a ".com" and targets an international audience. Lately (is it since Panda 22?), I've seen dramatic differences in rankings from both locations. Some keywords will rank in the top 10 on Google.com (from the USA) while they will appear on page 3, 4, 5 and even lower on Google.ca (from Canada). The thing is, the top 10 results on Google.ca are not even from canadian websites. Fact of the matter, there are even some results from India websites (.in) in the top 10! I understand that Google.ca will give advantage to websites from Canada (or targeting the canadian market / .ca domain name) over international / US websites but there's never been such a huge difference in rankings until lately. Has anybody else experienced this? What are your thoughts?
Search Behavior | | sbrault740 -
From #2 TO Not Found in top 100 Results, WTF?
So my website was #2 for a targeted keyword on google. (6 months ago: # 😎 (3 months ago: #6) (1 week ago: #2). As you can see, it was moving up nicely, now it's not found on the top 100 search results. Only appears #2/(B) in the local search section. Could that be the reason? Websites that appear on the local search no longer appears in the SERP?
Search Behavior | | Droidman860 -
Can I use Google Analytics to find out actual times of visits during the day??
Hi, I'm a newbie at all this - I hope someone can help me. We're thinking of running time-specific offers to try and convert as many of our customer site visits as possible e.g. 15% discount if you call between, say, 2 and 5pm. It would be really helpful to me to find out what times of day people are visiting our site. I can't seem to find a way to do this on Google Analytics. Can anyone help? Thanks so much Sue
Search Behavior | | 3Amigos0