Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When, if ever, would you need to use: example.com/en
Is there any reason to have /en on your website if your website is only in English? Or is it worth having in case you are preparing to translate into other languages? And is there any advantage to being: en.example.com over example.com/en Thanks
On-Page Optimization | | CosiCrawley0 -
Using Escaped Fragments with SEO
Our e-commerce platform is in the process of changing to what we call app based stores (essentially running in a browser as single page web-app) With these new stores they are being built in HTML 5 and using escaped fragments.
On-Page Optimization | | marketing_zoovy.com
Currently merchants are usually running 2 stores until we launch to app site at 100%. My questions are really concerning the app stores which right now show on a subdomain but will essentially take over the primary domain. Here is an example:
app.tikimater.com and app.sportsworld.com Since I am not a developer, I'm really having a hard time understanding the escaped fragments. I'm using this but https://developers.google.com/webmasters/ajax-crawling/docs/getting-started I'm not sure what my actual urls should look like and what the canonical should be set to. Right now they have been removed but previously they had http:app.tikimaster.com#!v=1 Also, and how I should be setting up my meta information for Google so 1) pages are indexed timely 2) pages are indexed with the correct information. I am still setting the meta titles and descriptions but in some instances Google uses other info. With the new platform we are moving away from on page content (written paragraphs) but category pages would have related products embedded. Should I still be pushing to have some type of intro text, since it would solely be for SEO and not the shoppers experience. All product pages have content (product description etc) Thank you for any advice0 -
Need I add rel="dofollow" or not?
Hello, My website is http://www.vietnamvisacorp.com is using the href links without meta tag rel="dofollow" such as I am using . Should I put ref="dofollow" in this: Thank you!
On-Page Optimization | | JohnHuynh0 -
"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
On-Page Optimization | | Deb_VHB0 -
Using fathead page keywords for directories and as a red herring to competitors
I'm fairly new to SEO and I have been reading a lot on here and the SEOmoz guides over the last few days, finding it very interesting. I am wondering about page keywords, I read that the engines no longer use them. In this thread they say they still use them because of directories. http://www.seomoz.org/ugc/why-i-still-use-meta-keywords#new-comment So I was wondering, because directories are always smaller scale than search engines would keywords that are normally high competition have more clout in them? If so then using them could be misleading to competition if they think they are the actual keywords? or might this contradict between your actual chosen keywords in the directories backlink anchor text or something?
On-Page Optimization | | Zoolander0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Follow up on "Canonical Tag Placement - Every Page?"
But if it is like Pete said, I don't understand why e.g. SEO Moz has a Canonical Tag on this Page http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps Which leads to the exact same page!? What is the benefit of doing so? Regards
On-Page Optimization | | Here4You0 -
Does it matter if a rel = "canonical" element is added to the beginning or the end of a URL?
I am curious to know if adding a rel = "canonical" tag to the end of a link element will affect its purpose?
On-Page Optimization | | Sharecare0