Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to deal with rel=canonical when using POST parameters
Hi there,
On-Page Optimization | | mjk26
I currently have a number of URLs throughout my site of the form: https://www.concerthotels.com/venue-hotels/o2-academy-islington-hotels/256133#checkin_4-21-2024&checkout_4-22-2024&rooms_1&guests_2&artistid_15878:256133 This sends the user through to a page showing hotels near the O2 Academy Islington. Once the page loads, my code looks at the parameters specified in the # part of the URL, and uses them to fill in a form, before submitting the form as a POST. This basically reloads the page, but checks the availability of the hotels first, and therefore returns slightly different content to the "canonical" version of this page (which simply lists the hotels before any availability checks done). Until now, I've marked the page that has had availability checks as noindex,follow. But because the form was submitted with POST parameters, the URL looks exactly like the canonical one. So the two URLs are identical, but due to POST parameters, the content is slightly different. Does that make sense? My question is, should both versions of this page be marked as index,follow? Thanks
Mike0 -
When, if ever, would you need to use: example.com/en
Is there any reason to have /en on your website if your website is only in English? Or is it worth having in case you are preparing to translate into other languages? And is there any advantage to being: en.example.com over example.com/en Thanks
On-Page Optimization | | CosiCrawley0 -
How "Top" or "Best" are considered when in front of keyword
I would like to know if someone has proven info how google today counts words "Top" or "Best" when in front of main keywords you try to rank for. For example, if I have a keyword like "Restaurants in Madrid" and I optimize that page without using words "top" or "best" will it have good rankings for keywords "top restaurants in madrid" and "best restaurants in madrid" ? I suppose that google is smart enough to know that web page should be good ranked even without using those 2 words but would like to know percentage of my loss if I just exclude those words from title tag and other important onpage factors. I want to rank high for all the 3 combinations, with "top", with "best" and without it in front so searching for best solution. I plan just to add one of those words, for example "top" and hope that google will know that "top" = "best" 🙂
On-Page Optimization | | m2webs0 -
Using Subdomains to Avoid Sitewide Penalties?
If I'm using a subdomain on my website, for instance news.mywebsite.com, and it gets penalized by Panda (or whatever animal update), would that affect the main domain and/or other subdomains?
On-Page Optimization | | sbrault740 -
Sitemap error is reported when using a sitemap-index generated by Yoast
I've installed the Yoast SEO Plugin for wordpress and I've setup the sitemaps using it. I saw the tool has generated the Sitemap index file http://www.phraseexpander.com/sitemap_index.xml with different indexes for posts and pages I've submitted that to google and it's indexed. When I use Seoquake to check my website, I see that it says that the sitemap is missing (in fact http://www.phraseexpander.com/sitemap.xml) is returning 404. Shall I fix that? Shall I do a 301 redirect in my .htaccess file to http://www.phraseexpander.com/sitemap_index.xml Thanks.
On-Page Optimization | | nagar0 -
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them.
My report indicated that I have 340 crawl warnings. Not sure how to fix them. Please provide links on where I need to go to fix them. http://pro.seomoz.org/campaigns/95663/issues#notice-issues
On-Page Optimization | | cyaindc0 -
Duplicate Content using templates
Hi, Our web site is designed using a template, which means the header and footer is consistent across all pages. Only the body content is unique on each page. Is the google bot able to see that the header and footer content is defined by the common template? Will this have any impact in terms of duplicate content? For example, we have a two line text in the footer that summarize the services we provide. Because the same text is in the footer of all pages, i am concerned about creating duplicate content. Finally, does it make sense to include keywords in header and footer of the template? Will it have any positive or negative SEO impact?
On-Page Optimization | | petersen0 -
Has anyone noticed a big delta between Google and Bing rankings? For example, we rank favorably in Google, but not so favorably in Bing. Are there different tactics I should use to rank better in Bing?
An example is in Google, we currently rank #1 & #2 for "yoga pants" for Athleta and Old Navy. In Bing, I'm on page 2. Any thoughts here?
On-Page Optimization | | kpr0