Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should we use the same word in multiple mega menu link text?
We are currently working on a redesign of our site and I'm working on our mega menus. In the middle of this we had an agency evaluate our current/old site for improvements. (yes the timing is strange) One of the things they recommended is updating our link text in our mega menu. See image: old mega menu. The links are to pages for installation guides in our installation section. They recommended adding Installation to every link. To me this looks spammy. And using the same word in the menu and therefor every page seems spammy. The question is do we need to do this on the new site? In my opinion no. We have the links under a section for installation. Is that enough or should we use installation in every menu item? 6JcZwiU krbzxqx
On-Page Optimization | | BbeS0 -
Optimizing images on website. Is it bad to use the same alt names and keywords?
I have a webpage that I have 6 separate images showing step by step instructions of how to use our product with a sentence describing instruction inside image. I took screenshots of the instructions from the products App and uploaded them to webpage because they provide a great visual. I want to make sure I optimize correctly, can I use the same keywords for all the image names?
On-Page Optimization | | artscube.biz0 -
Can we use internal no-follow links without negatively affecting rankings
we are creating a site structure for a travel website. the site homepage has a top navigation bar with 8 top level links and a total link count of 33 links in this (within menus). There are also 10 footer and ad-hoc links As this top navigation bar is a site-wide navigation, when entering s specific "travel destination" page, the "travel destination" page has its own contextual links and reference links, making the total inks on the destination page approx 107. do you think its ok to make all links in the top navigation bar no follow on all pages except the homepage? how would you approach this to create less links to maintain effective link-juice flow to required pages
On-Page Optimization | | Direct_Ram0 -
Tags vs. Categories? What should I use?
I'm starting with a blog (self-hosted wordpress) and I'm thinking of the following content structure so that the readers are easily able to locate relevant content: Background: It's a blog which gives people relevant info about government jobs. To start with we will just be publishing information about these jobs but over a period of time also intend to post content that helps readers prepare for these jobs. In other words, right now it's just about detailed job notifications but in the coming months, we shall also post about preparation-related information. Typically, each of the job notifications can be bifurcated like: Jobs basis industry Banking Railways Clinical, etc. Jobs basis company ABC co. DEF co. XYZ co. etc. Jobs basis State / City City 1 City 2, etc. Jobs basis educational qualification Graduation Post-Graduation, etc. Now, I'm seriously confused how should I structure this data from the perspective of Categories & Tags such that it's reader as well as SEO-friendly. Do note that each of the government jobs post ideally falls in a couple of above mentioned categories. Thanks..
On-Page Optimization | | Shalin.TJ0 -
Question about overoptimization and images "alt"
Hello, I own a shop with lots of categories, in each category there is a lot of pictures, some have already alt, must I put the attribute alt in all images, it would be 100% beneficial for my site or could would I be over-optimizing the site? Thank you
On-Page Optimization | | yuyuyu0 -
Can rel="canonical" refer to another website page?
I want to republish the post from another website with their permission and want to abide by Google guidelines. Google guidelines is clear when you are using the same content at different parts of the same site however not when using it on another site in a legitimate way. Is there some way to use rel="canonical" refer to another website page of you are reproducing the content from same page?
On-Page Optimization | | h1seo0