Screaming Frog for competitor research

[lgrozeva]

Hi guys,

Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website?  It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?

Thank you.

Cheers

[lgrozeva]

Thank you, guys!

@ Dan, I will get in touch! Thanks for your quick response

Cheers,

Ani

[screamingfrog]

Thanks for helping out with the above query guys, all good answers.

Hey @Margarita,

If you'd like to drop an email through to [email protected], I can tell you what the exact issue is

Cheers,

Dan

[Kara.Wallace]

Hi Margarita,

I have encountered similar issues when you need to enter a password or where the site might require cookies.

Here is what Screaming Frog has to say on the topic:

"13) Why Won’t The SEO Spider Start? Top ↑

This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.

14) Why Won’t The SEO Spider Crawl My Website? Top ↑

This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."

Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.

Hope this helps.

Mike

[mihaiaperghis]

Hey Margarita,

Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.

Hope this helps!