Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Advice on redirects for a category I want to reuse.
Hi, We have a current category set up that is starting to rank OK but we are going through a site re-build and this category URL will now better describe a new category of products. My dilemma is if I 301 redirect the current url to my new category I won't be able to use the URL for the new one. But if I don't redirect it will the pages that have already been ranked under this url then confuse customers and search engines. For example - Products and sub-categories under the URL /personalised-toys will now become /personalised-toys-for-boys but I want to use the /personalised-toys URL for a different set of sub categories and products. Any assistance or ideas or just definitely don't do it in a particular way would be greatly appreciated
Intermediate & Advanced SEO | | neil_stickergizmo0 -
SEO'ing a sports advice website
Hi Team Moz, Despite being in tech/product development for 10+ years, I'm relatively new to SEO (and completely new to this forum) so was hoping for community advice before I dive in to see how Google likes (or perhaps doesn't) my soon to be built content. I'm building a site (BetSharper, an early-stage work in progress) that will deliver practical, data orientated predictive advice prior to sporting events commencing. The initial user personas I am targeting would need advice on specific games so, as an example, I would build a specific page for the upcoming Stanley Cup Game 1 between the Capitals and the Tampa Bay Lighting. I'm in the midst of keyword research and believe I have found some easier to achieve initial keywords (I'm realistic, building my DA will take time!) that include the team names but don't reference dates or state of the tournament. The question is, hypothetically if I ranked for this page for this sporting event this year, would it make sense to refresh the same page with 2019 matchup content when they meet again next year, or create a new page? I am assuming I would be targeting the same intended keywords but wondering if I get google credit for 2018 engagement post 2019 refresh. Or should I start fresh with a new page and specifically target keywords afresh each time? I read some background info on canonical tabs but wasn't sure if it was relevant in my case. I hope I've managed to articulate myself on what feels like an edge case within the wonderful world of SEO. Any advice the community delivers would be much appreciated...... Kind Regards James.
Intermediate & Advanced SEO | | JB19770 -
Site's pages has GA codes based on Tag Manager but in Screaming Frog, it is not recognized
Using Tag Assistant (Google Chrome add-on), we have found that the site's pages has GA codes. (also see screenshot 1) However, when we used Screaming Frog's filter feature -- Configuration > Custom > Search > Contain/Does Not Contain, (see screenshot 2) SF is displaying several URLs (maybe all) of the site under 'Does Not Contain' which means that in SF's crawl, the site's pages has no GA code. (see screenshot 3) What could be the problem why SF states that there is no GA code in the site's pages when in fact, there are codes based on Tag Assistant/Manager? Please give us steps/ways on how to fix this issue. Thanks! SgTovPf VQNOJMF RCtBibP
Intermediate & Advanced SEO | | jayoliverwright0 -
Can you give me some advices to rank this domain?
Hi Moz community, I've a coleague that's working to rank this site: www.devsar.com. The selected keywords are:
Intermediate & Advanced SEO | | Gaston Riera
Mobile development
Web development
Django Development
Python Development I've checked the site: It's fast and clean. Has a good PA and DA. It's responsive and good lookking. Meta description , title, hreflang.. everything in order. Link profile a little rare (checked with ahref.com), it's because someone made a mistake redirecting some expired domain Can you help me to help my mate out?
Thanks
GR.0 -
1 site on 2 domains (interesting situation, expert advice needed)
Dear all, i have read many posts about having one content on 2 different domains, how to combine those two to avoid duplicate content. However the story of my two domains makes this question really difficult. Domain 1: chillispot.org ( http://www.opensiteexplorer.org/links?site=chillispot.org ) The original site was on this domain, started 9 years ago. That time the owner of the domain was not me. The site was very popular with lots of links to it. Then after 5 years of operation, the site closed. I have managed to save the content to: Domain 2: chillispot.info ( http://www.opensiteexplorer.org/links?site=chillispot.info ) The content i put there was basically the same. Many links were changed to chillispot.info on external sites when they noticed the change. But lots of links are still unchanged and pointing to .ord domain. The .info is doing well in search engines (for example for keyword 'chillispot'). Now i managed to buy the original chillispot.org domain. As you can see the domain authority of the .org domain is still higher than the .info one and it has more valuable links. Question is: what would be the best approach to offer content on both domains without having penalized by google for duplicated content? Which domain should we keep the content on? The original .org one, which is still a better domain but not working for several years or the .info one who has the content for several years now and doing well on search engines? And then, after we decide this, what would be the best approach to send users to the real content? Thanks for the answers!
Intermediate & Advanced SEO | | Fudge0 -
Conversion Rate Optimisation - advice before seeking out a specialist
Hello! I have a site I've been working on for a lovely client, a small business start up since Christmas. The site has a very simple layout, is ranking well and maintaining its positions, has solid social media, is receiving enough traffic and ranking for a number of terms. The problem is - conversions! The site just isn't converting. I have spoken with a few peers who have said advanced CRO will be too much for me to learn in terms of Psychology of Buying, learning about colors, fonts etc. I understand meta descriptions for example are something that I can do, I was wondering if anyone could give me advice on any other basic CRO techniques I could apply to the site before going to a specialist. Any advice would be MUCH appreciated - the moz community is always so helpful! Charlotte 🙂
Intermediate & Advanced SEO | | CharlotteWaller1 -
Seo advice / plan ? penalized ?
I built an ecommerce site for a client of mine just over 9 months ago now. To begin with the serps were great, everything was listed in the results but for some reason a few weeks in all the results vanished from google and now we're lucky to find anything. I've been as far as page 200 and havent found any results. Its been like this for a solid 8 months so i can only presume that the site has been penalised in some form. Searching for unique phrases from the site doesnt even return results. The website in question is = http://goo.gl/A6Gz2
Intermediate & Advanced SEO | | gfxpixeldesigns
keywords we're aiming for = coloured contact lenses, fashion contact lenses
Target = Google UK Now im not really an seo guy but regardless of this my client has hired me to see whats going on and correct it. I've been scratching my head thinking all sorts of things but none of which im certain about so i'm looking for someone to point me in the right direction before i do anything drastic. So to begin with here are some of my suspicions which i personally think are affecting ranking and possible penalisation. #1 - Too many links on the page
#2 - Possibly over optimised
#3 - Lack of content on the product and category pages
#4 - Lack of backlinks and links in general coming from other sites My main concern is the lack of links from other sites and the odd link coming from low quality sites. I've also just found out that my client has been using an automatic link submitter which i've always thought of as a big no no. Some of the sites these links have been submitted to have nothing to do with the keyword we are targetting and are sort of spammy sites containing all sorts of links. Im wondering if these poor quality links could have caused the site to be penalised, google may be seeing it as a spammy site due to this. Whats your opinions on the above, are my suspicions correct and can this be recovered ? My planned course of action is to be as follows: #1 - Re write the content currently on the site so that it is better written and include more keywords, especially long tails since i think these will help bring the serps up.
#2 - Write detailed category and product descriptions as well as making sure every page has some well written content with links and keywords.
#3 - Keep the above pages to one main subject / keyword so that google doesnt get confused.
#4 - Get some links on popular and relevant sites, the only problem here is the lack of fashion contact lens sites. Does anyone have any advice on how to find these or where i should be getting links placed. Are directories worth while ?
#5 - Get more involved in the social side ie facebook, twitter I will be building on the above over time, aswell as running google ads moderately for our chosen keywords. Is there anything i have missed, anything i shouldnt be doing. Please advise. Thanks.0 -
We've just bought a new domain - need advice on the exact procedure to follow...
Hi guys, We've just bought a 3 letter .co.uk domain to replace our current 20 character old domain. Our existing domain is PR5 with quite a few links (that we can modify no problem) We're currently .301 redirecting the new domain to the old domain. I was looking at the procedure in one of the guides but as it's slightly different - is this the correct procedure? 1. prep the duplicate site on new domain and prep the individual htaccess .301 redirects 2. Add new the domain to google webmaster tools bing Webmaster centre 2. On the switchover date - modify all possible incoming links from external sites 3. On the switchover date - apply the .301 redirects and make the new site live 4. On the switchover date - apply the new sitemaps to google & bing 5. on the switchover date - fill out the change of address form in webmaster tools 6. Do the happy dance? many thanks in advance, Tony.
Intermediate & Advanced SEO | | posh_tiger0