Value of having a good crawl budget?
-
Hi,
I've seen several questions where people give advice on how to increase the crawl budget. What I haven't seen anyone comment is what the value of this really is if you have many pages that doesn't get updated very often.
Take for example the typical agency page - 50 pages, most of them rarely gets updates. In a monthly basis normally 10% of the website gets updated.
Is there really any value then of having 100% of the website crawled on a daily basis?
-
Check out this awesome post by AJ Kohn
To answer your question...
"What I’ve observed over the last few years is that pages that haven’t been crawled recently are given less authority in the index. To be more blunt, if a page hasn’t been crawled recently, it won’t rank well."
"The pages that aren’t crawled as often are pages with little to no PageRank. CrawlRank is the difference in this very large pool of pages. You win if you get your low PageRank pages crawled more frequently than the competition."
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
404 crawl errors ending with your domain name??
Hello, I have a crawl test with numerous 404 errors ending with my domain name..? Not sure what the cause is. Plugins? Ecommerce? I use Wordpress if that could lead to an answer. Thanks for your time. K
Technical SEO | | Hydraulicgirl0 -
I'm looking for a good SEO
I need someone that can help me with my SEO. I am too busy to do it and the last person that did it, I really feel didn't do a good job. Please message me (If that is possible). I am looking for on-page, probably disavowing a good bit of links, and anything else that someone can point me in the right direction to do. I'm having some pretty major issues with my guy right now and I just feel like my rankings are falling off of the map because of it. Thanks!
Technical SEO | | Veebs0 -
Maintaining Link Value Of Old URLS With 301 Redirects
Large ecommerce site that has been around for a long time (15+ years.) During that time technology has changed a lot and we are running into issues maintaining 301 redirects for very old urls. For example we have a good amount of links to product and category pages. Some of the old links are to products that still exist and will exist for many years to come.(of note little to no traffic comes via these links. Most of them are close to 9 years old so they are buried deep within articles, forums, or websites) However as we make changes to the site and URL structure these old urls are taking up more resources to continue to maintain 301 redirects. I am Leary of no longer supporting them because I do not want it to impact rankings however there is concern on how much development time and technology resources it takes to continue to support as time goes on. Does anyone have experience handling redirects 3 or 4 url structures old? Looking for insight from someone who has crossed this bridge before.
Technical SEO | | RMATVMC0 -
Can anyone recommend a good template club for joomla
Hi, i am looking for a good template club for joomla. I am looking for a template club that offers magazine and news style templates which are optimised and are not slow. A template i have been using has been reported as running slow and i want to change our template. any help would be great.
Technical SEO | | ClaireH-1848860 -
How do you diagnose if on your site is only 50% crawled?
Good Morning from 7 degrees C, goodbye arctic conditions wetherby UK, If a site had 100 pages for example & that site was plugged into Webmaster Tools how could you diagnose if all the pages had been crawled? The thing is I want to learn how to diagnose crawl issues with sites, is their a known methodology for this? Thanks in advance, David
Technical SEO | | Nightwing0 -
Are article sites a good way to generate traffic and links
I read a lot that article sites are a great way to generate links and traffic but i would like to hear from people on here to see about their experience with article sites. I have tried article sites and i have to be honest that most of them are no follow links and have also not generate any traffic. I have written over 30 articles for free article sites with no traffic coming from them and this has been done over a eight month period can anyone recommend any good article sites that can produce traffic and also ones that produce do follow If you feel i am wasting my time putting articles on free sites then please do tell me
Technical SEO | | ClaireH-1848860 -
Magento - Google Webmaster Crawl Errors
Hi guys, Started my free trial - very impressed - just thought I'd ask a question or two while I can. I've set up the website for http://www.worldofbooks.com (large bookseller in the UK), using Magento. I'm getting a huge amount of not found crawl errors (27,808), I think this is due to URL rewrites, all the errors are in this format (non search friendly): http://www.worldofbooks.com/search_inventory.php?search_text=&category=&tag=Ure&gift_code=&dd_sort_by=price_desc&dd_records_per_page=40&dd_page_number=1 As oppose to this format: http://www.worldofbooks.com/arts-books/history-of-art-design-styles/the-art-book-by-phaidon.html (the re-written URL). This doesn't seem to really be affecting our rankings, we targeted 'cheap books' and 'bargain books' heavily - we're up to 2nd for Cheap Books and 3rd for Bargain Books. So my question is - are these large amount of Crawl errors cause for concern or is it something that will work itself out? And secondly - if it is cause for concern will it be affecting our rankings negatively in any way and what could we do to resolve this issue? Any points in the right direction much appreciated. If you need any more clarification regarding any points I've raised just let me know. Benjamin Edwards
Technical SEO | | Benj250 -
Crawling image folders / crawl allowance
We recently removed /img and /imgp from our robots.txt file thus allowing googlebot to crawl our image folders. Not sure why we had these blocked in the first place, but we opened them up in response to an email from Google Product Search about not being able to crawl images - which can/has hurt our traffic from Google Shopping. My question is: will allowing Google to crawl our image files eat up our 'crawl allowance'? We wouldn't want Google to not crawl/index certain pages, and ding our organic traffic, because more of our allotted crawl bandwidth is getting chewed up crawling image files. Outside of the non-detailed crawl stat graphs from Webmaster Tools, what's the best way to check how frequently/ deeply our site is getting crawled? Thanks all!
Technical SEO | | evoNick0