Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
-
Hi Everyone!
My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
-
Hi Farhan
Yes sure, we've definitely found this helps bring the crawlers back to the site more regular!
Hope it helps
Dan
-
Hi Dan! Thanks a lot for your views! This might be it! Content on the website is not updated very regularly owing to the nature of the site. Perhaps updating the blog more regularly or making tweaks in the meta script could help as per your suggestions?
-
Hi Anders! They are all available to the Googlebot! I’m afraid, it won’t be possible for me to share the URL on a public forum! Please check your inbox! Thank you so much for your insights and concern!
-
Hi Umar! There’s been no change in web hosting nor a penalty! We regularly monitor the Search Console and the speed is good too! Thanks a lot for your time though!
-
Hi
How regular are you updating content on the site?
We've found that regular updating and tweaks to content on pages to improve the pages content & user's experience helps the page indexing & frequency.
Looking at analytics and seeing what customers are searching for on your site can help you then improve the content on pages, by going back amending and improving page content we've found Google returns more often to re-index.
Dan
-
Hi!
Are all pages available for GoogleBot, or are they perhaps blocked by robots.txt or a meta tag?
Do you have a URL for the website so we can take a look?
Anders
-
Hello Farhan,
Yes, it's not normal. Did you change your web hosting in this period? or have you faced any kind of penalty in recent past? Usually this happens when you just recovered.
You also need to look out your Search Console for crawling problems, are there any crawling or server issues you're facing? Sometimes, your .htaccess creates crawling problems, make sure to check again. Also, see how fasts your pages are loading, if the speed is low it also cause the crawling delayed.
Hope this helps!
Umar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website indexing issues
My website is being indexed with both https - https with www. and no leader at all. example. https//www.example.com and https//example.com and example.com 3 different versions are being indexed. How would I begin resolving this? Hosting?
Technical SEO | | DigitalRipples0 -
Can you force Google to use meta description?
Is it possible to force Google to use only the Meta description put in place for a page and not gather additional text from the page?
Technical SEO | | A_Q0 -
Can you have 2 different websites on 1 webmaster tools account
Someone set up both our sites on the one webmaster tools account is this the best way to do it or should we have 2 different accounts. We are having problems with our site verification not working and our google shopping feeds not working could this be the cause.
Technical SEO | | CostumeD0 -
Website not crawled
i added website www.nsale.in in add campaign, it shows only 1 page crawled. but its working fine for other sites, any idea why it failed ?
Technical SEO | | Dhinesh0 -
Can SEOMoz crawl a single page as oppose to an entire subfolder?
I would like the following page to be crawled: http://www.ob.org/_programs/water/water_index.asp Instead, SEOMoz changes the page to the following subfolder which is an invalid url: http://www.ob.org/_programs/water/
Technical SEO | | OBIAnalytics0 -
Can anyone recommend an SEO friendly Joomla extension to use for SEO on an existing website?
Can anyone recommend an SEO friendly Joomla extension to use for SEO on an existing website? I have downloaded sh404sef but I don't want to change my URL's and make them longer than they are at the moment. Any ideas?
Technical SEO | | Karen_Dauncey0 -
How can I tell Google, that a page has not changed?
Hello, we have a website with many thousands of pages. Some of them change frequently, some never. Our problem is, that googlebot is generating way too much traffic. Half of our page views are generated by googlebot. We would like to tell googlebot, to stop crawling pages that never change. This one for instance: http://www.prinz.de/party/partybilder/bilder-party-pics,412598,9545978-1,VnPartypics.html As you can see, there is almost no content on the page and the picture will never change.So I am wondering, if it makes sense to tell google that there is no need to come back. The following header fields might be relevant. Currently our webserver answers with the following headers: Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0, public
Technical SEO | | bimp
Pragma: no-cache
Expires: Thu, 19 Nov 1981 08:52:00 GMT Does Google honor these fields? Should we remove no-cache, must-revalidate, pragma: no-cache and set expires e.g. to 30 days in the future? I also read, that a webpage that has not changed, should answer with 304 instead of 200. Does it make sense to implement that? Unfortunatly that would be quite hard for us. Maybe Google would also spend more time then on pages that actually changed, instead of wasting it on unchanged pages. Do you have any other suggestions, how we can reduce the traffic of google bot on unrelevant pages? Thanks for your help Cord0 -
Can anyone tell me where my site can be improved
My website is www.theradiatorgallery.com I have been trying to get on 1st page for designer radiators. I am currently ranked 21st just dropped to the 3rd page within the past hour, i was 20th before this. There are some sites that i would think i would rank better than due do i have better on page optimization, higher PA, DA and PR based on seo moz reports. Is there something wrong with my site that i am missing? the sites i feel i should rank better than are; radsnrails.co.uk, warmrooms.co.uk and a couple others Any help would be great! Thanks Guys
Technical SEO | | myloseo0