Bot Crawling issues
-
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site.
Some more examples are shown below with other websites:
cache:www.vastuwebsite.com ("NOT" showing when Google visited this site)
cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site)
cache:www.shubhavaastu.com ("NOT" showing when Google visited this site)
cache:www.subhavastu.com (Showing when Google visited this site)
To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404.
what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods.
If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google.
Require experts analyzation on this doubt. Thanks in Advance.
-
Dear SeoMAN,
Thanks for your answer. I just tried at google, its shows normal, I just checked google search console and webmaster tools, there is nothing serious shows. Can it be possible to try to find out what was the main cause.
Is I taken the right image, is I searched exactly what you are asking about?
Many thanks.
-
Dear Vijay, Thank You.
You are an expert in this field, I respect your thought, but I am thinking there should be some other reason, that might cause of this problem. I am MOZ PRO subscriber, if there is, canonical issues, I will get such notifications. Long back, there are some canonical issues, we fix them.
Site is totally free from malware and viruses, I just now also observed Google webmaster tools and checked other ways, there is nothing viruses found in my site.
Can it be possible to find any other issues which caused this problem.
Many thanks.
-
You are asking quite a lot of questions here, basically if your site has been around for a reasonable amount of time and your page is accessible and can be crawled by search engines you should be able to Google your domain name (including the TLD) like "mydomain.com", if that doesn't show up you either have a serious technical error or you have had a penalty.
As for telling whether it is bot or manual Google search console should give you a bit more information. Personally as far as I can see a penalty is a penalty and needs to be overcome one way or another, whether it is automatic or manual is a bit beside the point!
-
It can be due to a lot of SEO/technical issues, Here are a couple of observations/suggestions:
- Your site works with https:// and with http:// both and also with www. and without www. , this might be an issue for duplicate content. Please make sure that you have only one version with or without www. or https://
- Check your website for malware and viruses just as a precaution.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long does google takes to crawl a single site ?
lately i have been thinking , when a crawler visits an already visited site or indexed site, whats the duration of its scanning?
Algorithm Updates | | Sam09schulz0 -
Website have Caching/Indexing / Ranking Issue
Hi, My Website (https://www.v3cars.com) is not cached or indexed on regular basic from last 15 days. before this it was cached or indexed on regular basic. We are uploading fresh content on daily basic. Currently my new content is not ranked anywhere in Google even after cached or indexed. Please help and suggest. Sandeep - Love to Cars
Algorithm Updates | | onlinesandeep0 -
Is googlebot the slowest bot?
This morning, I wrote a breaking news story about a "Wolf of Wall Street" It was published at 12:05:49 Googlebot, which used to be on my site within a minute or less, didn't bother to visit for 53 minutes. And now, 32 minutes later, even though it has been crawled, this story doesn't even show up in google search. Except that it is in the top 10 stories today, at #2, so the headline appears in every page on the site, so every page that has been crawled today, around 10 minutes after it was published, contains that text, so they show up. EINnews, which also crawls our pages is listed for the headline text. Finally, the page turns up in search results 4 hours later, and says that it is 4 hours old. Does anyone else see this slow motion mode? If you do see this, what is wrong with the site that causes this recalcitrant behavior? The headline of the story is "A 'Wolf of Wall Street' Raided By FBI In Florida" and the link is http://shar.es/1bW5Sw
Algorithm Updates | | loopyal0 -
Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level? BT
Algorithm Updates | | achituv0 -
Images added to website automatically become URLs - is this an issue?
Hello Mozzers! I've just been trawling through a website and noticed all of the images had their own URLs. There's a bespoke CMS and that's how it works with images... So out of 1447 urls, 1314 are images. Firstly, is this an issue / problem from an SEO perspective. If it is, how should I deal with it? Thanks in advance, Luke
Algorithm Updates | | McTaggart0 -
A Serious drop in Pages crawled per day
On 21st April ,I spotted a sudden decrease in pages crawled per day.Previously it was about 5,000 bust after the drop it reached to 225.From the crawl rate never spiked. Here is my website url - http://www.wpstuffs.com/ 8fQHW2G.png
Algorithm Updates | | vividvilla0 -
Google Webmaster Tools: Quality Issues on http://www.enakliyat.com.tr/
Specifically, we detected low-quality pages on your site which do not provide substantially unique content or added value. Examples could include thin affiliate pages, doorway pages, automatically generated content, or copied content. We encourage you to make changes to your site so that it meets our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results. Google Webmaster Tool send me this message I think the low-quality pages is like the this http://www.enakliyat.com.tr/detaylar/bursa-fethiye-ucevler-nakliye-5834 page and we have so many pages like this... Example 1: http://www.enakliyat.com.tr/detaylar/evden-eve-nakliyat-5906 **Example 2 : **http://www.enakliyat.com.tr/detaylar/cekmekoy-izmit-5905 **Example 3: **http://www.enakliyat.com.tr/detaylar/evden-eve-nakliyat-5906 What should I do to these pages HELP 😞
Algorithm Updates | | iskq0 -
Large site with faceted navigation using rel=canonical, but Google still has issues
First off, I just wanted to mention I did post this on one other forum so I hope that is not completely against the rules here or anything. Just trying to get an idea from some of the pros at both sources. Hope this is received well. Now for the question..... "Googlebot found an extremely high number of URLs on your site:" Gotta love these messages in GWT. Anyway, I wanted to get some other opinions here so if anyone has experienced something similar or has any recommendations I would love to hear them. First off, the site is very large and utilizes faceted navigation to help visitors sift through results. I have implemented rel=canonical for many months now to have each page url that is created based on the faceted nav filters, push back to the main category page. However, I still get these damn messages from Google every month or so saying that they found too many pages on the site. My main concern obviously is wasting crawler time on all these pages that I am trying to do what they ask in these instances and tell them to ignore and find the content on page x. So at this point I am thinking about possibly using robots.txt file to handle these, but wanted to see what others around here thought before I dive into this arduous task. Plus I am a little ticked off that Google is not following a standard they helped bring to the table. Thanks for those who take the time to respond in advance.
Algorithm Updates | | PeteGregory0