Bot Crawling issues
-
Dear all, Is this cache:www.subhavaastu.com is not working now with Google. Why it is not showing when the site is crawled. What were the new algorithms which Google is adopted? in my searches of all of my site internal links of www.SubhaVaastu.com, I observed only 404 instead of Google visiting time and date. After observing this 404 of each of my site links, I understand Google stopped crawling my site.
Some more examples are shown below with other websites:
cache:www.vastuwebsite.com ("NOT" showing when Google visited this site)
cache:www.vastuconsultantusa.com ("NOT" showing when Google visited this site)
cache:www.shubhavaastu.com ("NOT" showing when Google visited this site)
cache:www.subhavastu.com (Showing when Google visited this site)
To my surprise, I noticed that Google crawled latest links in my site, which I added a new link (https://www.subhavaastu.com/remove-negativity.html) just 10 days back, this new link was clearly crawled by Google. I typed "remove negativity subhavaastu", I saw the results with this new page in the SERP, but on the same way when I typed "cache:www.subhavaastu.com/remove-negativity.html", it is showing again 404.
what is happening with Google, is Google is following any new algorithms now. Is google changed any new concept? or, is my site is penalized in any case, I think it may not be, because if my site is penalized, then Google should not visit the new links and should not show the results with my site. Coming with my site, its pure from viruses, no malicious codes. Indeed it's an article based site, which has a good reputation. This domain is taken in the year 2003. We never spam anywhere we never did any wrong methods.
If my site is penalized, then is it manual penalized or bot penalized. I thoroughly checked webmaster tools and google console, I never found any notices or any note from Google.
Require experts analyzation on this doubt. Thanks in Advance.
-
Dear SeoMAN,
Thanks for your answer. I just tried at google, its shows normal, I just checked google search console and webmaster tools, there is nothing serious shows. Can it be possible to try to find out what was the main cause.
Is I taken the right image, is I searched exactly what you are asking about?
Many thanks.
-
Dear Vijay, Thank You.
You are an expert in this field, I respect your thought, but I am thinking there should be some other reason, that might cause of this problem. I am MOZ PRO subscriber, if there is, canonical issues, I will get such notifications. Long back, there are some canonical issues, we fix them.
Site is totally free from malware and viruses, I just now also observed Google webmaster tools and checked other ways, there is nothing viruses found in my site.
Can it be possible to find any other issues which caused this problem.
Many thanks.
-
You are asking quite a lot of questions here, basically if your site has been around for a reasonable amount of time and your page is accessible and can be crawled by search engines you should be able to Google your domain name (including the TLD) like "mydomain.com", if that doesn't show up you either have a serious technical error or you have had a penalty.
As for telling whether it is bot or manual Google search console should give you a bit more information. Personally as far as I can see a penalty is a penalty and needs to be overcome one way or another, whether it is automatic or manual is a bit beside the point!
-
It can be due to a lot of SEO/technical issues, Here are a couple of observations/suggestions:
- Your site works with https:// and with http:// both and also with www. and without www. , this might be an issue for duplicate content. Please make sure that you have only one version with or without www. or https://
- Check your website for malware and viruses just as a precaution.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SERP Cannibalization Issue
Hi Moz Community, Glad to be here with you. I need some help from you guys. I have a new client website: [https://deluxemassage.co.uk/](link url) My client runs this type of business offering professional therapeutic health services. It offers massage services on demand ("mobile", therapists visiting clients at home). The client has his business registered, and the therapists listed on the website are carefully vetted, all having training in different types of massage. The client's problem is that until 2 months ago the site was in position 3-4 in Google (somewhat stable) with the main page for the keyword "mobile massage London". Recently I noticed that another page on the site that lists all therapists - i.e. this page [https://deluxemassage.co.uk/therapists/](link url)- is on the first page of Google for the above keyword, although both the title tag, meta description and the page on the site do not target the above keyword.
Algorithm Updates | | Koyel9754
I notice that both pages have serious fluctuations in the SERP for the mentioned keyword, sometimes appearing both in the results.
The Home page is marked as rel=canonical Of course the position in SERP fluctuates from 8, 11, 12, 7. What should I do to solve this problem? Any thoughts here would be appreciated. Thanks0 -
Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem. To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board. Context that might be helpful: I did check technical settings, and the sites are crawlable. The sites do appear in search but seem to be losing organic search traffic. Thanks for any help you can provide!
Algorithm Updates | | geodigitalmarketing0 -
I'm Pulling Hairs! - Duplicate Content Issue on 3 Sites
Hi, I'm an SEO intern trying to solve a duplicate content issue on three wine retailer sites. I have read up on the Moz Blog Posts and other helpful articles that were flooded with information on how to fix duplicate content. However, I have tried using canonical tags for duplicates and redirects for expiring pages on these sites and it hasn't fixed the duplicate content problem. My Moz report indicated that we have 1000s of duplicates content pages. I understand that it's a common problem among other e-commerce sites and the way we create landing pages and apply dynamic search results pages kind of conflicts with our SEO progress. Sometimes we'll create landing pages with the same URLs as an older landing page that expired. Unfortunately, I can't go around this problem since this is how customer marketing and recruitment manage their offers and landing pages. Would it be best to nofollow these expired pages or redirect them? Also I tried to use self-referencing canonical tags and canonical tags that point to the higher authority on search results pages and even though it worked for some pages on the site, it didn't work for a lot of the other search result pages. Is there something that we can do to these search result pages that will let google understand that these search results pages on our site are original pages? There are a lot of factors that I can't change and I'm kind of concerned that the three sites won't rank as well and also drive traffic that won't convert on the site. I understand that Google won't penalize your sites with duplicate content unless it's spammy. So If I can't fix these errors -- since the company I work conducts business where we won't ever run out of duplicate content -- Is it worth going on to other priorities in SEO like Keyword research, On/Off page optimization? Or should we really concentrate on fixing these technical issues before doing anything else? I'm curious to know what you think. Thanks!
Algorithm Updates | | drewstorys0 -
Could we run into issues with duplicate content penalties if we were to borrow product descriptions?
Hello, I work for an online retailer that has the opportunity to add a lot of SKUs to our site in a relatively short amount of time by borrowing content from another site (with their permission). There are a lot of positives for us to do this, but one big question we have is what the borrowed content will do to our search rankings (we normally write our own original content in house for a couple thousand SKUs). Organic search traffic brings in a significant chunk of our business and we definitely don't want to do something that would jeopardize our rankings. Could we run into issues with duplicate content penalties if we were to use the borrowed product descriptions? Is there a rule of thumb for what proportion of the site should be original content vs. duplicate content without running into issues with our search rankings? Thank you for your help!
Algorithm Updates | | airnwater0 -
Webmaster Tools Crawl Stats Drop - Ranks Dropped too
Hello, So I have a situation I haven't seen that happened today. I checked in on my keyword rankings, and all seemed to drop off the rankings in one day. Even googling my domain name, I'm not the first result (when I always have been with multiple sublinks under). There were no unusual errors in the webmaster tools, no issues "Fetching as Google". The only thing I can find is in Crawl Stats, my "Kilobytes downloaded per day" and "Time spent downloading a page (in milliseconds)" dropped a lot in the last day or so (by about 90%). Has anyone had this happen? Any ideas for how to bounce back from this? Domain referenced is mediamegamall.com Thanks!
Algorithm Updates | | Ask_MMM0 -
Should we crawl user profiles?
Hi there, We have a service whereby we measure people's personalities and reputation. We would like to be able to search: "Reputation of XXX" and you would find the particular reputation profile of that person. However the differences for each page are limited. Effectively the name, and a few numbers that measure such reputation. Would crawling these profiles affect us due to "thin content"? Would you consider any alternative strategy? Thanks!
Algorithm Updates | | josei0 -
SERPS Issue. Why is this happening?
All of our pages have been indexed by Google and we are ranking very well thus far. Here is an example of a search where, based on my searches, we rank well. Search for "Indiana ERISA Bond". You can search for any state name followed by "ERISA Bond" and we rank really well (Top of page 1). The pages that should show up for you are here: 1. https://www.suretegrity.com/surety-bonds/indiana/erisa-bond 2. https://www.suretegrity.com/surety-bonds/indiana/erisa My issue is with a different product we offer. Search for "Georgia Mortgage Lender Bond". The pages I would hope to come up, in order of priority, are: 1. https://www.suretegrity.com/surety-bonds/georgia/mortgage-lender 2. https://www.suretegrity.com/surety-bonds/georgia/mortgage-broker-banker-lender When I do my searches I show up at the bottom of page 2 and the URL that is being found is #2 above and not #1. I have spent a LOT of time thinking about this and am in need of some direction. What am I doing wrong? Thanks, Alex
Algorithm Updates | | dbuckles0 -
Search bots that use referrers?
Can someone point me to a list or just tell me specific search bots that use referrers?
Algorithm Updates | | BostonWright0