Is real google bot like "fetch"or more like "fetch & render"?
-
In GWT we have two options to mimic googlebot visits, "fetch" and "fetch and render", but when the real googlebot visit a page, is he behaving like the former or the latter?
I can see fetch does fetch only the html, while fetch and render does fetch .js and .css as well.
But what does the real googlebot does?
I have checked the web server logs, and I can see the real googlebot sometimes request the .js files too, but not every time it visit a page, sometimes it does, sometimes it does not.
Has anyone figured out when googlebot actually request javascript files?
-
That was my guess, that google somehow cache the .js files, so I changed one javascript file name to a random generated name. Did it last night, I will recheck logs in a couple of days to see what happen.
-
Googlebot is behaving like the latter, "Googlebot constantly crawls and renders URLs across the entire web..." from https://support.google.com/webmasters/answer/6066467, but has the option of streamlining the process by not rendering every time. For example, if many sites are referencing the same centrally hosted code base of js, googlebot doesn't need to constantly recheck that js every time it comes across the link.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How To Stop Google's "Fetch & Render" From Showing Up In Google Analytics
Hi all, Within Google's "Fetch & Render" (found in Google Search Console) is the ability to index certain pages from my website on-demand. Unfortunately, every time I ask Google to index a page, it registers as a bounce in Google Analytics. Also, if it means anything, my website (www.knowtro.com) is a single-page application, functioning similarly to Google. If you guys know of any solution to this problem, please help! I originally thought that Google would know to block its own Fetch & Render crawler from Google Analytics but that doesn't seem to be the case. Thanks, Austin
Reporting & Analytics | | A_Krauss0 -
Google Analytics Set-Up for site with both http & https pages
We have a client that migrated to https last September. The site uses canonicals pointing to the https version. The client IT team is reluctant to put 301 redirects from the non-secure to the secure and we are not sure why they object. We ran a screaming frog report and it is showing both URLs for the same page (http and https). The non-secure version has a canonical pointing to the secure version. For every secure page there is a non-secure version in ScreamingFrog so Google must be ignoring the canonical and still indexing the page however, when we run a site: we see that most URLs are the secure version. At that time we did not change the Google Analytics setup option to use: "https" instead of "http" BUT GA appears to be recording data correctly. Yesterday we set up a new profile and selected "https" but our question is: Does the GAnalytics http/https version make a difference if so, what difference is it?
Reporting & Analytics | | RosemaryB1 -
Google Analytics and DNS change
Our new alumni application is going be tested at domain uva.imodules.com . We are going to collect traffic data with a Google analytics account number UA-884652-XX. So going to uva.imodules.com/myPage.html would send its data to Google Analytics with that account number. Then when it is ready for production we are going to just change the domain name of the application and switch the DNS over to dardencommunity.darden.virginia.edu . So going to dardencommunity.darden.virginia.edu /myPage.html would send its data to Google Analtics with that SAME account number. Aside from having the testing domain data in the same profile are there any other issues/problems we may run into?
Reporting & Analytics | | Darden0 -
Google analytics reality check?
Looking back over a 9 month period tracking analytics with getclicky my site showed a 29% bounce rate, with only about 1/4 of visitors spending 1 minute or less on my site. I've recently implemented GA (removed old clicky code) and although traffic is strong, my site now shows a bounce rate of about 82%. Engagement stats also show that 82% of visitors spend between 0-10 seconds on my site. My site is built on Wordpress and the GA tracking code wasn't placed directly in the footer, my developer built a field in the admin area to insert the UA number which automatically adds the code to all pages. I've checked the code and the tracking seems to appear on all pages. I took a look at AW Stats. It corroborates GA and says that 80% of visitors are spending 0-30 seconds on the site. Potential issues/clues: browser tests show small loading problems in Internet Explorer 7,8,9 (the phone number at the top of the header loads on the wrong side of the page) and major issues in Internet Explorer 6 (site doesn't load at all in IE 6). The thing is no one who uses IE 6 is coming to the site. Second, the site gets a grade of C in YSlow, it's not lightning fast at the moment. GA is showing average page load of 2.4 seconds, but don't think either of these issues should cause an 82% 0-10 seconds engagement number. My site is content rich/focused with very minimal advertising. Content is accessible well above the fold. My question: Does the fact that AW Stats and GA agree mean that those numbers are accurate, or is there a bug I should be looking for? How to explain the clicky numbers?
Reporting & Analytics | | JSOC0 -
Google WM Tools: Canonical URL
Hello, We have a Google Webmaster Tools account under nlpca(dot)com but our preferred URL is www.nlpca(dot)com Do I need to delete the account and start over with the correct domain version, or is there a setting I can set somewhere? The code is in Joomla so I can't set a rel="canonical" in just the home page. Currently we have all home page version redirected to www.nlpca(dot)com Thanks!
Reporting & Analytics | | BobGW0 -
My rankings are up and down like a Yo-Yo. Any ideas why?
One of the companies that I work for sells the latest smartphones. It ranks fairly well for key terms in an extremely competitive field. Recently the rankings of one particular handset have been extremely erratic and do not follow the pattern of other products, I am wondering if anyone can offer some insight as to why this is happening? When the handset was first announced, it ranked fairly well (20th). This is good when you bare in mind that it is an ecommerce site competing with top tech sites. Prior to release, the ranking gradually declined to 33rd. I believe this may be due to reviews being publsihed (as a result of review handsets being sent out prior to release) as Google has started to favour reviews and news updates. I have seen a similar pattern with other handsets, reviews and news are favoured by Google prior to release, but once the product is available, my rankings rise and remain on the first page provided that links are still coming in. However, the rankings of this handset dropped from 27th to below 50th upon launch. The next week the ranking rose from >50th to 7th, dropped to below 50th again the following week, rose back to 9th the week after and has now dropped again to below 50th. These are by far the biggest changes in rankings I have seen on a week to week basis. The page has a strong link profile in order to achieve such a high ranking in the first place. With this being the case I have no idea why ranking can drop off so much in the space of a week? Update: I forgot to mention initially that I have the variant listed twice, one for each colour variant. All content is the same apart from the colour being altered in the appropriate places. This has not been a problem in the past, but perhaps it is due to recent changes to Google algorithm?
Reporting & Analytics | | pugh0 -
Should we "no-follow" archives or categories?
I'm reading some reports from my first crawl of 10K pages and I'm wondering if it's wise to mark the archives "no-follow." I have a WP tool that provides a tool that offers the no follow for categories or archives recommending to choose either one or the other but not both. What would be the best solution?
Reporting & Analytics | | JavaManOne0