Why Can't Googlebot Fetch Its Own Map on Our Site?
-
I created a custom map using google maps creator and I embedded it on our site. However, when I ran the fetch and render through Search Console, it said it was blocked by our robots.txt file. I read in the Search Console Help section that: 'For resources blocked by robots.txt files that you don't own, reach out to the resource site owners and ask them to unblock those resources to Googlebot."
I did not setup our robtos.txt file. However, I can't imagine it would be setup to block google from crawling a map. i will look into that, but before I go messing with it (since I'm not familiar with it) does google automatically block their maps from their own googlebot? Has anyone encountered this before?
Here is what the robot.txt file says in Search Console:
User-agent: * Allow: /maps/api/js? Allow: /maps/api/js/DirectionsService.Route Allow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix Allow: /maps/api/js/ElevationService.GetElevationForLine Allow: /maps/api/js/GeocodeService.Search Allow: /maps/api/js/KmlOverlayService.GetFeature Allow: /maps/api/js/KmlOverlayService.GetOverlays Allow: /maps/api/js/LayersService.GetFeature Disallow: / Any assistance would be greatly appreciated. Thanks, Ruben
-
Awesome, Croy. That was extremely helpful.
Thanks,
Ruben
-
Hello Ruben!
Not too long ago, I saw a hangout with John Mueller who got the same question and gives a logical answer. Here's the URL at timestamp 46:06 https://www.youtube.com/watch?v=i_xnKznRNCc&feature=player_embedded#t=2766
I'd transcript if I had the time, but it will take you a couple of minutes. It essentially says that even though it is embedded, it won't access that block of content because maps.googleapis.com/robots.txt blocks it. If I understood wrong, please let me know!
I had the same problem where part of my content was coming from the pins INSIDE the map, but since it was blocked, it couldn't reach the content itself. Had to do a workaround
Hope it helps!
EDIT: Just jumping back in to say that you should look back at your site. Seems like Google can correctly crawl its own map and the content served through it. Our site has no more problems related to this. November the 15th I still had issues, but today I've tested multiple times and it looked okay.
Croy
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will putting a one page site up for all other countries stop Googlebot from crawling my UK website?
I have a client that only wants UK users to be able to purchase from the UK site. Currently, there are customers from the US and other countries purchasing from the UK site. They want to have a single webpage that is displayed to users trying to access the UK site that are outside the UK. This is fine but what impact would this have on Google bots trying to crawl the UK website? I have scoured the web for an answer but can't find one. Any help will be greatly appreciated. Thanks 🙂
Technical SEO | | lbagley0 -
Finding websites that don't have meta descriptions
Hi everyone, as a way to find new business leads I thought about targeting websites that have poor meta descriptions or where they are simply missing. A quick look at SERPs shows this is still a major issue for many businesses. Is there any way I can quickly find pages for which meta description is lacking? Thank you! Best regards, Florian
Technical SEO | | agencepicnic0 -
Duplicate Titles Aren't Actually Duplicate
I am seeing duplicate title errors, but when I go to fix the problem, the titles are not actually identical. Any advice? Becky
Technical SEO | | Becky_Converge0 -
Where did the 'Contributor To' area go in Google+
I went into my Google+ profile this morning to try to add a new guest blog in the 'Contributor To' section but I can't find it. Did they move it somewhere?
Technical SEO | | JonathanGoodman0 -
OSE says URL redirects to URL with trailing slash but it doesn't.
Site is www.example.com/folder/us and OSE says this URL redirects to www.example.com/folder/us/, but it does not. When I look at the OSE report for the latter version with the "/" it says "No Data Available For This URL". Why would that be? The original URL is www.example.com and it redirects to www.example.com/folder/us. Is this anything I need to worry about? I thought that the trailing / doesn't really mean much anymore but nonetheless, why does it think it redirects there?
Technical SEO | | rock220 -
Can we use our existing site content on new site?
We added 1000s of pages unique content on our site and soon after google release penguin and we loose our ranking for major keywords and after months of efforts we decided to start a new site. If we use all the existing site content on new domain does google going to penalized the site for duplicate content or it will be treated as unique? Thanks
Technical SEO | | mozfreak0 -
Webmaster Tools Site Map Question
I have TLD that has authority and a number of micro-sites built off of the primary domain. All sites relate to the same topic, as I am promoting a destination. The primary site and each micro-site have their own CMS installation, but the domains are mapped accordingly. www.regionalsite.com/ <- primary
Technical SEO | | VERBInteractive
www.regioanlsite.com/theme1/ <- theme 1
www.regioanlsite.com/theme2/ <- theme 2
www.regionalsite.com/theme3/ <- theme 3 Question: Should my XML site map for Webmaster Tools feed all sites off of the primary domain site map or are there penalties for this? Thanks.0 -
Thumbnail-based navigation like YouTube Sidebar - they don't use ALT tag
Notice on the YouTube sidebar, each video has a thumbnail and a title. But, for the ALT tag, YouTube simply uses the word "thumbnail". In the past, i was using a keyword phrase for my thumbnail ALT tag. I thought I was being clever. But is this superflous? We note that the A tag on the YouTube items, encompasses the SPAN that is the video + title. Does Google associate the text of the title as valid "anchor text" despite the existance of other info in that span --- e.g. like View Count and the User Name of the video creator? Thanks!
Technical SEO | | anthony-3054420