Googlebot being redirected but not users?
-
Hi,
We seem to have a slightly odd issue.
We noticed that a number of our location category pages were slipping off 1 page, and onto page 2 in our niche. On inspection, we noticed that our Arizona page had started ranking in place of a number of other location pages - Cali, Idaho, NJ etc. Weirdly, the pages they had replaced were no longer indexed, and would remain so, despite being fetched, tweeted etc.
One test was to see when the dropped out pages had been last crawled, or at least cached. When conducting the 'cache:domain.com/category/location' on these pages, we were getting 301 redirected to, you guessed it, the Arizona page. Very odd. However, the dropped out pages were serving 200 OK when run through header checker tools, screaming frog etc.
On the face of it, it would seem Googlebot is getting redirected when it is hitting a number of our key location pages, but users are not. Has anyone experienced anything like this? The theming of the pages are quite different in terms of content, meta etc.
Thanks.
-
Can anyone offer any insight on this? The issue shows no sign of improving, and our organic is tanking.
If this really is an issue on Google's side, and it appears based on our technical set up, how can I force Google to take note and reindex the pages?
Here are some more examples, from across a number of categories..
Category A:
Page that has dropped out of index: https://goo.gl/KEQ8Yh
Cached version of that page: https://goo.gl/DPzFWMPage that has dropped out of index: https://goo.gl/5KiQ4s Cached version of that page: https://goo.gl/myRWNg Category B: Page that has dropped out of index: https://goo.gl/pr3YQs
Cached version of that page: https://goo.gl/8SEYi5 Page that has dropped out of index: https://goo.gl/LqzDrg
Cached version of that page: https://goo.gl/iwPs45 Category
Page that has dropped out of index: https://goo.gl/YBZS7c Cached version of that page:https://goo.gl/n33QzG Page that has dropped out of index: https://goo.gl/Ht4gfO
Cached version of that page: https://goo.gl/u81vbA -
Weirdly, at the time this seemed to correct itself overnight without any fixes our end. However 6 months on and it has happened again... Only this time, all location pages have dropped out of index, but latest cache versions all redirect to the Kansas page.
Have checked log files for these relevant URLs, and all show 200 (and being crawled regularly). When you fetch them in SC the show as complete, and they are all still indexed in Bing.
Very confused!
-
Hi Nicola,
Have you checked your canonical tags to be sure that they are pointing to the correct URL? Sometimes with storefront pages people just copy/paste the same HTML, swapping out the contact information. But maybe they forgot to update the canonical tags?
Keep us updated!
-
Thanks for advice Chris. Have checked, and can't see any issues with htaccess or robots seem. A very strange one indeed!
-
Hi Anthony,
Desktop returns 200, whilst mobile returns a 302 to the mDOT site.
Thanks.
-
Hi Nicola,
Sounds like a very strange one indeed. I can't say I've ever heard of Google giving a redirect when other 3rd party crawlers give you a 200.
My only suggestion to begin the troubleshooting would be to have a look at your htaccess and maybe Robots to see if there are any directives in there addressing Google specifically.
Anthony's suggestion is also a great idea. At least you'll see what Google is picking up on those pages right now rather what they saw last time it was cached.
-
You mention these pages are returning 200 in SEO tools, but would you please share what are these URLs are returning when you Fetch as Googlebot in Google Search Console? Test it in both desktop and mobile versions.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Is Indexing my 301 Redirects to Other sites
Long story but now i have a few links from my site 301 redirecting to youtube videos or eCommerce stores. They carry a considerable amount of traffic that i benefit from so i can't take them down, and that traffic is people from other websites, so basically i have backlinks from places that i don't own, to my redirect urls (Ex. http://example.com/redirect) My problem is that google is indexing them and doesn't let them go, i have tried blocking that url from robots.txt but google is still indexing it uncrawled, i have also tried allowing google to crawl it and adding noindex from robots.txt, i have tried removing it from GWT but it pops back again after a few days. Any ideas? Thanks!
Intermediate & Advanced SEO | | cuarto7150 -
Redirecting a Few URLs to a New Domain
We are in the process of buying the blog section of a site. Let's say Site A is buying Site B. We have taken the content from Site B and replicated it on Site A, along with the exact url besides the TLD. We then issued 301 redirects from Site B to Site A and initiated a crawl on those original Site B urls so Google would understand they are now redirecting to Site A. The new urls for Site A, with the same content are now showing up in Google's index if we do a site:SiteA.com search on the big G. Anyone have any experience with this as to how long before Site A urls should replace Site B urls in the search results? I undestand there may be a ranking difference and CTR difference based on domain bias, etc... I'm just asking if everything goes as planned and there isn't a huge issue, does the process take weeks or months?
Intermediate & Advanced SEO | | seoaustin0 -
Worth it to redirect .swf during a website migration?
Recently one of our clients switched web providers. We've been implementing 301 redirects to the new URL structure for the most important pages. We noticed upon doing a site search, Google has indexed a .swf URL on page 3 of search results. Please see attached image. 1. Should I implement a redirect for this? There are no links to this filetype.
Intermediate & Advanced SEO | | EEE3
2. There are three others (.swf) that are indexed I would think the best thing to do is to have the new website return a 404 for this page, as it's currently doing, and it will eventually drop out of the index. Any other suggestions? Thanks! g2tur4r0 -
How to 301 redirect old wordpress category?
Hi All, In order to avoid duplication errors we've decided to redirect old categories (merge some categories).
Intermediate & Advanced SEO | | BeytzNet
In the past we have been very generous with the number of categories we assigned each post. One category needs to be redirected back to blog home (removed completely) while a couple others should be merged. Afterwords we will re-categorize some of the old posts. What is the proper way to do so?
We are not technical, Is there a plugin that can assist? Thanks0 -
301 redirect on Windows IIS. HELP!
Hi My six-year-old domain has always existed in four forms: http://www**.**mydomain.com/index.html http://mydomain.com/index.html http://mydomain.com/ http://www.mydomain.com My webmaster claims it’s “impossible” to do a 301 redirect from the first three to the fourth. I need simple instructions to guide him. The site’s hosted on Windows running IIS Here’s his rationale: These are all the same page, so they can’t redirect to themselves. Index.html is the default page that loads automatically if you don’t specify a page. If I put a redirect into index.html it would just run an infinite redirect loop. As you can see from the IIS set up, both www.mydomain and mydomain.com point to the same location ( VIEW IMAGE HERE ) _Both of these use index.html as the default document ( VIEW IMAGE 2 HERE ) _
Intermediate & Advanced SEO | | Jeepster0 -
Is this a Correct Time to Use 302 Redirects?
Hi Mozzers! We are going through a rebranding process, and as of this morning we have 3 domains, all with identical content. For example (not real domain names): www.fantastic.com
Intermediate & Advanced SEO | | Travis-W
www.fantasticfireworks.com
www.fireworks.com We are using 3 domains to ease the rebranding transition. We currently only want people to visit 'www.fantastic.com,' and if they visit the other 2 domains we want them redirected. Since we will be using these other domains eventually, should we use 302 redirects instead of 301s? The other domains are new and do not have any domain authority or sites linking in, so we do not need to worry about link juice. Does it really matter what type of redirect we use? Thanks!0 -
What happens with a 301 redirected page?
Hi All, What happens with an indexed page that I 301 redirect?
Intermediate & Advanced SEO | | BeytzNet
Is it removed from the Google index after a while? Thanks0 -
Reducing pages with canonical & redirects
We have a site that has a ridiculous number of pages. Its a directory of service providers that is organized by city and sub-category of the vertical. Each provider is on the main city page, then when you click on a category, it will only show those folks who offer that subcategory of this service. example: colorado/denver - main city page colorado/denver/subcat1 - subcategory page There are 37 subcategories. So, 38 pages that essentially have the same content - minus a provider or two - for each city. There are approx 40K locations in our database. So rough math puts us at 1.5 million results pages, with 97% of those pages being duplicate content! This is clearly a problem. But many of these obscure pages do rank and get traffic. A fair amount when you aggregate all these pages together. We are about to go through a redesign and want to consolidate pages so we can reduce the dupe content, get crawl budget allocated to more meaningful pages, etc. Here's what I'm thinking we should do with this site, and I would love to have your input: Canonicalize Before the redesign use the canonical tag on all the sub-category pages and push all the value from those pages (colorado/denver/subcat1, /subcat2, /subcat3... etc) to the main city page (colorado/denver/subcat1) 301 Redirect On the new site (we're moving to a new CMS) we don't publish the duplicate sub-category pages and do 301 redirects from the sub-category URLs to the main city page urls. We'd still have the sub-categories (keywords) on-page and use some Javascript filtering to narrow results. We could cut to the chase and just do the redirects, but would like to use canonicalization as a proof of concept internally at my company that getting rid of these pages is a good thing, or at least wont have a negative impact on traffic. i.e. by the time we are ready to relaunch traffic and value has been transfered to the /state/city page Trying to create the right plan and build my argument. Any feedback you have will help.
Intermediate & Advanced SEO | | trentc0