Website dropped out from Google index
-
Howdy, fellow mozzers.
I got approached by my friend - their website is https://www.hauteheadquarters.com
She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure.
I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good.
Any ideas why that happen? Any ideas what I should check?
P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though.
P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
-
"P.P.S. Just noticed that there is noindex x-robots-tag in headers"
That will do it. You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Get on this ASAP!
-
Hi Dmitri,
I also see the homepage in Google, but very few pages indexed beyond that, so there does appear to be a serious problem. I don't see anything immediately regarding problems with robots.txt or no index tags. Screaming Frog was able to crawl this site without any problems.
One thing I did see in the few pages that are indexed is the presence of a lot of internal search results pages being indexed.
For example:
https://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descThese two pages are exactly the same products, just in different order. This page also exists: https://www.hauteheadquarters.com/shop/rings/2 - the same products again. For all practical purposes all three of these pages are exactly the same content. Unfortunately, they price sort pages are not blocked from being crawled and indexed AND they are using self-referencing canonical tags.Based on pages like these and other duplicate/thin content issues across the site, I wouldn't rule out a Panda Penalty. It is highly likely that this site may have been penalized. Just because there is no manual action doesn't mean a penalty isn't in play.Recommendations:1. Audit sitewide content and determine which pages should be in Google2. Implement directives in the robots.txt file to prevent the URLs containing query parameters that don't provide unique content from being crawled.3. Implement canonical tags referencing the original URL without query parameters. Examplehttps://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descShould both be canonicalized to https://www.hauteheadquarters.com/shop/rings/24. Rebuild the XML sitemap and include only important URLs5. Resubmit the XML sitemap in GSC Wait a anywhere from a couple of days to a couple of weeks after resubmitting the sitemap, then evaluate if this has remedied the problem.Don't file a reconsideration request. This won't do any good because if it is a penalty, it was done via the algorithm and not manually.Hope that helps a little and good luck!Sincerely,Dana
-
Me too!
-
Absolutely, I'm glad you got things squared!
-
Thanks for response!
Well, basically, as I mentioned, the problem was due to http-header robots tag. So, after removing it, and requesting "fetch as google", it's all up and running now. The crawl time proves that as well.
Thanks for giving me idea for looking into cache times in the future though!
-
I see the homepage in my results - https://www.google.com/#q=site%3Ahttps%3A%2F%2Fwww.hauteheadquarters.com
Homepage was also cached today: http://webcache.googleusercontent.com/search?q=cache:https://www.hauteheadquarters.com&bav=on.2,or.r_cp.&biw=1920&bih=955&dpr=1&ion=1&ech=1&psi=EYbEV6CLN8aweJDvktgD.1472497162096.3&ei=EYbEV6CLN8aweJDvktgD&emsg=NCSR&noj=1
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Indexing Request - Typical Time to Complete?
In Google Search Console, when you request the (re) indexing of a fetched page, what's the average amount of time it takes to re-index and does it vary that much from site to site or are manual re-index request put in a queue and served on a first come - first serve basis despite the site characteristics like domain/page authority?
Intermediate & Advanced SEO | | SEO18050 -
Why is my website not ranking for it's brand name in SERPs but has been indexed by Google?
The website https://christchurch.crowneplaza.com has been live for a couple of months but is not being found in Google search results - even when searching for it's own brand name 'crowne plaza christchurch.' Google has indexed the site - but we are still not showing - https://www.google.co.nz/search?q=site%3Ahttp%3A%2F%2Fchristchurch.crowneplaza.com&rlz=1C1NHXL_enNZ735NZ735&oq=site%3A&aqs=chrome.0.69i59j69i57j69i58j69i59l2j69i65.896j0j7&sourceid=chrome&ie=UTF-8 Any ideas as to why? I think it may be because their are two versions of the site, http and https, both with their own rel=canonical tags. Could this be the cause? Any help much appreciated.
Intermediate & Advanced SEO | | Timmy30 -
Does including your site in Google News (and Google) Alerts helps with SEO?
Based on the following article http://homebusiness.about.com/od/yourbusinesswebsite/a/google-alerts.htm in order to check if you are included you need to run site:domain.com and click the news search tab. If you are not there then... I ran the test on MOZ and got no results which surprised me. Next step according to :https://support.google.com/news/publisher/answer/40787?hl=en#ts=3179198 is to submit your site for inclusion. Should I? Will it help? P.S.
Intermediate & Advanced SEO | | BeytzNet
This is a followup question to the following: http://moz.com/community/q/what-makes-a-site-appear-in-google-alerts-and-does-it-mean-anything0 -
Why extreme drop in number of pages indexed via GWMT sitemaps?
Any tips on why our GWMT Sitemaps indexed pages dropped to 27% of total submitted entries (2290 pages submitted, 622 indexed)? Already checked the obvious Test Sitemap, valid URLs etc. We had typically been at 95% of submitted getting indexed.
Intermediate & Advanced SEO | | jkinnisch0 -
Page position dropped on Google
Hey Guys, My web designer has recommended this forum to use, the reason being: my google position has been dropped from page 1 to page 10 in the last week. The site is weloveschoolsigns.co.uk, but our main business site is textstyles.co.uk the school signs are a product of text styles. I have been told off my SEO company, that because I have changed the school logo to the text styles logo, Google have penalised me for it, and dropped us from page 1 for numerous keywords, to page 10 or more. They have also said that duplicate content within the school site http://www.weloveschoolsigns.co.uk/school-signs-made-easy/ has also a contributed to the drop in positions. (this content is not on the textstyles site) Lastly they said, that having the same telephone number is a definate no no. They said that I have been penalised, because google see the above as trying to monopolise on the market. I don’t know if all this is true, as the SEO is way above my head, but they have quoted me £1250 to repair all the errors, when the site only cost £750. They have also mentioned that because of the above changes, the main text styles site will also be punished. Any thoughts on this matter would be much appreciated as I don't know whether to pay them to crack on, or accept the new positions. Either way I'm very confused. Thanks Thomas
Intermediate & Advanced SEO | | TextStylesUK0 -
Language Subdirectory homepage not indexed by Google
Hi mozzers, Our Spanish homepage doesn't seem to be indexed or cached in Google, despite being online for over a month or two. All Spanish subpages are indexed and have started to rank but not the homepage. I have submitted sitemap xml to GWTools and have checked there's no noindex on the page - it seems to be in order. And when I run site: command in Google it shows all pages except homepage. What could be the problem? Here's the page: http://www.bosphorusyacht.com/es/
Intermediate & Advanced SEO | | emerald0 -
Previously ranking #1 in google, web page has 301 / url rewrite, indexed but now showing for keyword search?
Two web pages on my website, previously ranked well in google, consistent top 3 places for 6months+, but when the site was modified, these two pages previously ending .php had the page names changed to the keyword to further improve (or so I thought). Since then the page doesn't rank at all for that search term in google. I used google webmaster tools to remove the previous page from Cache and search results, re submitted a sitemap, and where possible fixed links to the new page from other sites. On previous advice to fix I purchased links, web directories, social and articles etc to the new page but so far nothing... Its been almost 5 months and its very frustrating as these two pages previously ranked well and as a landing page ended in conversions. This problem is only appearing in google. The pages still rank well in Bing and Yahoo. Google has got the page indexed if I do a search by the url, but the page never shows under any search term it should, despite being heavily optimised for certain terms. I've spoke to my developers and they are stumped also, they've now added this text to the effected page(s) to see if this helps. Header("HTTP/1.1 301 Moved Permanently");
Intermediate & Advanced SEO | | seanclc
$newurl=SITE_URL.$seo;
Header("Location:$newurl"); Can Google still index a web page but refuse to show it in search results? All other pages on my site rank well, just these two that were once called something different has caused issues? Any advice? Any ideas, Have I missed something? Im at a loss...0 -
URL Parameter is not available in website which was monitored by Google
I was checking URL parameters section over Google webmaster tools. Google have monitored following parameters and exclude it from crawling. utm_campaign utm_medium utm_source I have built URLs with following tool to track visits from vertical search engine like Google shopping and other comparison shopping engines. http://www.google.com/support/analytics/bin/answer.py?answer=55578 So, I am quite confuse to see over my data. Will Google consider external URLs which are available with above parameters or require to consist on live website? Note: I am asking for my eCommerce website. http://www.lampslightingandmore.com/
Intermediate & Advanced SEO | | CommercePundit0