Website dropped out from Google index
-
Howdy, fellow mozzers.
I got approached by my friend - their website is https://www.hauteheadquarters.com
She is saying that they dropped from google index over night - and, as you can see if you google their name, website url or even site: , most of the pages are not indexed. Home page is nowhere to be found - that's for sure.
I know that they were indexed before. Google webmaster tools don't have any manual actions (at least yet). No sudden changes in content or backlink profile. robots.txt has some weird rule - disallow everything for EtaoSpider. I don't know if google would listen to that - robots checker in GWT says it's all good.
Any ideas why that happen? Any ideas what I should check?
P.S. Just noticed in GWT there was a huge drop in indexed pages within first week of August. Still no idea why though.
P.P.S. Just noticed that there is noindex x-robots-tag in headers... Anyone knows where this can be set?
-
"P.P.S. Just noticed that there is noindex x-robots-tag in headers"
That will do it. You are telling Google to take all of your pages out of Google. You set that at the web server level and so you will need to get into your apache or nginx setup
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Get on this ASAP!
-
Hi Dmitri,
I also see the homepage in Google, but very few pages indexed beyond that, so there does appear to be a serious problem. I don't see anything immediately regarding problems with robots.txt or no index tags. Screaming Frog was able to crawl this site without any problems.
One thing I did see in the few pages that are indexed is the presence of a lot of internal search results pages being indexed.
For example:
https://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descThese two pages are exactly the same products, just in different order. This page also exists: https://www.hauteheadquarters.com/shop/rings/2 - the same products again. For all practical purposes all three of these pages are exactly the same content. Unfortunately, they price sort pages are not blocked from being crawled and indexed AND they are using self-referencing canonical tags.Based on pages like these and other duplicate/thin content issues across the site, I wouldn't rule out a Panda Penalty. It is highly likely that this site may have been penalized. Just because there is no manual action doesn't mean a penalty isn't in play.Recommendations:1. Audit sitewide content and determine which pages should be in Google2. Implement directives in the robots.txt file to prevent the URLs containing query parameters that don't provide unique content from being crawled.3. Implement canonical tags referencing the original URL without query parameters. Examplehttps://www.hauteheadquarters.com/shop/rings/2?sort_price=ascandhttps://www.hauteheadquarters.com/shop/rings/2?sort_price=descShould both be canonicalized to https://www.hauteheadquarters.com/shop/rings/24. Rebuild the XML sitemap and include only important URLs5. Resubmit the XML sitemap in GSC Wait a anywhere from a couple of days to a couple of weeks after resubmitting the sitemap, then evaluate if this has remedied the problem.Don't file a reconsideration request. This won't do any good because if it is a penalty, it was done via the algorithm and not manually.Hope that helps a little and good luck!Sincerely,Dana
-
Me too!
-
Absolutely, I'm glad you got things squared!
-
Thanks for response!
Well, basically, as I mentioned, the problem was due to http-header robots tag. So, after removing it, and requesting "fetch as google", it's all up and running now. The crawl time proves that as well.
Thanks for giving me idea for looking into cache times in the future though!
-
I see the homepage in my results - https://www.google.com/#q=site%3Ahttps%3A%2F%2Fwww.hauteheadquarters.com
Homepage was also cached today: http://webcache.googleusercontent.com/search?q=cache:https://www.hauteheadquarters.com&bav=on.2,or.r_cp.&biw=1920&bih=955&dpr=1&ion=1&ech=1&psi=EYbEV6CLN8aweJDvktgD.1472497162096.3&ei=EYbEV6CLN8aweJDvktgD&emsg=NCSR&noj=1
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Apps content Google indexation ?
I read some months back that Google was indexing the apps content to display it into its SERP. Does anyone got any update on this recently ? I'll be very interesting to know more on it 🙂
Intermediate & Advanced SEO | | JoomGeek0 -
Google de-indexed a page on my site
I have a site which is around 9 months old. For most search terms we rank fine (including top 3 rankings for competitive terms). Recently one of our pages has been fluctuating wildly in the rankings and has now disappeared altogether from the rankings for over 1 week. As a test I added a similar page to one of my other sites and it ranks fine. I've checked webmaster tools and there is nothing of note there. I'm not really sure what to do at this stage. Any advice would me much appreciated!
Intermediate & Advanced SEO | | deelo5550 -
Homepage not ranking in Google AU, but ranking in Google UK?
Hey everyone, My homepage has not been ranking for it's primary keyword in Google Australia for many months now. Yesterday when I was using a UK Proxy and searching via Google UK I found my homepage/primary keyword ranked on page 8 in the UK. Now in Australia my website ranks on page 6 but it's for other pages on my website (and it always changes from different page to page). Previously my page was popping up at the bottom of page 1 and page 2. I've been trying many things and waiting weeks to see if it had any impact for over 4 months but I'm pretty lost for ideas now. Especially after what I saw yesterday in Google UK. I'd be very grateful if someone has had the same experience of suggestions and what I should try doing. I did a small audit on my page and because the site is focused on one product and features the primary keyword I took steps to try and fix the issue. I did the following: I noticed the developer had added H1 tags to many places on the homepage so I removed them all to make sure I wasn't getting an over optimization penalty. Cleaned up some of my links because I was not sure if this was the issue (I've never had a warning within Google webmaster tools) Changed the title tags/h tags on secondary pages not to feature the primary keyword as much Made some pages 'noindex' to try and see if this would take away the emphases on the secondary pages Resubmitted by XML sitemaps to Google Just recently claimed a local listings place in Google (still need to verify) and fixed up citations of my address/phone numbers etc (However it's not a local business - sells Australia wide) Added some new backlinks from AU sites (only a handful though) The only other option I can think of is to replace the name of the product on secondary pages to a different appreciation to make sure that the keyword isn't featured there. Some other notes on the site: When site do a 'site:url' search my homepage comes up at the top The site sometimes ranked for a secondary keyword on the front page in specific locations in Australia (but goes to a localised City page). I've noindexed these as a test to see if something with localisation is messing it around. I do have links from AU but I do have links from .com and wherever else. Any tips, advice, would be fantastic. Thanks
Intermediate & Advanced SEO | | AdaptDigital0 -
Website penalised can't find where the problem is. Google went INSANE
Hello, I desperately need a hand here! Firstly I just want to say that I we never infracted google guidelines as far as we know. I have been around in this field for about 6 years and have had success with many websites on the way relying only in natural SEO and was never penalised until now. The problem is that our website www.turbosconto.it is and we have no idea why. (not manual) The web has been online for more than 6 months and it NEVER started to rank. it has about 2 organic visits a day at max. In this time we got several links from good websites which are related to our topic which actually keep sending us about 50 visits a day. Nevertheless our organic visita are still 1 or 2 a day. All the pages seem to be heavily penalised ... when you perform a search for any of our "shops"even including our Url, no entries for the domain appear. A search example: http://www.turbosconto.it zalando What I will expect to find as a result: http://www.turbosconto.it/buono-sconto-zalando The same case repeats for all of the pages for the "shops" we promote. Searching any of the brads + our domain shows no result except from "nike" and "euroclinix" (I see no relationship between these 2) Some days before for these same type of searches it was showing pages from the domain which we blocked by robots months ago, and which go to 404 error instead of our optimised landing pages which cannot be found in the first 50 results. These pages are generated by our rating system... We already send requests to de index all theses pages but they keep appearing for every new page that we create. And the real pages nowhere to be found... Here isan example: http://www.turbosconto.it/shops/codice-promozionale-pimkie/rat
Intermediate & Advanced SEO | | sebastiankoch
You can see how google indexes that for as in this search: site:www.turbosconto.it rate Why on earth will google show a page which is blocked by the robots.txt displaying that the content cannot retrieved because it is blocked by the robots instead of showing pages which are totally SEO Friendly and content rich... All the script from TurboSconto is the same one that we use in our spanish version www.turbocupones.com. With this last one we have awesome results, so it makes things even more weird... Ok apart from those weird issues with the indexation and the robots, why did a research on out backlinks and we where surprised to fin a few bad links that we never asked for. Never the less there are just a few and we have many HIGH QUALITY LINKS, which makes it hard to believe that this could be the reason. Just to be sure we, we used the disavow tool for these links, here are the bad links we submitted 2 days ago: domain: www.drilldown.it #we did not ask for this domain: www.indicizza.net #we did not ask for this domain: urlbook.in #we did not ask for this, moreover is a spammy one http://inpe.br.way2seo.org/domain-list-878 #we did not ask for this, moreover is a spammy one http://shady.nu.gomarathi.com/domain-list-789 #we did not ask for this, moreover is a spammy one http://www.clicdopoclic.it/2013/12/i-migliori-siti-italiani-di-coupon-e.html #we did not ask for this, moreover and is a copy of a post of an other blog http://typo.domain.bi/turbosconto.it I have no clue what can it be, we have no warning messages in the webmaster tools or anything.
For me it looks as if google has a BUG and went crazy on judging our italian website. Or perhaps we are just missing something ??? If anyone could throw some light on this I will be really glad and willing to pay some compensation for the help provided. THANKS A LOT!0 -
Index or not index Categories
We are using Yoast Seo plugin. On the main menu we have only categories which has consist of posts and one page. We have category with villas, category with villa hotels etc. Initially we set to index and include in the sitemap posts and excluded categories, but I guess it was not correct. Would be a better way to index and include categories in the sitemap and exclude the posts in order to avoid the duplicate? It somehow does not make sense for me, If the posts are excluded and the categories included, will not then be the categories empty for google? I guess I will get crazy of this. Somebody has perhaps more experiences with this?
Intermediate & Advanced SEO | | Rebeca10 -
How long does google take to show the results in SERP once the pages are indexed ?
Hi...I am a newbie & trying to optimize the website www.peprismine.com. I have 3 questions - A little background about this : Initially, close to 150 pages were indexed by google. However, we decided to remove close to 100 URLs (as they were quite similar). After the changes, we submitted the NEW sitemap (with close to 50 pages) & google has indexed those URLs in sitemap. 1. My pages were indexed by google few days back. How long does google take to display the URL in SERP once the pages get indexed ? 2. Does google give more preference to websites with more number of pages than those with lesser number of pages to display results in SERP (I have just 50 pages). Does the NUMBER of pages really matter ? 3. Does removal / change of URLs have any negative effect on ranking ? (Many of these URLs were not shown on the 1st page) An answer from SEO experts will be highly appreciated. Thnx !
Intermediate & Advanced SEO | | PepMozBot0 -
Content Marketing: Should we build a separate website or built in site within the Website itself?
Hi Mozzers, Client: Big carpet cleaner player in the carpet cleaning industry Main Goal: Creating good content to Get more organic traffic to our main site Structure of the extra content: It will act like a blog but will be differentiated from the regular site by not selling anything but just creating good content. The look and design will be different from the client's site. SEO question: In terms of SEO, what would be the most beneficial for us to do, should we built in this new section/site outside or inside the client's site? I personally think that it should be separated from the main site because of the main reasons: A followed link to the main site Anchor texts implementation linking back to our service pages If we would to choose to build in this content, it would be highly beneficial for getting organic traffic within the main site but I am afraid this will not provide us any link juice since anchor texts won't be accounted the same since all of those would be located in the Nav bar of the main site. Can someone tell me what would be the best in terms of SEO? P.S: My boss doesn't agree with me and would rather go the second option (build in within the main site) that's why i am asking you guys what would be the most beneficial? Thank you Guys
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
What metrics is Google looking for to classify a websites as a "Store" or "Brand"
Our company is both a store and brand as we sell manufacture direct. We are not included in Google's "Related Searches for widgets:" Picture attached as reference (we are not selling computers ... just an example) What is Google looking for to pull these brands and stores? hXSLn.gif
Intermediate & Advanced SEO | | tatermarketing0