How preproduction website is getting indexed in Google.
-
Hi team,
Can anybody please help me to find how my preproduction website and urls are getting indexed in Google.
-
As Eric hinted, the best method to prevent any pages being indexed would be to use htaccess password protection dialog on your development site. It's fairly easy to implement. You can find instructions to do so here: http://www.htaccesstools.com/articles/password-protection/
-
Hi Anoop! Have everyone's answers helped? Do you still have any questions?
-
Anoop, when a 'development' or 'preproduction' website or subdomain is getting indexed, that means that you haven't stopped the search engines from crawling it. The search engines, especially Google, are very aggressive at crawling, and they will crawl just about any URL that they find. It seems as though all you have to do is visit that page and it's going to get crawled.
Best way to stop Google from crawling (then indexing) a website is to stop it from getting crawled using the robots.txt file. Keep in mind, though, that even if you tell them to stay out of it using the robots.txt file they will still index those URLs.
The only way to stop Google from crawling would be to password protect the website or make it available only on a private server, or available via VPN only.
-
In addition to noindexing the pages using the meta tag, if you have WMT / Search Console set up, you can request Google remove those URLs from their index for the time being. I've found that this may take up to a couple of hours from the removal request to the time of actual removal.
As to how they were found, there's a good chance that Google crawled a link to a preproduction webpage and went from there.
-
Hi
To prevent most search engine web crawlers from indexing a page on your site, place the following meta tag into the section of your page:
To prevent only Google web crawlers from indexing a page:
You should be aware that some search engine web crawlers might interpret the
noindex
directive differently. As a result, it is possible that your page might still appear in results from other search engines.here is complete guide: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1
-
Hi,
Have you noindexed & nofollowed the site and pages? I would also suggest you block all crawlers by disallowing access in the robots.txt file.
Do you know if this has all been done?
-Andy
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google only indexed 19/94 images
I'm using Yoast SEO and have images (attachments) excluded from sitemaps, which is the recommended method (but could this be wrong?). Most of my images are in my posts; here's the sitemap for posts: https://edwardsturm.com/post-sitemap.xml I also appear on p1 for some good keywords, and my site is getting organic traffic, so I'm not sure why the images aren't being indexed. Here's an example of a well performing article: https://edwardsturm.com/best-games-youtube-2016/ Thanks!
Technical SEO | | Edward_Sturm0 -
Why wont google Index this page?
A week ago i accidentally changed this page settings in my CMS to "disable & dont index" as i was going to replace this page with another, but this didnt happen, but i forgot to switch the settings back! http://www.over50choices.co.uk/funeral-planning/funeral-plans Anyhow in an effort to get it back up quickly i submitted in GWTs but its still not indexed. When i use several SEO on page checking tools it has the Meta Title data as "Form" and not the correct title. Any ideas please? Yours frustrated Ash
Technical SEO | | AshShep10 -
My website keeps getting hit every other month. What should I do?
Since April 2012, my website impressions has dropped about 88% according to GWT. Every other month or so, the impressions are dropping by about 30%. My total traffic (visitors, not impressions) has dropped by about 25% but now I am getting lots of junk traffic. A few of the major keywords I used to rank for are still ranking in the top 10 but only in the USA. Lots of the keywords have gone to page 2 or 3 in the US and are gone to hell in other countries. Now I know I'm mostly responsible for this mess. About 3 years ago, I hired a freelancer to write news for my blog and she did a great job for quite some time so I stopped monitoring her work for duplicated content. Unfortunately, she started to provide me with copied content after a while and did so for almost 9 months before I noticed it. I had also hired RankPoop - errr I mean RankPop - to build some backlinks and that eventually got me in trouble too. I got an unnatural links warning in GWT in July 2012. Since then, I had more than 50% of the bad links taken down. There are still lots of them but they sure account for way less than 50% of all the backlinks. I have not submitted a reconsideration request yet as I haven't compensate for all the links taken down yet. I also started adding LOTS of fresh, unique and useful content to the website. I've added near 400 articles (sometimes up to 7 or 8 articles a day) over the last 5 months. I've also set lots of the duplicated posts to "noindex" and when they're not indexed anymore, I completely removed them in order to avoid any copyright issues (some were 100% identical to the source). I keep doing this gradually to avoid 404 errors. In early March of this year, I did a complete redesign of the site. The navigation structure stayed the same and visually, the layout is quite the same although the graphical elements are much more professional and the site is much faster. As much as I would've like to avoid a complete redesign, major technical issues from the previous design (and development platform) was now forcing me to do so. Unfortunately, I updated the website design right before the last Panda update so now I don't know if the recent traffic lost is due to the new design or because of Panda... or both. Google is like a police officer who repeatedly give you tickets for the same offense, yet they won't tell you what that offense is. My website is located at http://www.thewebhostinghero.com/ Any advice is welcomed. P.S. sorry for my english, I speak french.
Technical SEO | | sbrault740 -
How should i knows google to indexed my new pages ?
I have added many products in my ecommerce site but most of the google still not indexed yet. I already submitted sitemap a month ago but indexed process was very slow. Is there anyway to know the google to indexed my products or pages immediately. I can do ping but always doing ping is not the good idea. Any more suggestions ?
Technical SEO | | chandubaba1 -
Looking for someone to help get reconsidered in Google search.
Hi Guys, I need some extra hands and recently got some notifications from Google about unnatural links and this is something I've known about already. I've tried a few times to email webmasters but there is little to no results. I'm looking for some individuals that are great at getting reconsidered to Google. The site I am working on already is being ranked but this is a partial ban. Please PM me your rates and success with previous campaigns. Ideally, I want a flat rate for 100% guarantee reconsideration. TIA!
Technical SEO | | William.Lau0 -
Websites not being included on google from mobiles?
Hi, Just had a call from a guy saying that google have made a statement saying that it will be stopping people finding websites from mobule devices if they dont have a mobile domain name. Doesn anyone know anything about any Google statements or is this just rubbish?
Technical SEO | | Ant710 -
Google Has Indexed Most of My Site, why won't Bing?
We've got 600K+ pages indexed by Google and have submitted our same sitemap.xml's to Bing, but have only seen 100-200 pages get indexed by Bing. Is this fairly typical? Is there anything further we can do to increase indexation on Bing?
Technical SEO | | jamesti0 -
Has google panelized us ? If so, why ? How do I know if our website is panelized ?
We were ranked on first page among top 5 position a year ago for most of our pages. On one fine day, google decided to drop us from the results although google keeps indexing our pages. Google index our pages regularly but doesn't show them in its results. All google traffic we receive is for our own site name and its variations. I wanted to know - how do we know if google has panelized us. Why has google panelized us ? If they have panelized us, what can we do to get out of it ? Also I wanted to know if any tool will help me identify such thing. We have not done any link building. Our site page rank is 4 (it was 5 few months ago). All we did was on page optimization. Thanks for your help!
Technical SEO | | seoidea0