How did my dev site end up in the search results?
-
We use a subdomain for our dev site. I never thought anything of it because the only way you can reach the dev site is through a vpn. Google has somehow indexed it. Any ideas on how that happened? I am adding the noindex tag, should I used canonical? Or is there anything else you can think of?
-
Personally, I'd still recommend using robots.txt to disallow all crawlers, even if more steps are taken.
-
Don't use tool removal, it can go bad indeed. Now, are you sure that there are no external links coming from anywhere?
For now I'd recommend putting noindex, nofollow on that dev subdomain and do manual recrawl through GWT.
-
It just uses internal links. Do you think I should try the webmaster tools removal? That seems like it could go wrong.
-
I never used screaming frog, does it check both external and internal links?
-
I have ran screaming frog to see if there are any links to any pages and but couldn't see any. Even if Google did try to follow it the firewall would stop them. It is so strange.
-
Then my first assumption is that it's linked from somewhere - read my comment a little above.
-
Then there is a leak somewhere - Google bots can "see" your subdomain.
Or it's been simply linked from somewhere. Then Google will try to follow the link and that would make it indexed.
-
They are telling me that there are no holes, and I have tried getting to the pages but can not do it unless I am on my vpn.
-
We never updated the robots.txt because the site was behind a firewall. If you click on any of the results it will not load the page unless on my VPN.
-
Robots.txt won't help anyhow. Bots still can see that there is such directory, they just won't see what's inside of those directories/subdomains.
-
Hi there.
If what you say is true, then there are only two answers: you got a leak somewhere or your settings/configuration is messed up.I'd say go talk to your system admin and make sure that everything what's supposed to be closed is closed, IPs, which are supposed to be open for use are open and those IPs only.
-
Have you updated the dev sites robots.txt to disallow everything? It is up to the bot to listen, but that combined with removing all of the dev URLs from Google Webmaster tools should do the trick.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Interested ranking results
Hello, I have different pages that are written on the same model. I updated their content 2 weeks ago. Some of those climbed to 1 st page or beginning or 2 nd some didn't move or stayed where they originally were, 3 rd of 4 th page ? My question is whether there is a longer delay for some of them to rank (due to the term being more competitive for example) or if I should look somewhere else in maybe something that I wrote in the content of the pages that ranked versus those than didn't move. Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Sitemap for SmartPhone site
Hello I have a smartphone site (e.g.m.abc.com). To my understanding we do not need a mobile sitemap as its not a traditional mobile site. Shall I add those mobile site links in my regular www XML sitemap or not bother to add the links as we already have rel = canonical (on m.abc.com ) and rel= alternate in place (on www site) to respective pages. Please suggests a solution. I really look forward to an answer as I haven't found the "official" answer to this question anywhere.
Intermediate & Advanced SEO | | AdobeVAS0 -
My website (non-adult) is not appearing in Google search results when i have safe search settings on. How can i fix this?
Hi, I have this issue where my website does not appear in Google search results when i have the safe search settings on. If i turn the safe search settings off, my site appears no problem. I'm guessing Google is categorizing my website as adult, which it definitely is not. Has anyone had this issue before? Or does anyone know how to resolve this issue? Any help would be much appreciated. Thanks
Intermediate & Advanced SEO | | CupidTeam0 -
Is this ok for content on our site?
We run a printing company and as an example the grey box (at the bottom of the page) is what we have on each page http://www.discountbannerprinting.co.uk/banners/vinyl-pvc-banners.html We used to use this but tried to get most of the content on the page, but we now want to add a bit more in-depth information to each page. The question i have is - would a 1200 word document be ok in there and not look bad to Google.
Intermediate & Advanced SEO | | BobAnderson0 -
301s from previous site
Hi! Got quite a tricky problem regarding a client, http://www.muchbetteradventures.com/ and their previous site, http://v1.muchbetteradventures.com/ Here's the background: We have approx 1500 'listing' pages like this: http://v1.muchbetteradventures.com/listing/view/1925/the-barre-des-ecrins-or-the-dome-des-ecrins-mountaineering-trip They bring in min 2k hits/month, and also add to the overall site authority I suspect. They will eventually all have a home on main domain. When they do, they will also each have been rewritten to be unique, so the value of them will increase (many are currently not). We also have landing pages like this: http://v1.muchbetteradventures.com/view/559/volunteering-holidays- which despite being hideous are ranked fairly well (page 1 for key terms). We cannot currently fulfil all these on main domain, but do not want to shut them down and lose positioning. Choices as I see it: Make a landing page e.g. muchbetteradventures.com/volunteering and a) redirect from old landing page, b) redirect all related 'listings' to this page. May help preserve rankings of main landing page (the most important), but not of any listings? Import all listings to have a home on main domain, (probably as children of a landing page, but not rewritten to be unique just yet). Make them not accessible from homepage, and change functionality of them so that new visitors from google are told we cannot currently help them with this trip. This is more work to complete so will take longer to do and is a distraction from our core focus so needs good justification! Stay running largely as we are, slowly redirecting 1 page at a time as we carry over more and more options to main domain. This will take over 12 months min.
Intermediate & Advanced SEO | | neooptic0 -
Niche sites: how to optimize them?
Dear SEOmozzers, I am focusing on those keywords I find using the "Keyword Analysis" tool that are not too competitive (among 20%-30% competitiveness). I then buy domains that include those keywords. The sites are in Italian and targeting the Italian search engines, so the competition is lower than it would be in the US. Basically, I'd like to build niche sites and I'd like to ask a few questions that I hope somebody with a good experience in this field can answer: How do I optimize a niche site. Specifically, how do I go about link building? How many backlinks should I get to see some results? How long does it take for a typical niche site to start appearing in the search engines for a certain keyword, after launching an effective link building campaign? Please kindly provide any recommendations you believe to be important when building niche sites. For example, is there a company/professional you know that specializes in this field and who is trustworthy/reliable? Thank you very much for your help. All best, Sal
Intermediate & Advanced SEO | | salvyy0 -
Organic Search Problems?
Hey guys, I am in need of a little help! I am currently an aspiring SEO (trying to absorb as much information as I can and implement changes to help my site organically)... Most of my experience revolves around SEM. That being said, I have a problem. My site is doing well through paid search... great quality scores, etc. However, the content on my site (and even my site as a whole) does not "appear" to rank well in Organic. To explain further... My site is federalautoloan.com... and when I type in exact article names (or even federal auto loan) into Google, nothing shows up. And yes, my content is all original/unique content. I've even recently added a unique Calculator to my site. site:federalautoloan.com in the search bar shows results for all of my pages... but it just seems as though Google does not like my site for some reason. At least in Organic. The odd thing is, none of my other sites have this problem. Do you guys have any advice? The only thing I can think of is that somehow my 301 redirect was performed improperly. Yes, I had a permanent redirect performed on my site about 4 months back. The URL we were using prior just wasn't performing as well in Paid Search. But seeing as how that is the preferred method by Google... I'm really at a loss... Again, my site is FederalAutoLoan.com. Any help would be GREATLY appreciated. Even generic SEO advice would be appreciated. Edit: Two other things to note... I have plugged my site into the SEOmoz Pro tool... the tool is not showing any issues for my site. I am also making use of Google Webmaster Tools and the only error that shows up for my site is a Soft 404 for one of my pmcs... Not sure why it is even pulling one of my pmcs... but as far as I can tell, there really shouldn't be any problems. Note on the 404 for anyone who might give a response on that issue... http://www.seoconsultants.com/tools/headers returns a 200 OK response. Edit2: Question presented below.
Intermediate & Advanced SEO | | WPColt0 -
Press Release Sites
Ok, I am getting a lot of conflicting information about press release sites. i have been doing press release's for a while (mostly manually), I have also tried a few companies that claim to do it well (never do). After the Panda update the PR sites I have been using are just not as effective. Does anyone else have this problem or are there better PR sites that can be recommended.
Intermediate & Advanced SEO | | TomBarker820