Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Titles and Metas disappeared?? Help please!
Hi everyone, Had a bit of a concern today, my weekly report has come through and my crawl issues have skyrockets by over 400! It says my metas and titles are missing but when I check through the site manually they seem to all still be there, I'm getting the same problem when I use screaming frog to crawl the site. I would really appreciate an explanation from someone as to why this is happening as I am quite confused about the situation. Thank you people Charlie Our website is www.homelogic.co.uk 🙂
Intermediate & Advanced SEO | | MintySEO0 -
Video SERP Help
Hello Friends,
Intermediate & Advanced SEO | | KINQDOM
I try to appear on search results of property related search terms with my property videos. Here is a sample property video
http://www.antalyahomes.com/videositemap.asp May you please check it and tell me what I do wrong? Thanks in advance for your time.0 -
How to take out international URL from google US index/hreflang help
Hi Moz Community, Weird/confusing question so I'll try my best. The company I work for also has an Australian retail website. When you do a site:ourbrand.com search the second result that pops up is au.brand.com, which redirects to the actual brand.com.au website. The Australian site owner removed this redirect per my bosses request and now it leads to a an unavailable webpage. I'm confused as to best approach, is there a way to noindex the au.brand.com URL from US based searches? My only problem is that the au.brand.com URL is ranking higher than all of the actual US based sub-cat pages when using a site search. Is this an appropriate place for an hreflang tag? Let me know how I can help clarify the issue. Thanks,
Intermediate & Advanced SEO | | IceIcebaby
-Reed0 -
Title tags seem to be going against Rand and suggestions here, what can I do? Can you help me please?
Hi to all, Ok so I have been beating myself up trying to figure out how to create a good ranking title for my websites homepage. My competition uses commas in its title tags But these are 100% against what Rand and many posts say here. I watched his whiteboard where he says don't use commas. There are posts that say, don't use commas. The biggest one is mortonbuildings(dot)com - followed by - lesterbuildings(dot)com They basically have the same title tags with commas. So what's the catch here? What am I missing? of course these sites have Great Domain Authority, but I swear what they are doing is going against what Rand and many other seo professional's are preaching. I thought it should read better for visitors? But to me it looks like keyword stuffing with a brand name? Any help is greatly appreciated to let me know if these commas and strategies are good to do? or bad news? Again thank you for your time to help a small business. Chris
Intermediate & Advanced SEO | | asbchris0 -
Rankings Nose Diving Help Needed
Hey There SEO Community, I am trying to help these people: http://goo.gl/B1smo They once ranked in the top 10 for "lifewave" and "lifewave patches" but have disappeared. Any idea why and what I can do to help? Thanks!
Intermediate & Advanced SEO | | siteoptimized0 -
Will links to a subdomain help it rank?
I have an affiliate subdomain on a larger company's domain. (For example I have: www.victor.company.com on www.company.com). Would working to attain backlinks to the subdomain help it rank or will I just be putting forth my effort and helping the domain rank?
Intermediate & Advanced SEO | | VictorVC0 -
XML Sitemap instruction in robots.txt = Worth doing?
Hi fellow SEO's, Just a quick one, I was reading a few guides on Bing Webmaster tools and found that you can use the robots.txt file to point crawlers/bots to your XML sitemap (they don't look for it by default). I was just wondering if it would be worth creating a robots.txt file purely for the purpose of pointing bots to the XML sitemap? I've submitted it manually to Google and Bing webmaster tools but I was thinking more for the other bots (I.e. Mozbot, the SEOmoz bot?). Any thoughts would be appreciated! 🙂 Regards, Ash
Intermediate & Advanced SEO | | AshSEO20110 -
Getting Google to Correct a Misspelled Site Link...Help!
My company website recently got its site links in google search... WooHoo! However, when you type TECHeGO into Google Search one of the links is spelled incorrectly. Instead of 'CONversion Optimization' its 'COversion Optimization'. At first I thought there was a misspelling on that page somewhere but there is not and have come to the conclusion that Google has made a mistake. I know that I can block the page in webmaster tools (No Thanks) but how in the crap can I get them to correct the spelling when no one really knows how to get them to appear in the first place? Riddle Me That Folks! sitelink.jpg
Intermediate & Advanced SEO | | TECHeGO0