Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I use noindex or robots to remove pages from the Google index?
I have a Magento site and just realized we have about 800 review pages indexed. The /review directory is disallowed in robots.txt but the pages are still indexed. From my understanding robots means it will not crawl the pages BUT if the pages are still indexed if they are linked from somewhere else. I can add the noindex tag to the review pages but they wont be crawled. https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html Should I remove the robots.txt and add the noindex? Or just add the noindex to what I already have?
Intermediate & Advanced SEO | | Tylerj0 -
Help needed on HTTPS
Hi Can anyone help me. I need some help with some example of large e-commerce sites going to https. Ideally these need to be e-commerce sites but any large site will be great, so that I can show the impact. The site can be based anywhere in the world, its more the impact I am after. Thanks Andy
Intermediate & Advanced SEO | | Andy-Halliday0 -
Homepage is losing rankings...need help!
Hi folks, Our company has been doing SEO for www.3dincites.com since October. Since the website hadn't been optimized at all, one of the first things we did was to add keyword-optimized page title and meta description to the homepage. The person who implemented didn't do it right the first time so we needed to change the page title again after a week or so. We also changed the preferred domain in GWT. Ever since those changes were implemented, the homepage has been decreasing in rankings for the the keywords we optimized it for. This is very surprising as the website has high-quality unique content and pretty solid backlinks. Moreover, the search traffic to other pages is through the roof (2K increase since October) so the client is happy, however, this decrease in rankings for the homepage (from 3d to 6th page for "3d IC technology" which is the main keyword) doesn't let me sleep well at nights. Any ideas why this is happening?
Intermediate & Advanced SEO | | zoomzoom0 -
My site falls more than 50 in two days, help me!
My site appeared in the top 10 for that link (http://www.vipgoldrj.com/paginas/ensaios.html) and not by this (http://www.vipgoldrj.com), was well at 2 months, and he suddenly disappeared, I wanted to know if he had been penalized and Google told me it was not. What should I do? The. Sorry my English, I am Brazilian and I'm using Google translator. Warning from SEOmoz staff: this is an escort site with full frontal nudity and is not safe for most workplaces.
Intermediate & Advanced SEO | | WebMaster0210 -
Where do I start? A little overwhelmed - Really appreciate any help.
Hello everyone, I was hoping to be able to get some advice if at all possible. I'm know there's some really skilled people on these forums. I'm not sure of the etiquette of posting links so if it's crossing the line to post the link of the site I'm working on please let me know. I was just hired as the Director of Digital Marketing for a decent sized company. I have a lot of experience with PPC, media buys and digital integration but unfortunately I'm a noob with SEO. I'm trying to learn as quickly as possible, and I'm reading everything I can (including everything I can find from Rand) The issue is that I have to start somewhere but there's so much that needs to be done that I'm getting a bit lost. We used to have 3 separate sites that ranked really well for our main keywords on Google, but with the growth of our company we consolidated the different market sites into one main one and 301'd the older ones. In doing so we lost nearly all of our search rankings. Our current site is: Http://www.GoldMaxUSA.com/ Does anyone have any advice on where to dive in here? I'm currently paying a ton each month for traffic through search PPC and contextual display, along with a bunch of geographically targeted brand awareness display campaigns, and I really need to show some progress on our organic search. Thanks in advance for any help!
Intermediate & Advanced SEO | | JFritton0 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0 -
Help with creating a widget
I would like to create a widget that I can give to other website owners to place on their blog. Obviously the point of doing this is to get backlinks. The widget is a simple calculator. (Think of it like a mortgage calculator). I see that there are two ways of creating widgets: 1. Javascript 2. Iframe I've been reading this excellent tutorial on building a widget using javascript and jquery:http://alexmarandon.com/articles/web_widget_jquery/. However, with my limited knowledge of javascript it's going over my head. I understand that if I offer a widget that is in an iframe that the links don't get counted as backlinks, but rather as links from my own site. But, what if I offered code like this: <iframe src="http://www.mysite.com" width="300" height="250"></iframe> This tool was provided by MySite Would that be helpful to me?
Intermediate & Advanced SEO | | MarieHaynes0 -
Need Help Finding Directories to Submit To
I am looking for a lot of free "do follow" technology directories to submit to. Does anyone know of a good directory or a list of some sort of technology directories or something similar? Actually, I guess any directory that has a technology category would be helpful.
Intermediate & Advanced SEO | | MyNet0