Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If my website do not have a robot.txt file, does it hurt my website ranking?
After a site audit, I find out that my website don't have a robot.txt. Does it hurt my website rankings? One more thing, when I type mywebsite.com/robot.txt, it automatically redirect to the homepage. Please help!
Intermediate & Advanced SEO | | binhlai0 -
Help With Href Lang Implementation
Hi, We've received the following Href Lang errors in Google Search Console. After much research and playing around with the tags we're still no closer to fixing the issues. ERRORS; http://prntscr.com/du8ei8 AND http://prntscr.com/du8evi As you'll see, Google is telling us that some URLs do not have return tags, but they do seem to! Does anyone know of a pro that can provide a service to fix this for us? Many thanks in advance, Lee.
Intermediate & Advanced SEO | | Webpresence0 -
Please help need experienced eyes
We own discount banner printing and we are trying to rank 1 for pvc banners or vinyl banners and cannot understand for example how the below is correct, we did suffer a link penalty years ago but we fixed this and the domain has some good links (more and better quality than the sites above us) and cannot understand how we rank below most of the sites above us? If we type on for example pvc banners we get http://www.bannershop.co.uk/cats/pvc_banners.htm https://www.hfe-signs.co.uk/banners.php http://bannerprintingandroid.co.uk/pvc-banners/ http://www.discountbannerprinting.co.uk/banners/vinyl-pvc-banners.html (our website) And if we type in vinyl banners we get http://www.vistaprint.co.uk/banners.aspx http://www.bigvaluebanners.co.uk/ http://vinylbannersprinting.co.uk/ http://www.discountdisplays.co.uk/html/vinyl_banners.html https://www.buildasign.co.uk/banners http://www.monkey-print.com/outdoor banners/budget-outdoor-banners http://www.discountbannerprinting.co.uk/banners/vinyl-pvc-banners.html (Our website)
Intermediate & Advanced SEO | | BobAnderson0 -
Category 404 Error in Wordpress | Help!!!
Hello Gurus hope everyone is having a fantastic day. Right so I've been pulling my hair with this 404 error on links as such: htttp://manvanlondon.co.uk/category/clients/removals/man-and-van-wandsworth This link appears in the category page clients and the /removals/man-and-van-wandsworth part is the link that should take the user on the Man and Van Wandsworth page in the/ from the footer. However this link and all other links in the footer on these category pages/posts appear to be broken ONLY on this category pages, thus creating 404 errors. And those pages(i.e man and van Wandsworth) are not even categorized. The website is www.manvanlondon.co.uk . We tried various things on Wordpress and nothing is working including non-indexing. Has anyone met this problem before? Is there a way to fix it? Thank you for your time, and hope my explanations make sens. Monica
Intermediate & Advanced SEO | | monicapopa0 -
Help me solve a keyword ranking mystery please
I posted this and had some help (thank you!) but found some new things, so I thought I'd just start a new thread so no info. is missed. Hi everyone, I'm new here 🙂 So far I've had wonderful success seo wise and none of the updates (Penguin nor Panda) affected any sites, until this one. For example, one site has 7 keywords I'm optimizing for. Out of those 7, all but 2 (and variations of the 2 - one word vs long-tail) completely tanked. These keywords were all on page 2/3. One of the two survivors never budged from page 2 (it's a brand keyword so I was very happy to finally get it to page 2) Now when I check rankings, the other terms show up in the 200-400 spots, but NOT for the URL I was optimizing for (category page) but instead for random products in the category. The only thing I've done differently with the 2 keywords that are still doing well, was focus - we did more link-building for those, but not an extreme amount. Never over-optimize. My question is, how did 2 survive and 5 are still floating up and down. Last night I saw one go up 122 spots, now today down 14. I'm really struggling with this. I just ran another diagnostic crawl here and the report found 0 errors and 0 warnings. I checked category content with a plagiarism checker and found some external duplicate content which I've already taken care of. No critical warnings/messages in WMT either. I'm stumped 😞 Thank you for any help.
Intermediate & Advanced SEO | | Freelancer130 -
I need help with setting the preferred domain; www. or not??
Hi! I'm kinda new to the SEO game and struggling with this site I'm working on: http://www.moondoggieinc.com I set the preferred domain to www. in GWT but I'm not seeing it reroute to that? I can't seem to get any of my internal pages to rank, and I was thinking it's possiblly b/c of a duplicate content issue cause by this problem. Any help or guidance on the right way to set preferred domain for this site and whiy I can't get my internal pages to rank? THANKS! KristyO
Intermediate & Advanced SEO | | KristyO0 -
Rel=alternate to help localize sites
I am wondering about the efficiency of the rel=alternate tag and how well it works at specifically localizing content. Example: I have a website on a few ccTLD's but for some reason my .com shows up on Google.co.uk before my .co.uk version of my page. Some people have mentioned using rel=alternate but in my research this only seems to be applicable for duplicate content in another language. If I am wrong here can somebody please help me better understand this application of the rel=alternate tag. All my research leads me to rel=alternate hreflang= and I am not sure that is what I want. Thanks,
Intermediate & Advanced SEO | | DRSearchEngOpt
Chris Birkholm0 -
Multiple Keyword Research Questions, Help
Hello , I've been trying for several days to understand how keyword research works for a multi purpose website,I've read guides, articles even some chapters from the book" The Art of Seo" by O'Reilly and still no luck. It seems i can't wrap my head around keyword research,lets say I have a social gaming community website and I'm trying to rank it first on some low competition keywords + some long tail keywords.The website has functions like leaderboards, profiles,events, competitions,etc so it's not actually a news related website but it will have a blog. My website being on the games niche It would imply that I should target words that contain the word "Games" but this word generates millions of searches globally so ranking first its nearly impossible if the website is brand new. This made me pursue generic keywords formed with 2 / 3 words like fresh games, new games, mmorpg games, fps games,etc which still generate lets say 30.000 searches globally each. Due to the different areas of the website like latest game events,latest games competitions,etc I'm confused If i should pursue website specific keywords like latest games events, fresh games events, latest games competitions, upcoming games competitions but these too generate 30.000 global searches each,so... 0.should i use generic keywords or keywords that include site features? So let's say I decide to pursue generic "games" keywords,due to a high competition based on the keyword I decide to go a layer deeper and for the keyword "fresh games" I obtain keywords like** "fresh games 2011,top fresh games 2011, upcoming fresh games** " and thus building a list of 30 keywords that contain " fresh games".If i do this for the rest of the keywords: ** new games, mmorpg games, fps games,etc** I end up with a list of 10.000 keywords or more since each keyword generates other keywords. Is this the correct approach ? since generating 10.000 keywords sounds a lot and I'm getting the feeling that It's not how it supposed to be done,like were would I insert 10.000 keywords? So how do I know which keywords to pick and aim in order to try to get no.1 ranking? and why those? How many keywords should I use? and where should i put them? since it's not a news website so writing a lot of articles isn't an option. Should I focus on 2 words keywords with around 10.000-30.000 seaches or 2 words keywords + long tail keywords with less traffic like 100 - 5000? Is there a guide for the Keyword Analysis Tool since if i enter "fresh new games" i get a 39% keyword difficulty,is that hard to rank? and I don't know what all those color mean since some of them have higher numbers then others that are found at the top and how can i get beat a website that has has rank 10. So hopefully with your help & by some miracle I will finally be able to build a keyword list. Thank you !
Intermediate & Advanced SEO | | arching0