Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will this 301 redirects help me?
Hello, recently, I found out about all the SEO advantages from 301 redirects. I had 3 websites that are now expired, their topic was Counter Strike 1.6 servers. All of these websites were registered 9 years ago and have few good backlinks (from website with 1%-3% spam score and DA 30+). Now I have one website that is not only about Counter Strike 1.6 but also many other Steam shooter games. If I revive these 3 old domains and 301 redirect them to my new one, will it help me with SEO and increase my ranking on Google?
Intermediate & Advanced SEO | | Bonito19930 -
Please... Help me convince my boss that Keyword Density is not-important / damaging.
If you can, please provide any and all talking points that I can use in this argument. It seems that no matter what I show him, including Matt Cutts' video debunking Keyword Density back in 2011, it doesn't seem to stick. He is fully, 100% convinced that keyword density is hugely important and we need to focus our time and energy on it. Any sources you might have to help me show him that this is a myth would be hugely appreciated. Thank you.
Intermediate & Advanced SEO | | TaylorRHawkins2 -
Need Help - Lost 75% Of Traffic Since May 2018
Sorry to go in-depth here, but want to give all available information. We went live late April 2018 with our two websites in Shopify (moved from Magento, same admin, different storeviews...which we find later to cause some issues). Both of these websites sell close to the same products (we purchased a competitor about 5 years ago, which is why we have two). The nice thing is that they do almost identical amounts in sales. They have done very well for years, especially in the last two years. Well, the core algo update around May 22nd-24th 2018 happened and wiped out about 65% of our Google traffic for one website (MySupplementStore.com). And this latest update, wiped out another 20%. I couldn't figure out why this would have happened, because we were very cautious about keeping things separate, unique descriptions etc. So I did some digging and this is what I found: The reviews we migrated over from Magento somehow were combined and added to both websites. This is something I didn't notice. I had this resolved a month ago so that each site's reviews are now only on that website. Our blog section was duplicated across both websites during the migration. Again, something I didn't notice, as we have close to over 1,000 blog posts per site. This was resolved two weeks ago. As I was looking more, I found that the last 6 months, a person working for us (for 3 years), started writing descriptions and pasting them on both websites, instead of making them unique to each website. I trusted her for years, but I think she just got lazy. She quit about a month before the migration as well. We are currently working on this, but its been taking awhile because we have over 5,000 products on each site and have no idea which ones are duplicates. I did also notice: Site very slow when checking site speed tools. Working on that this week. When I take snippets of text or do searches, many times it shows up in omitted results. No messages in Google Webmaster Tools So the question is... Do you think it is the duplicate content issues that caused the drop? Our other site is Best Price Nutrition, which didn't see a big drop at all during that update. If not, any other ideas why?
Intermediate & Advanced SEO | | vetofunk0 -
Have a Robots.txt Issue
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one. Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
Intermediate & Advanced SEO | | primemediaconsultants0 -
Meta robots or robot.txt file?
Hi Mozzers! For parametric URL's would you recommend meta robot or robot.txt file?
Intermediate & Advanced SEO | | eLab_London
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs. And what's the real difference between the two? Thanks again! Kay0 -
Should all pages on a site be included in either your sitemap or robots.txt?
I don't have any specific scenario here but just curious as I come across sites fairly often that have, for example, 20,000 pages but only 1,000 in their sitemap. If they only think 1,000 of their URL's are ones that they want included in their sitemap and indexed, should the others be excluded using robots.txt or a page level exclusion? Is there a point to having pages that are included in neither and leaving it up to Google to decide?
Intermediate & Advanced SEO | | RossFruin1 -
If i disallow unfriendly URL via robots.txt, will its friendly counterpart still be indexed?
Our not-so-lovely CMS loves to render pages regardless of the URL structure, just as long as the page name itself is correct. For example, it will render the following as the same page: example.com/123.html example.com/dumb/123.html example.com/really/dumb/duplicative/URL/123.html To help combat this, we are creating mod rewrites with friendly urls, so all of the above would simply render as example.com/123 I understand robots.txt respects the wildcard (*), so I was considering adding this to our robots.txt: Disallow: */123.html If I move forward, will this block all of the potential permutations of the directories preceding 123.html yet not block our friendly example.com/123? Oh, and yes, we do use the canonical tag religiously - we're just mucking with the robots.txt as an added safety net.
Intermediate & Advanced SEO | | mrwestern0 -
Can I use a "no index, follow" command in a robot.txt file for a certain parameter on a domain?
I have a site that produces thousands of pages via file uploads. These pages are then linked to by users for others to download what they have uploaded. Naturally, the client has blocked the parameter which precedes these pages in an attempt to keep them from being indexed. What they did not consider, was they these pages are attracting hundreds of thousands of links that are not passing any authority to the main domain because they're being blocked in robots.txt Can I allow google to follow, but NOT index these pages via a robots.txt file --- or would this have to be done on a page by page basis?
Intermediate & Advanced SEO | | PapaRelevance0