Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will There Be Much Impact When Moving Site To New Root Folder?
Hi, ok so I have a pretty big site that is located on my sever /root/current-folder/. I want to rebuild the site completely as it's using software that is out of date and not our main focus anymore (OpenCart). We want to move to a Wordpress platform, but want to have as little impact on the SEO as possible. Our current strategy is: List all URLs/Titles/Meta indexed with Google on current site Create new folder on the server /root/new-folder/ My question is... if I move to a new folder on the server (same TLD) and then re-route the TLD to go to this new folder, will there be more of an impact on SEO that if I start a fresh in the current folder? Thanks
Web Design | | Easigrass0 -
Do search engines see copy/keywords when it appears only at the bottom of a page?
My client is looking to improve their SEO, and to date I've written meta data and made some initial recommendations. Thing is, on some of their pages, the body copy appears at the bottom of the page, past links and big, splashy images. My question is, will search engines even see that copy to crawl it for keywords? Thanks!
Web Design | | MarcieHill0 -
A/B Testing.. Are you doing? how is it been? What do you think would be the best path for who is starting now?
Hey Mozers, One of my 2014 resolutions is to start doing A/B Testing, so far I have been following "best practices" and "common sense" when comes to website design, but I would like to go above and beyond. I was hoping a could get a few tips some of you that are already doing A/B testing. How is it been? Do you see a great ROI? What do you think would be best path for who is starting now? Any book or links you would recommend? Thanks
Web Design | | Felip30 -
Nesting <a>tag for rel="nofollow"</a>
I just wanted to quickly run this past someone. I have some footer links I want set to nofollow: <map <span="" class="webkit-html-tag">name</map>="Map"><area <span="" class="webkit-html-tag">shape="rect" coords="10,39,73,101" href="URL" target="_blank" alt="some text"> should this be <map <span class="webkit-html-tag">name</map <span>="Map"><area <span class="webkit-html-tag">shape</area <span>="rect" coords="10,39,73,101 <a <span class="webkit-html-tag">href="URL" rel="nofollow" target="_blank" alt="some text"></a <span> Want to check before I advise. If this is not the way how can I fix?
Web Design | | MickEdwards0 -
Blog vs. News/Editorial Layout?
We're in the coupon blogging space & saw that one of the larger coupon sites move away from the more traditional blog layout: http://thekrazycouponlady.com they now have more of an editorial type layout. Here is another site which is more similar to our layout: http://hip2save.com. So here are my questions: Which layout type do you feel better serves their visitors & why? How does the affect the SEO of the site? How does it affect the advertising revenues? Which layout do you prefer? Is there strategy in this move for the coupon blog, or is this just a preference on how they now display their content? We're making some updates to our design soon & I wanted to get some feedback on the overall direction we take.
Web Design | | seointern0 -
Making a third-party hosted blog look like a folder on the main domain
I have a client that has a "completely pristine" Microsoft.net web environment that is unwilling to put a wordpress installation on their server. Their management team wants a wordpress blog for the marketing department. Is there a means where we can host the wordpress blog with a regular hosting company but yet have it appear as part of the main site e.g., mainsite.com/blog vs. having to put it in a subdomain (blog.mainsite.com) and lose all the SEO benefits of the blog content?
Web Design | | jtroia0 -
What is the current thought how/if cufon affects keywords?
What is the final opinion about how (and if) cufon affects keywords in H1 tags?
Web Design | | Court_LOQUA0