Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for list Pro's & Con's of removing Folder from URL?
Hi We have a sub-folder ("/shop-by-department/") which is pretty much useless on our site and I'm looking to remove it. But the team want a list of the Pro's & Con's in doing so. So for example I'll be changing www.example.ie/shop-by-department/furniture/beds/product-a to www.example.ie/furniture/beds/product-a I know there will be an intial hit as Google adjusts to the change but think it's definitely the way to go. I was lookng for a complete list of the Pro's & Con's to send onto the team. It'll be going to the traditional marketing (print, radio, etc.) too so can ve top-level points too. Hope you can help! Thanks
Web Design | | Frankie-BTDublin0 -
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Fixing Render Blocking Javascript and CSS in the Above-the-fold content
We don't have a responsive design site yet, and our mobile site is built through Dudamobile. I know it's not the best, but I'm trying to do whatever we can until we get around to redesigning it. Is there anything I can do about the following Page Speed Insight errors or are they just a function of using Dudamobile? Eliminate render-blocking JavaScript and CSS in above-the-fold content Your page has 3 blocking script resources and 5 blocking CSS resources. This causes a delay in rendering your page.None of the above-the-fold content on your page could be rendered without waiting for the following resources to load. Try to defer or asynchronously load blocking resources, or inline the critical portions of those resources directly in the HTML.Remove render-blocking JavaScript: http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js http://mobile.dudamobile.com/…ckage.min.js?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…pts/blogs.js?version=2015-04-02T13:36:04 Optimize CSS Delivery of the following: http://fonts.googleapis.com/…:400|Great+Vibes|Signika:400,300,600,700 http://mobile.dudamobile.com/…ont-pack.css?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…kage.min.css?version=2015-04-02T13:36:04 http://irp-cdn.multiscreensite.com/kempruge/files/kempruge_0.min.css?v=6 http://irp-cdn.multiscreensite.com/…mpruge/files/kempruge_home_0.min.css?v=6 Thanks for any tips, Ruben
Web Design | | KempRugeLawGroup0 -
Google pagespeed / lazy image load
Hi, we are using the apache module of google pagespeed. It works really great, helps a lot. But today I've asked me one question: Does the "lazy load" feature for images harm the ranking? The module reworks the page to load the images only if the are visible at the screen. Is this behavior also triggered by the google bot? Or are the images invisible for google? Any expirience about that? Best wishes, Georg.
Web Design | | GeorgFranz0 -
Is it bad to have /index.php at the end of a uri?
Is it bad for SEO if traffic is directed to "http://www.example.com/someuri/index.php" instead of "http://www.example.com/someuri/" and would it be works setting up a redirect rule at htaccess level?
Web Design | | NoisyLittleMonkey1 -
Comparing the site structure/design of my live site to my new design
Hi SEOmoz team, for the last few months I've been working on a new design for my website, the old, live design can be viewed at http://www.concerthotels.com - it is primarily focused on helping users find hotels close to concert venues throughout North America. The old structure was built in such a way that each concert venue had a number of different pages associated with it (all connected via tabs) - a page with information about the venue, a page with nearby hotels to the venue, a page of upcoming events, a page of venue reviews. An example of these pages can be seen at: http://www.concerthotels.com/venue/madison-square-garden/304484 http://www.concerthotels.com/venue-hotels/madison-square-garden-hotels/304484 http://www.concerthotels.com/venue-events/madison-square-garden-events/304484 http://www.concerthotels.com/venue-reviews/madison-square-garden-reviews/304484 The /venue-hotels/ pages are the most important pages on my website - and there is one of these pages for each concert venue - they are the landing pages for about 90% of the traffic on the website. I decided that having four pages for each venue was probably a poor design, since many of the pages ended up having little or no useful, unique content. So my new design attempts to bring a lot of the venue information together into fewer pages. My new website redesign is temporarily situated at: (not currently launched to the public) http://www.concerthotels.com/frontend The equivalent pages for Madison Square Garden are now: http://www.concerthotels.com/frontend/venue/madison-square-garden/304484 (the page above contains venue information, events and reviews) and http://www.concerthotels.com/frontend/venue-hotels/madison-square-garden-hotels/304484 I would really appreciate any feedback from you guys, based on what you think of the new site design compared to the old design from an SEO point of view. Of course, any feedback on site speed, easy of use etc compared to the old design would also be greatly appreciated. 🙂 My main fear is that when I launch the new design (the new URLs will be identical to the old ones), Google will take a dislike to it - I currently receive a large percentage of my traffic through Google organic search, so I don't want to launch a design that might damage that traffic. My gut instinct tells me that Google should prefer the new design - vastly reduced number of pages, each page now contains more unique content, and it's very much designed for users, so I'm hoping bounce rate, conversion etc will improve too. But my gut has been wrong in the past! 🙂 But I'd love to hear your thoughts, and thanks in advance for any feedback, Cheers Mike
Web Design | | mjk260 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0 -
Link Pages/Directory
Hello, What is best practise for dealing with alot of links. I was thinking of breaking them download to alphabet pages i.e. all A on one page etc... BUT should I then make the links clickable on this list OR that they load to a sub company page which has a clickable link to there website.
Web Design | | JohnW-UK0