Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Matt Cutts talked about this subject.
The summary was, don't try to chase where Google is going. Try to focus on providing web users with the best possible online experience. That is what Google is chasing.
I think that idea is perfectly valid. People will always need a way to obtain information quickly. Google will seek to offer that information. How can we help?
Adding a Google sitemap MAY help SEO. It is something which I would highly recommend for every site.
Sitemaps can offer the following benefits:
the allow search engines to see your full site's content. If you have deep content, poor linked content, or any "island" pages, they may not be found without a site map.
HTML sitemaps help your users find content on your site.
sitemaps can alert search engines to new content on your site. You can have a 1000 page site and the sitemap would let Google know exactly which pages have been updated since your last crawl.
There is tons of sitemap software out there. One example is from Google:
I currently have a Tweet feed. All tweets which match my search criteria are posted to my site in a forum section just for this purpose. By keeping tweets as their own posts, I can update them real-time and easily remove spam.
This process could be expanded upon by having an end-of-day cron job which takes all the individual posts and compiles them into a single daily summary.
The robots.txt file is designed to completely block content. Normally, if your robots.txt file was a factor then your content would not appear in SERPs at all.
It is possible for content to appear in SERPs even though it is blocked by robots.txt if it is linked from other sources. Since this is new content, it is less likely that is the case unless you are immediately sharing links and Google is seeing those links within the time frame you shared.
The first place I would look is your sitemap or whatever tool is used to inform Google that you have new content. When you publish a new blog article, your software should ping Google and inform them there is new content. That is where any investigation should begin. Next step is to check server logs to see how long it takes Google to respond to the alert. If it takes them 12 hours, then there is nothing further you can do about it.
I would be interested in a lot more detail. How many articles how you confirmed as being affected by this issue. Exactly how did you confirm the issue?
As a side note, your robots.txt file is bloated and doesn't adhere to any standards I have seen. How exactly was it created? Did someone go in and make manual modifications to the file?
EGOL,
I agree with you that I would rather have the link from Berkeley based on their reptuation. But your comparison of 100 drunks vs one Pope seems completely unfair.
EzineArticles is the 161st most popular site in the world (Alexa ranking) where Berkeley ranks 1829. EzineArticles also has a lot more (40k+) linking domains when compared to Berkeley. A link from EzineArticles will be seen by more people, while a link from Berkeley will be seen as more credible.
Most webmasters are not in a position to get a link from Berkeley, where they can from EzineArticles.
Again, the term "well" is very relative. You can be a relatively bad site, but if the other sites using that term are worse, then you can rank well.
Can you offer some examples for key words that this site ranks in the top 3 for? If not, then choose their best ranking key word from the first page of SERP.
Isn't a "Follow Link" still a "Follow Link" OR am I completely wasting my time on this devalued website?
Yes, a follow link is still a follow link post Panda.
All links are not equal. A link's value comes from the authority of the source. The domain authority on EzineArticles is an impressive 91. The page ranking, relevance, anchor text and other factors will play an important role in the link's value.
EzineArticles is a source of information. They are linked to be 200k+ domains. They offer exposure for your site. You have an opportunity to reach out to an audience who otherwise might not find you. I would suggest publishing an on-topic article once per quarter would be a positive experience for your site traffic. Google and other search engines will agree and rank the links accordingly.
Hi Erik.
For an example of the canonical tag, right click on this thread's page, choose View Page Source, and look at the top of the document. About the sixth line down you will see a line of code which reads as: . You will also notice the URL of this page matches exactly with that portion of the URL. The code which adds the http://www.seomoz.org prefix is located elsewhere in the code.
I would recommend every page of your site have a canonical tag added. It goes in between the tags of your page's HTML code.
The idea is that there are multiple ways to get to the same page. For example, you can reach this page by typing in "http://www.seomoz.org/q/help-with-canonical-tag" or "http://seomoz.org/q/help-with-canonical-tag" into your browser. Which page is the correct URL? The www version? or the version without the www?
By adding a canonical to your page, you are telling search engines which version you prefer for your site.
The Link Acquisition Assistant is a SEOmoz PRO member tool which provides that information. Try these searches in Google
real estate + "submit site"
real estate + "submit url"
real estate + "intitle:directory"
real estate + "intitle:resources"
real estate + "directory list"
real estate + "inurl:list"
real estate + "intitle:list"
real estate + "inurl:directory"
real estate + "inurl:resources"
real estate + "add url"
real estate + "submit url"
real estate + "suggest url"
real estate + "suggest a url"
real estate + "submit site"
real estate + "submit a site"
real estate + "submit your site"
real estate + "add a url"
real estate + "add a site"
real estate + "add listing"
real estate + "add your listing"
real estate + "submit a listing"
real estate + "submit listing"
real estate + "submit your website"
Google + Bing + Yahoo + Ask = over 98% of search engine traffic in the US. What the other search engines do is irrelevant in my opinion. I am not aware of any search engine which uses meta keywords.
As far as search engines changing their minds, it is entirely possible but highly unlikely. Meta keywords go directly against everything Google has worked towards...showing the same content to the public as to the search engine. The public does not see the meta keywords tag.
This is a very minor detail, and you can go either way. Since you brought the topic up, I will stand by my original recommendation that this unnecessary code be removed from the site.
Solution could therefore be to implement the rel=canonical tag on the intermediary site URL's pointing to the new site URL's
No.
The canonical tag can only be used within a domain.
Since you have determined the URL structure will change, I would recommend finding a logic that would apply to all URLs so you can map from the old site to the new one. Even if each logic piece only applied to 10% of the site, then you can map everything over in a total of 10 redirects.
If your current sites URL is www.oldsite.com/product1 and the new site URL is www.newsite.com/retail/us/stores/items/products/1 you can still use logic to make the conversion IF this same logic can be applied to all, or a worthwhile percentage, of the site.
Alexa is ranking the site's TRAFFIC. That is an important thing to understand.
I can create a bad, one-page site, advertise it, promote it, and Alexa will still show the site's traffic.
That is much different the organic page ranking. The Google Panda update has to do with how sites appear in SERP. The correlation is, that sites which do better in SERP will do better in Alexa. It is possible for a site to do well in Alexa even if they don't do well in Google.
I have a lot of thoughts on this subject. If I was to make a blog entry on this topic, it would span multiple pages or have to be broken down into sub-topics.
I do think there is a correlation between good code and search engine rankings. I do not think there is a correlation between a w3c validated page and search engine rankings. The validator is not current enough, nor flexible enough, to accommodate the real world situations which websites encounter.
Example A: HTML 5 is recognized by all major browsers. W3C validation of HTML 5 is still experimental. A specific example that applies to SEO is the canonical tag. According the the W3C validation site, the canonical tag is not currently valid. Take a snippet of HTML5 code which passes validation, add a canonical meta tag to the header, and the code will no longer pass validation. This is a direct conflict between best practices and validation.
Example B: The world's most popular web page, google.com, does not pass validation. Matt Cutts discussed the topic. In short, they had a choice between providing code which validated, or code which worked. They chose to go with the working code.
Example 
The standard facebook widget code, youtube video code, and other popular code does not pass validation.
Whenever I design a website, I check the code in the validator to look for errors. Initially, I will find numerous errors related to code outside of my control such as social sharing widgets or youtube videos. Once I remove that code, the page often validates. I have researched the issue and it is possible to modify the facebook code or youtube code so that it still functions and passes validation. Doing so requires extra effort, it provides absolutely zero benefit other then saying "hey, I pass validation", and there are often drawbacks such as having to add extra javascript to your site which can otherwise be viewed as unnecessary code.
I would recommend option 1. Remove the meta keyword tag from the site, while following EGOL's advice of including your client in the loop.
Those words offer ZERO value. You and I both know that. It is a best practice of coding, and I would suggest good SEO, to remove unnecessary code. This is especially important when that code appears on every page of the site.
I would be less worried about competitors trying to see what words you are targeting. Any good SEO will know those words have no effect, and they will also be able to look at the page itself, or click a button on a page tool, to understand what key words you are targeting.
Why would someone redirect from www.domain.com to www.domain.com/home.aspx.
There is no logical reason I can think of as to why I would make such a re-direct.
If I was to try and get into the head of the person who did the redirect, I would guess they had a index.html or similar page that showed up when users went to www.domain.com. Rather then remove the index.html page they chose to implement the redirect.
How did you determine they have "good" ranking?
They have a domain authority of 15, which I wouldn't refer to as good ranking.
Hi James. Please allow me to offer some clarifications:
A. server speed- too many 301 redirects is going to put an unnecessary load on the server and reduce speed =BAD
The concern is HOW the redirects are made. If your client has a URL www.oldsite.com, and he moves his entire 10k page site to a new URL, www.newsite.com, and he keeps his URL structure the same at the new site, then the entire re-direct can be done in a single line. In this case there is absolutely no additional load on the server.
Alternatively, if the new site structure is such that hundreds of rules need to be written to properly complete the re-directs, then yes, your speed can be impacted as each request requires the server to iterate through hundreds of lines of code to seek a match.
You shared there would be a change to the URL structure. Your 301 logic would depend on if the change follows a logical pattern to where a regex mapping can be created in a few rules.
B. being penalised- too many 301 redirects can be viewed as aggressive PR sculpting and your 301's can be devalued
C. Avoiding 301 redirect chains- Matt Cutt's interview with Rand in 2010 said 301 chains are not a good thing as a general rule
This refers to chain 301s, a practice that I do not use and would never advise. Go to the original site and ensure each page is properly directed to it's final location.
You can redirect all 100k pages of site A to site B and that would not be considered "too many redirects" nor page sculpting. When you re-direct from site A to site X to site Y to site Z then to site B (the final destination), that would be considered too many re-directs. This could happen even if the re-directs were within the same couple of sites. Even if the re-directs all worked, each hop is a leak in the link juice pipeline.
404's are not necessarily a bad thing
I agree, but a 301 is far superior IF you are keeping the content.
Let's say someone is looking for an article on the 1982 Corvette Stingray. He locates a search result, clicks on it and is taken to the article on your site. He is a happy search engine user, and now a happy visitor on your site. Everyone wins.
Using the same example, the person gets a 404 page on your site. There is an extremely high chance the user will simply return to Google and move on to the next result. Everyone looses.
404s should be used for content when it is highly unlikely a user will ever look for it in a search result OR if you no longer have the content. You would never want to 404 a link when you still have the content and know where it is located.
With all of the above noted, I agree with your plan. The pages with no value, meaning the pages are not searched for or you no longer have the content, can 404.
I block my Terms of Use and Privacy pages with robots.txt.
I use a tool on my site which ads any search terms a user entered in to a search engine to reach the page. "Users found this page by searching for..." and then the box will fill up with search terms if the page is selected from SERPs.
I noticed my Terms and Privacy pages were getting hits. I don't want those pages competing with the rest of my site content in Google. If you allow those pages to be indexed, you run the risk of those pages appearing instead of a more relevant page. This can cause searchers to skip over your result, or click on the result and bounce off your site.
There are outside factors which need to be considered. The answer will not always be the same.
Clearly you prefer the cardealer.com domain. I would investigate that domain before making any decision.
Questions to ask:
What type of site is your friend trying to establish? Car sales?
What type of site currently operates the cardealer.com domain?
If you go with the .net domain, there will clearly be a percentage of your customers who type in the .com URL and wind up on cardealer.com. On one end of the spectrum, you may find that cardealer.com is a forum site discussing the good and bad points of various car dealers. That's not so bad. Some sites are even friendly enough to place a link on their home page such as "Were you looking for cardealer.com? Click here".
On the other hand, the site may be owned by a direct competitor. You will surely lose some customers who visit the wrong site. This issue can be worsened if your competitor's site is better designed then yours or offers lower prices.
You need to ask yourself "What can I do to get 100% of customers who seek me to my site without them landing on the .com site?" Determine which URL will achieve that result, and there is your answer.
I am probably sharing too much of my own personal experiences and expectations. Perhaps I lived a sheltered life 
I had assumed working with someone from England would be like working with someone from another US state. We all speak English, right?
I never anticipated not being able to fully understand everything that was said in a conversation. Nor did I expect such a different system of taxation. These were two areas of culture shock for me.
I am sure there are a wide variety of virtual offices. I have never used one so I cannot speak from personal experience. According to the link I offered, the office can be used to meet clients and hold meetings. In this way, a virtual office is a big step up from a PO Box.
Let's take a tax preparer for example. Perhaps 90%+ of the business is done online. The virtual office can be used for first time client meetups and those who wish to visit in person for whatever reason.
As with most systems, it can be good or bad depending on how it is used.
Example #1 - I decide to operate an online store from my home. I do not wish my home address to be known or used for business. I purchase a virtual office in my area, use that office address for all business dealings which include a local listing for my business.
This seems to be a completely legitimate use of a virtual office. If Google or anyone had a problem with the use of the virtual office address, I can explain my reasoning and offer proof of my home address. I don't believe anyone would have a problem with this idea.
Example #2 - My business is in Texas but I desire a better listing in Los Angeles so I purchase a Beverly Hills virtual office and use that address to gain a local listing in the LA area. I promote my location in local directories and advertise myself as a local business.
Clearly this is a deceptive practice.
Can search engines discover what you are doing? Of course. They have access to what business addresses are being used for this purpose. They may or many not actively investigate this issue, but surely they have the ability to if they felt it was necessary.
A few thoughts I would share from my experiences in England:
England has VAT which is a tax on goods and services. The laws surrounding VAT are quite different then what we are accustomed to for taxation in the US. You may want to gain a high level understanding of how VAT works depending on your role in the SEO process.
"British" English can be quite different from American English. I had serious communication problems when I first began speaking to people from England. One example is "Prices here are much dearer". I guessed that either meant more expensive or cheaper, but had no idea which. You may want to read a quick guide such as: http://www.effingpot.com/slang.shtml
Do not just treat England as if it was another US state. The culture is quite a bit different.
There isn't any other more direct advice I have to offer.
One way to go is the virtual office.
There are businesses which provide general office services. You can pay a small monthly fee and use their address with a modifier such as Suite 110. Many will offer a full range of services including mail forwarding, local phone number, receptionist to answer calls, meeting rooms on request, etc.
As dignan suggested, I wouldn't recommend trying to use this method to fake a presence where the company did not have one. It is quite useful if, as you suggested, an online retailer does not wish to give out their home address.
An example of one such service: http://manhattanvirtualoffice.com/index.html
When you ask about how many links are "too many", we need to understand the context.
Are you referring to internal or external links?
Are you referring to followed links or nofollow links?
What is your goal with these links?
If you simply care about helping users navigate your site, then use as many as you deem helpful. If you are trying to pass link juice, then you want the smallest number of links possible. The smaller the number, the more juice that will flow to each link.
The first question I would ask is, why do you wish to change your current structure? What benefit does it offer you or your visitors? If there isn't a clear, direct benefit I would advise against making a change.
If there is a benefit and you wish to make the change, it shouldn't be a problem at all. You can re-arrange your site as long as you 301 every page properly. There will be a minor loss of link juice, but if this is indeed an improvement for your site, that loss should be offset by your improved site navigation.
I don't believe the directory page you mention is not ranking due to being the only page in a directory. The questions I would ask are how deep is that page within your site? I do not mean how many directories deep. I refer to how many clicks would a user need to make to arrive on that page from your home page? If a user can get there is one click, it should be indexed relatively quickly. If it takes several clicks, then that page is buried within your site and it can take longer to be indexed.
You hit upon many excellent, free methods of getting the word out.
The bottom line is, the more useful or desirable your infographic or tool is, the more site which will use it, the more it will be shared which will mean you get more links.
I wouldn't go with a calculator. I would prefer a tool which has to link back to my site in order to work. A weather widget would be an example. A user enters a zip code, the widget sends the zip code to my server which then returns the information.
Bottom line, either create something to fill a need that no one else has, or create something better then anyone else. That would offer you the usage you desire.
Wordpress and other software either comes designed for SEO, or has plug-ins which offer SEO abilities.
They work by automating some tasks for you. For example, they can automatically canonicalize pages or use friendly URLs. Any SEO options you set up must be thoroughly inspected. While they do make things easier, I would never treat it as a "set it and forget it". Many people get away with that approach, but others do not.
I realize your focus is fixing your .htm issue, but I would like to make one suggestion. Start using friendly URLs which don't show an extension. It should be a win for your site.
Reconsideration requests are for when Google removes a site from it's index for spam or other reasons. There would not be any reason to use one in this instance.
First and foremost, fix the setting. Then verify the noindex tag is removed by visiting a few of your site's pages. Right-click, choose View Page Source and ensure the tag is gone.
Next, as John suggested create a fresh site map and submit it to Google. Log into Google WMT to ensure they have received the updated map. Since this is a new site and you seem anxious to fix this issue, take a moment and check your robots.txt file from Google WMT to ensure there are not any issues.
If your site is large, it may take time for all your pages to re-appear. If your site is small, you will see results faster.
If there is any 1 or 2 articles you feel are critically important to be indexed fast, then I would suggest tweeting a link for the page to help increase it's visibility to Google. I have no knowledge of this working but I even heard of a person tweeting a link to their site map to get indexed faster. I have no idea if it worked but I love the creativity.
Good luck.
There are too many factors involved to offer the analysis you seek while on a general level. Can you offer us your site URL along with the keywords involved?
By looking at your site, the keywords, and the sites who are outranking you we can have a chance to get some ideas as to why these sites are outperforming you.
Some key factors to examine:
the keywords and phrases involved, how competitive are they?
the sites involved, how strong are their root domains?
how strong are your links versus your competitor's links?
how well is your site optimized for SEO vs your competitors?
Is there any form of standardization? I can't imagine 18k pages which were independently developed.
There should be a templating system or some logic which controls code common to all pages. Most pages should share the same header, footer and sidebar, along with standards for things like a canonicalization tag, title and meta description.
If that is not the case, the EGOL's comment should be considered. It is not reasonable to maintain a site which lacks standards.
It works the same way.
In the upper right corner you should see "Adminstrate Comments". Are you logged into the same facebook account used for your site? Are you an admin for the facebook app? As long as the answer to both questions is yes, you will be able to moderate the comments.
What kind of site is it?
With 18k+ pages I will take a guess that it is a forum site. Definitely check with your forum software provider. There should be some form of "page container" which is used as a template for all the site's pages. If you can determine the logic you want to use, such as go with the post title or H1 tag, then you can modify the template according to your logic and take care of your entire site quickly and easily.
Hi Dunamis.
I understand your confusion. Facebook cleverly hides the icon for deleting comments.
To delete a comment you must be logged into Facebook and on your site's page. Go to your WALL, find the comment you wish to delete, then move your mouse to the upper right corner of the comment. The area will appear blank until you hover your mouse in the corner. Then a snowflake like icon will appear.
Clicking that icon will offer you the option of removing the comment, hiding the comment, and banning the user from your page.
I am watching a SEOmoz webinar: Future of Link Building.At 13 minutes into the move an example search is shown where the search term is linkdomain:seomoz.org -seomoz. Rand explains this search will show the results of all pages which mention seomoz.org but do not offer a followed link.
I tried that search for one of my sites: linkdomain:www.terapvp.com -terapvp. It doesn't work at all. It shows a list of sites discussing "tera pvp", none of which have links to me at all.
That leads to two questions:
how does the linkdomain: query work?
how would I ask Google to show a results page of sites with nofollow links and/or citations to my site?
Alan, can you share why it is important to link to the source at the opening of the article as opposed to the end?
There is some discussion on this same topic you can take a look at: http://www.seomoz.org/q/few-high-quality-links-or-a-plethora-of-mediocre-links
Most SEO experts will share the same thought "you want to build your site over time with high quality links". I completely agree but many sites would like a boost to get started. Others have good content but due to heavy competition or other factors desire to perform better in SERP.
It is my understanding having a lot of low quality links can help a site, and cannot harm a site. For those who feel otherwise, I would appreciate the opportunity to further discuss the topic. I would love to see any information from Google or Matt Cutts on the topic.
The term "penalty" often used to describe these links or the site which receives them usually refers to the loss of link juice from the bad link. I am not aware of any negative effects outside of discounting the bad link.
The Pandora effect, or any drop of ranking, is due to the loss of juice from the offending links, not a penalty from Google. The site still benefited. For a time, they had higher rankings and more exposure to the public. During that period the site could have earned additional sales or picked up readers who otherwise may have not seen the site. Those additional customers and readers can directly lead to the site ranking higher then it would of if it never received the "bad" link.
The worst thing you can do is nothing.
Above is 5 examples of URLs which COULD all lead to the same page. There are numerous other possibilities as well. If you don't let Google know which version of the page is correct, then you will suffer the consequences of duplicate content.
What happens is Google doesn't know which page is correct. They will pick one of the non-www versions because that is what your Google WMT is set up to do. Meanwhile other versions of the pages are being used.
You are sending your link juice to a page, but it is a complete waste as it is not being considered by Google for SERP. You MUST resolve this issue if you care about SEO at all.
You are welcome to do so. Go to Google WMT, change your current option to the www, then adjust your .htaccess file as Steven suggested.
Also, canonicalize your pages to help ensure this issue can't happen again. Your .htaccess changes will work as long as the file is there, but things happen so it's better to be covered.
You need to make a decision. Do you want your site address to be seen with or without the www?
Try to assess which version of your URL would require the least number of re-directs. You mentioned the links you built mostly include the www. Take a look at all of your links. You may have a higher number of organic links without the www. Evaluate all the links, then make a decision.
Once you make a decision, stick with it. Canonicalize all your pages with the correct version of the URL. Search your site for all internal links and standardize them.
While you are on this project standardize whether you use a "/" on the end of your url as well. www.mysite.com is not the same as www.mysite.com/. I make this suggestion because if you will go through the painful process of standardizing your site for the www issue, you should resolve all issues at once.
Does this type of link building work well?
Absolutely
** Is 15 links a month from PR 2-5 blogs (writing on my subject matter) enough to help out?**
Yes.
Paid blogging exists because it works. With that said, Google is getting better at adjusting for low quality content and the blog farms. Usually a paid blogger posts in perfect English, with the right post title, the right keyword usage, etc. They are trained to write for SEO. The problem is the actual quality of the content they write is usually low. They don't know nor understand the topics they are writing about.
On the other hand, many experts will write about topics, but they are not always proficient with English. Also they don't understand SEO mechanics so while they share great information, it is often not presented optimally from a SEO perspective. So when Google balances things out, a paid blog article can rank higher then an article written from an expert on the topic. There are many factors involved of course, but these are some of them.
I am sure there will be others who will disagree with the perspective I am sharing. Of course all sites should strive to create great content. The internet has too many paid bloggers and others who just fill the internet with fluff making it harder for users to get the real content they need. Google still rewards sites for using paid bloggers with higher rankings. As long as that is the case, they will still be used by those without the resources to create the content on their own.
With respect to the amount of links, it is all relative. How many links does your site have presently? Are these articles linking to your home page? Or are they linking to deeper pages? Are the links from 15 different blogs with different root domains? Are they blogging from a general wordpress domain or from a topic specific blog? There are many factors used to evaluate the value of a link. Without seeing your site I would suggest that 15 links/month is enough to help most small to medium sites quite nicely.
Google adds pages based on following links. If your site offers a link to the page, or even if any other site offers a link, then it can be crawled and potentially indexed.
If you do not wish the page to be listed you have a few options. You can canonicalize the page to point to the non-secure version. For example let's assume you have the page http://www.mysite.com/info.html. Let us also assume you have a secure version of the page. On both pages I would recommend adding the following code:
That code tells Google the http:// version of the page is the preferred version, and the other page is a duplication.
If you do not have a duplicate page issue and you simply don't wish a page to be listed in SERP, then add the "noindex" tag to the page.
What is the easiest way to nail down why Google is using the https version?
Google will list any pages it finds. That's their role. Unless you specifically provide a reason for them not to index a page, such as the canonical tag, a noindex tag, or block them with robots.txt, they will list a page. They have no way of knowing you wanted the http:// version listed unless you tell them.
To fix the issue, you can add the canonical and the next time they check the page they will update their data. It may take a bit of time depending on the size of your site.
Can we just use the top style dropdown like www.adventurefinder.com has?
Yes.
If you are ever unsure, any easy test is to right-click on the page and View Page Source. Search for the text used in the navigation menu. If you can read the text in the page's source code, then Google can definitely read it as well.
Even if you can't read it, Google MAY be able to still pull the data. It's one of those things where if you can see it then you are fine, but if you can't see it you may still be fine as well.
I can share a personal experience that might help clarify how the duplicate site issue works.
My site contains a sitemap which is automatically updated each day. When the new sitemap is created, Google and Bing are pinged. I also have a backup job scheduled to run daily. It is a perfectly standardized setup.
I wanted to do some testing with my site, so I restored my most recent backup to www.mydomain.com/test. The /test site was an exact duplicate of my domain. My lack of forethought was quite costly.
The /test site generated a sitemap, pinged the search engines, and was listed. In retrospect, I should have turned off the sitemap job and blocked the /test site in robots.txt. Worse, I didn't catch my error for over a month. While doing a search I noticed the result took me to the test site. After some troubleshooting I realized the error.
Bottom line, anyone can run two sites with identical information. The problem is that both sites will get listed. So if you have a 100 page site which is duplicated, then Site A might have pages 1 - 50 listed, and site B might have pages 51 - 100 listed. The problem is your domain authority is split between the two sites.
Ultimately the problem is fixed by 301'ing one site to the other. When that happens you will lose some link juice due to the 301.
Your competitor can rank well on both sites. But the same page wont rank well on both sites. Only one of the duplicated pages will appear in the SERP.
Google Analytics offers that information. If you are not already signed up, you should do such.