SEOmoz suddenly reporting duplicate content with no changes???

[Corp]

I am told the crawler has been updated and wanted to know if anyone else is seeing the same thing I am.

SEOmoz reports show many months of no duplicate content problems. As of last week though, I get a little over a thousand pages reported as dupe content errors.

Checking these pages I find there is similar content (hasn't changed) with keywords that are definitely different. Many of these pages rank well in Google, but SEOmoz is calling them out as duplicate content. Is SEOmoz attempting to closely imitate Google's perspective in this matter and therefore telling me that I need to seriously change the similar content?

Anyone else seeing something like this?

[jef2220]

Hi

We see the extreme raise in the duplicate content of our site too. If the sensitivity is adapted, will these graphs come down again?

What is your opinion on how Google sees a webshop with lot's of products and filter options? Our site www.dmlights.com/massive for example can have a lot of filtering but we try to counter this in Webmaster Tools with the URL parameters.

Do you suggest to adapt this for good seo?

Wondering about your opinions. Thanks.

[ChiarynMiranda]

Hey Scott,

Again, we're sorry about the odd jump in duplicate content errors!

We just launched a new crawler and it is being extremely sensitive to duplicate content. As of now we are picking up duplicate pages on your domain via:

https clones of URLs

Some pages have a “/” trailing after the URL and some don’t

We are also ignoring some rel=canonical directives

This is an issue that other users are seeing with their crawls. Our engineers have made some changes to the crawler to scale back the sensitivity to these issues on the crawler and you should be seeing the changes within a week or two.

We're really sorry for the confusion.

Best of Luck,

Chiaryn

[Corp]

Two good suggestions so far, and both I had checked. Thanks KJ Rogers and Ryan Kent.

This is starting to look like it boils down to how much the new SEOmoz crawler sees content in the same way that Google does.

We did not make any site-wide changes and the URLs identified as duplicate in the report are valid URLs that actually hold similar content (keywords and so forth were changed for each version of a slightly different product through an Excel Concatenate construct to build the content). We have actually seen these pages climb in rank over the months since the content was added.

So, like I said, the sudden identification of these as duplicate by the moz crawler is suspicious to me. Not sure it sees things the way Google does.

[RyanKent]

Without examining your site and the pages involved it is not possible for me to share feedback.

Is it possible you made any recent site wide changes? Changes to your header, navigation, footer or sidebar could have pushed you passed a certain threshhold of duplicate content which triggered a flag.

[KJ-Rodgers]

I got the same thing last week. I later found out that mine, using dynamic content on the same page, had speical characters in the url which was taking crawlers to an error page. The error page was showing a list of pages with the url's capitalized. I was able to fix some of them, but it scared the heck out of me.

I had to run a crawl test from SEOMoz to filter out what was going on. Perhaps you have something similar?