Crawler Stats
-
Hello,
On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com
Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www?
This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not?
Thanks for any help in advance.
-
This article about canonicalization might help:
-
This article about canonicalization might help:
-
It make take a full crawl or two before it fixes itself. Keep me posted.
-
Thanks for the reply. I am going to change it to your method now and see what happens.
Thanks!
-
I'm not sure by looking at it if it's wrong since I'm not really an Apache coder. But I'm just guessing it's probably wrong since the bots are still detecting both versions of your site. I have it the way I showed you in my .htaccess file and I don't have any problems with bots detecting multiple versions. Maybe try it this way. You can always make a backup of your current file and switch back if it doesn't make a difference.
-
RewriteEngine on rewritecond %{http_host} ^domainhere.com [nc] rewriterule ^(.*)$ http://www.domainhere.com/$1 [r=301,nc]
This is what I currently have in my .htaccess file - is this wrong, and should I change it to the method that you posted?
Any information is appreciated. Thanks!
-
Yes, it matters. It could mean that Google is detecting it that way too, and may view them as two different sites with duplicate content. This could be hurting your rankings. Do you have access to your .htaccess file? You should add this code:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www.domainhere.com$
RewriteRule (.*) http://www.domainhere.com/$1 [R=301,L]This will redirect the non-www version to the www version and tell the crawlers this is how you'd prefer your site to display.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
2 sets of stats for same site
Somehow on OSE I managed to get two different sets of results appear for my page. The column on the left (PA 34) is for mysite.com/ and the second column is for www.mysite.com/ .Note that these are the same site. Why do i have two different sets of results ?(note some things are the same such as google +1 & FB likes)Im concerned ive done something wrong and could have a bigger beast with both sets of results merged together. Any help much appreciated. Chris QFNeGh7
Technical SEO | | cjkimber0 -
Traffic stats disaster
Hi, We have recently moved a few old websites to an existing one and redesigned the existing website too. The visits in Analytics have fallen though the roof in the last few days. I would expect the traffic to have gone up because visitors from the old websites should now be on the redesigned one. Not sure what the problem is entirely. GA code is on the site but not before the closing head where I asked for it to be put. It's just before closing body tag. Does it make a difference? (It's a CMS site). GA code must be working because visits are being recorded (tens rather than the expected hundreds). Redirects have not been done properly - not sure what IT have done there. Would the lack of proper redirects be the sole reason for the drop? I am still hassling for 301's to be sorted. Thanks all
Technical SEO | | Houses0 -
SEOMoz Crawler vs Googlebot Question
I read somewhere that SEOMoz’s crawler marks a page in its Crawl Diagnostics as duplicate content if it doesn’t have more than 5% unique content.(I can’t find that statistic anywhere on SEOMoz to confirm though). We are an eCommerce site, so many of our pages share the same sidebar, header, and footer links. The pages flagged by SEOMoz as duplicates have these same links, but they have unique URLs and category names. Because they’re not actual duplicates of each other, canonical tags aren’t the answer. Also because inventory might automatically come back in stock, we can’t use 301 redirects on these “duplicate” pages. It seems like it’s the sidebar, header, and footer links that are what’s causing these pages to be flagged as duplicates. Does the SEOMoz crawler mimic the way Googlebot works? Also, is Googlebot smart enough not to count the sidebar and header/footer links when looking for duplicate content?
Technical SEO | | ElDude0 -
Duplicate Content Issue: Google/Moz Crawler recognize Chinese?
Hi! I am using Wordpress multisite and my Chinese version of the website is in www.mysite.com/cn Problem: I keep getting duplicate content errors within www.mysite.com/cn (NOT between www.mysite.com and www.mysite.com/cn) I have downloaded and checked the SEOmoz report and duplicate_page_content list in CSV file. I have no idea why it says they have the same content., they have nothing in common in content . www.mysite.com is the English version of the website,and the structure is the same for www.mysite.com/cn *I don't have any duplicate content issues within www.mysite.com Question: Does google Crawler properly recognizes chinese content??
Technical SEO | | joony20080 -
CSS Hiding Text - Does this matter to search engine crawlers
Hello, I'm working on a site and a developer is using CSS to mask crawlable links below. Then, java, advanced search links go on top of this. So, if you disable Java, but have CSS enabled, you don't have a lefthand nav. With both CSS and Java disabled you have a fully crawlable website. Is this a red flag? I understand a user without java would have a problem since most people don't disable CSS. But, is this a problem for search crawlers? Thanks!
Technical SEO | | Blenny0 -
Page not Accesible for crawler in on-page report
Hi All, We started using SEOMoz this week and ran into an issue regarding the crawler access in the on-page report module. The attached screen shot shows that the HTTP status is 200 but SEOMoz still says that the page is not accessible for crawlers. What could this be? Page in question
Technical SEO | | TiasNimbas
http://www.tiasnimbas.edu/Executive_MBA/pgeId=307 Regards, Coen SEOMoz.png0 -
Slash at end of URL causing Google crawler problems
Hello, We are having some problems with a few of our pages being crawled by Google and it looks like the slash at the end of the URL is causing the problem. Would appreciate any pointers on this. We have a redirect in place that redirects the "no slash" URL to the "slash" URL for all pages. The obvious solution would be to try turning this off, however, we're unable to figure our where this redirect is coming from. There doesn't appear to be an instruction in our .htaccess file doing this, and we've also tried using "DirectorySlash Off" in the .htaccess file, but that doesn't work either. (if it makes a difference it is a 302 redirect doing this, not a 301) If we can't get the above to work, then the other solution would be to somehow reconfigure the page so that it is recognizable with the slash at the end by Google. However, we're not sure how this would be done. I think the quickest solution would be to turn off the "add slash" redirect. Any ideas on where this command might be hiding, and how to turn it off would be greatly appreciated. Or any tips from people who have had similar crawl problems with google and any workarounds would be great! Thanks!
Technical SEO | | onetwentysix0 -
Mobile SEO or Block Crawlers?
We're in the process of launching mobile versions of many of our brand sites and our ecommerce site and one of our partners suggested that we should block crawlers on the mobile view so it doesn't compete for the same keywords as the standard site (We will be automatically redirecting mobile handsets to the mobile site). Does this advice make sense? It seems counterintuitive to me.
Technical SEO | | BruceMillard0