Crawler Stats
-
Hello,
On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com
Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www?
This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not?
Thanks for any help in advance.
-
This article about canonicalization might help:
-
This article about canonicalization might help:
-
It make take a full crawl or two before it fixes itself. Keep me posted.
-
Thanks for the reply. I am going to change it to your method now and see what happens.
Thanks!
-
I'm not sure by looking at it if it's wrong since I'm not really an Apache coder. But I'm just guessing it's probably wrong since the bots are still detecting both versions of your site. I have it the way I showed you in my .htaccess file and I don't have any problems with bots detecting multiple versions. Maybe try it this way. You can always make a backup of your current file and switch back if it doesn't make a difference.
-
RewriteEngine on rewritecond %{http_host} ^domainhere.com [nc] rewriterule ^(.*)$ http://www.domainhere.com/$1 [r=301,nc]
This is what I currently have in my .htaccess file - is this wrong, and should I change it to the method that you posted?
Any information is appreciated. Thanks!
-
Yes, it matters. It could mean that Google is detecting it that way too, and may view them as two different sites with duplicate content. This could be hurting your rankings. Do you have access to your .htaccess file? You should add this code:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www.domainhere.com$
RewriteRule (.*) http://www.domainhere.com/$1 [R=301,L]This will redirect the non-www version to the www version and tell the crawlers this is how you'd prefer your site to display.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
Hide messenger for crawlers
At Magnet.me we are using Intercom to communicate with our users. This means that we are actively adding javascript code which will load the Intercom javascript on each page, and render the button afterwards. However, this button has no value for crawlers, and slows the page down as the javascript is big and fairly slow. Therefore I considered to ship some code which disables this button, such that performance would improve. To give a ball pack estimate, the buttons javascript is around 3x bigger than the actual entire react application... Unfortunately this would result in giving users and crawlers slightly different content on the page. I'm unsure about the possible SEO impact: Would Google mark the page as faster due to less resources to load? Or would it penalize the page for showing slightly different content to users and search engines?
Technical SEO | | rogier_slag0 -
Critical crawler errors...4xx
Hey fam, I ran the Critical Crawler Issues and found 9 pages with critical crawler issues. I'm running a wordpress site and looked in the dashboard for Pages and Posts but the links aren't in the dashboard. Can you help fix? Thank you!
Technical SEO | | Myflgreen0 -
Will a google map loaded "on scroll" be ignored by the crawler?
One of my pages has two Google maps on it. This leads to a fairly high keyword density for words like "data", "map data" etc. Since one of the maps is basically at the bottom of the page I thought of loading it "on scroll" as soon as its container becomes visible (before loading the map div should be empty). Will the map then still be craweld by google (can they execute the JS in a way that the map is loaded anyways?) or would this help to reduce the keywords introduced by the maps?
Technical SEO | | ddspg0 -
Block Baidu crawler?
Hello! One of our websites receives a large amount of traffic from the Baidu crawler. We do not have any Chinese content or do any business with China since our market is Uk. Is it a good idea to block the Baidu crawler in the robots.txt or could it have any adverse effects on SEO of our site? What do you suggest?
Technical SEO | | AJPro0 -
How does your crawler treat ajax links?
Hello! It looks like the seomoz crawler (and google) follows ajax links. Is this normal behavior? We have implemented the canonical element and that seems to resolve most of the duplicate content issues. Anything else we can do? Example: Krom
Technical SEO | | AJPro0 -
Impact of "restricted by robots" crawler error in WT
I have been wondering about this for a while now with regards to several of my sites. I am getting a list of pages that I have blocked in the robots.txt file. If I restrict Google from crawling them, then how can they consider their existence an error? In one case, I have even removed the urls from the index. And do you have any idea of the negative impact associated with these errors. And how do you suggest I remedy the situation. Thanks for the help
Technical SEO | | phogan0 -
CSS Hiding Text - Does this matter to search engine crawlers
Hello, I'm working on a site and a developer is using CSS to mask crawlable links below. Then, java, advanced search links go on top of this. So, if you disable Java, but have CSS enabled, you don't have a lefthand nav. With both CSS and Java disabled you have a fully crawlable website. Is this a red flag? I understand a user without java would have a problem since most people don't disable CSS. But, is this a problem for search crawlers? Thanks!
Technical SEO | | Blenny0