What is best practice to eliminate my IP addr content from showing in SERPs?
-
Our eCommerce platform provider has our site load balanced in a few data centers. Our site has two of our own exclusive IP addresses associated with it (one in each data center).
Problem is Google is showing our IP addresses in the SERPs with what I would assume is bad duplicate content (our own at that).
I brought this to the attention of our provider and they say they must keep the IP addresses open to allow their site monitoring software to work. Their solution was to add robots.txt files for both IP addresses with site wide/root disallows.
As a side note, we just added canonical tags so the pages indexed within the IP addresses ultimately show the correct URL (non IP address) via the canonical.
So here are my questions.
-
Is there a better way?
-
If not, is there anything else we need to do get Google to drop the several hundred thousand indexed pages at the IP address level? Or do we sit back and wait now?
-
-
I would allow Google to crawl those pages for a little while longer just to ensure that they see the rel canonical tags. Then once you feel that they have recrawled the IP address pages you can disallow them again if you want, thought that isn't entirely necessary if you have the rel canonical tag set up properly.
Another option would be to 301 redirect the IP version of the page to the corresponding www. version.
If they still don't drop from the index you can use the URL Removal Tool in GWT, but you will have to set up a GWT account for each of the IP domains.
-
Thanks. Any suggestions on how to get Google to drop these pages (make them inactive)?
-
Hi,
Since doing the disallow on the IP address sites, they are no longer getting crawled.
** The disavow list won't stop google crawl those domain / pages. Google will just treat those links as no follow - so they won't pass Page Rank.
You will still see those in Web master tools, the links will still be active.
-
Sorry - I just thought of something that could pose a problem and was hoping to get your advice.
Since doing the disallow on the IP address sites, they are no longer getting crawled. Does that mean that the canonical tags within those IP address sites wont be able to do their work?
Or
Will the canonicals picked up from the proper domain help the search engines know they should consolidate the indexed pages from the now disallowed IP addresses?
I am seeing that the IP addresses are no longer being crawled, and the pages in their indexes about the same (not going down).
Thoughts?
-
Sorry - I just thought of something that could pose a problem and was hoping to get your advice.
Since doing the disallow on the IP address sites, they are no longer getting crawled. Does that mean that the canonical tags within those IP address sites wont be able to do their work?
Or
Will the canonicals picked up from the proper domain help the search engines know they should consolidate the indexed pages from the now disallowed IP addresses?
I am seeing that the IP addresses are no longer being crawled, and the pages in their indexes about the same (not going down).
Thoughts?
-
Thanks!
-
Thanks. We are getting large daily crawls (nearly 100k a day) so fingers crossed this will sort it out soon.
-
Hi,
The canonical solution should be enough however I would still build some xml sitemaps and submit those via Web master Tools to speed the process. You can also build some html sitemaps with a clear structure and add those in the footer - again, to speed up the proces a little bit.
If you split the content into multiple xml sitemaps you can also track the crawling process.
You should also check your crawling speed in Web Master Tools to see how many pages in avarage the google bot is hitting each day - based on those numbers you can run some prediction on how long it will take more or less for google to re crawl your pages.
If your numbers is "bad" you will need to improve it some how to help with process - it can do wonders...
Hope it helps.
-
The canonical solution you have implemented is perfect. If you have decent authority and get deep crawls every couple days, you should be fine and pages from your IP should start to disappear shortly.
I would not worry about it anymore. You are on the right track. Sit back, relax and enjoy your flight
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I need help on how best to do a complicated site migration. Replacing certain pages with all new content and tools, and keeping the same URL's. The rest just need to disappear safely. Somehow.
I'm completely rebranding a website but keeping the same domain. All content will be replaced and it will use a different theme and mostly new plugins. I've been building the new site as a different site in Dev mode on WPEngine. This means it currently has a made-up domain that needs to replace the current site. I know I need to somehow redirect the content from the old version of the site. But I'm never going to use that content again. (I could transfer it to be a Dev site for the current domain and automatically replace it with the click of a button - just as another option.) What's the best way to replace blahblah.com with a completely new blahblah.com if I'm not using any of the old content? There are only about 4 URL'st, such as blahblah.com/contact hat will remain the same - with all content replaced. There are about 100 URL's that will no longer be in use or have any part of them ever used again. Can this be done safely?
Intermediate & Advanced SEO | | brickbatmove1 -
Google for Jobs best practice for Job Boards?
I head up SEO for a niche job board. We disallowed our job ad pages (/job/) in the robots.txt as this is user-generated content and really eating up our crawl budget, causing penalties etc. Now Google for Jobs has hit the UK (our strongest region for traffic), I'm torn about what to do next. Our jobs will only show in GfJ if we remove the jobs pages from the robots.txt and apply the directed structured data to every single jobs page and monitor this constantly. I will also have to constantly invest in our website developers no indexing / canonicalizing new job pages and paginations. Is GfJ worth it? I have spoken to one other job board who has seen more brand awareness from appearing in GfJ but almost no traffic / application increase. But are we missing a trick here? Any advice would be greatly appreciated.
Intermediate & Advanced SEO | | gracekimberley11 -
Best SEO Strategy
Hi fellow Mozers: I have a question about strategy. I have a client who is a major real estate developer in our region. They build and sell condominiums and also built and manage several major rental apartments. All rental properties have their own websites and there is also a corporate website, which has been around for many years and has decent domain authority (+/- 40). The original intent of the corporate website was to communicate central brand positioning points, attract investors and offer individual profiles of all major properties. My client is interested in developing an organic search strategy which will reach consumers looking to rent apartments. Typical search strings would include the family whose core string would be 'apartments in Baltimore.' (Currently, the client runs PPC for each one of their properties. This is expensive and highly competitive.) In doing research, we've found that there are two local competitors who are able to break on to Page 1 and appear beside the National 'apartment search guides' who dominate the Page 1 SERPS (like apartments.com). The two local competitors have websites of either the same or lower authority than our client's; one has a better link profile, the other is comparable. Here's our problem: our local competitors only build and manage apartments. So, then, the home pages and all the content of their sites ONLY talk about apartment rental related information. Our client's apartment business is actually larger in scope than either local competitor but is only one of their major real estate verticals. So my question is this: if we want to build out a bunch of content which will rank competitively with our local competition, are we better off creating a new area of the corporate site, creating targeted content and resources appropriate for apartment seekers OR would we be better off creating an entirely new site, just devoted to the same? I'm wondering if a new section will ever rank well against competitors whose root domains actually feature content which is only rental related? Likewise, I'm wondering whether we'd be giving up too much, in terms of authority, by creating an entirely new site? I've also only found examples in the industry where an entirely new site was created, so it makes me question the strategy of building out a rental-specific section of a site which also contains information about their condo business. For instance, the Related Companies are a huge builder in the East; they have a corporate site and a site called https//relatedrentals.com . Any feedback would be greatly appreciated!
Intermediate & Advanced SEO | | Daaveey0 -
SERP cannibalization
Hi Moz Community, Recently I've been seeing multiple pages from my eCommerce site pop up in the SERPS for a couple of queries. Usually I would count this as a good thing but since both pages that generally pop up are so similar I'm starting to wonder if we would rank better with just one page. My example is the query "birthday gifts" Both of the URL's below show up in the search results one after the other on the first page. The URL on the top is our family page and the one below it is our subcat page, you can find both in the top nav. of our site. www.uncommongoods.com/gifts/birthday-gifts/birthday-gifts (family) www.uncommongoods.com/gifts/birthday-gifts (subcat) Both of these pages have different PA's and the subcat page that currently lives in our site nav is actually: **www.uncommongoods.com/gifts/birthday-**gifts?view=all. ****This url doesn't show up in the serps and is rel=canonicaled to the subcat page without the parameter listed above. We use this page in the nav because we think it's a better user experience than the actual subcat page. If we were to condense all three pages into one would we rank higher? Any thoughts here would be appreciated. Thanks
Intermediate & Advanced SEO | | znotes0 -
What is considered duplicate content?
Hi, We are working on a product page for bespoke camper vans: http://www.broadlane.co.uk/campervans/vw-campers/bespoke-campers . At the moment there is only one page but we are planning add similar pages for other brands of camper vans. Each page will receive its specifically targeted content however the 'Model choice' cart at the bottom (giving you the choice to select the internal structure of the van) will remain the same across all pages. Will this be considered as duplicate content? And if this is a case, what would be the ideal solution to limit penalty risk: A rel canonical tag seems wrong for this, as there is no original item as such. Would an iFrame around the 'model choice' enable us to isolate the content from being indexed at the same time than the page? Thanks, Celine
Intermediate & Advanced SEO | | A_Q0 -
Best practices with reoccurring event listings
On our client's events page there are a few reoccurring events that each have their own detail page. I'm trying to figure out what's the best practice for minimising duplicate content. For example, for the Bribie Island Markets that repeat weekly there are 2 (+more) detailed event pages: http://www.ourbribie.com/e/bribie-island-markets/1869/2013-12-07/2013-12-07
Intermediate & Advanced SEO | | michaelp85
http://www.ourbribie.com/e/bribie-island-markets/1869/2013-12-14/2013-12-14 While they both contain duplicated content, they're unique in that they display the specific events date/time. My thinking is that the future events (e.g. 2013-12-14) should have a canonical link to the upcoming/next event (i.e. 2013-12-07). However this would require constantly updating/changing the canonical links. What's the best way to deal with this from a duplicate content prospective? Any better recommendations?0 -
Links how long do they show?
How long do links show for in software such as Majestic ect once the link has been removed.
Intermediate & Advanced SEO | | BobAnderson0 -
301 redirect for ip address in SERPs
Hi, I've recently had the misfortune of my site's ip address being crawled and indexed by Google, which is causing some duplicate content issues. Due to the nature of the site we're not able to implement a canonical tag to fix this at present. Would a 301 redirect do the trick, and if so, could someone point me to what I'd need to add to our .htaccess file? Many thanks Chris
Intermediate & Advanced SEO | | ChrisHillfd0