Welcome to the Q&A Forum

Whenever the rogerbot comes back to my site for a crawl it seems to want to crawl urls that dont exist and thus causes errors to be reported...

Example:- The correct url is as follows:

/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/

But it seems to want to crawl the following:

/vw-baywindow/cab_door_slide_door_tailgate_engine_lid_parts/cab_door_seals/genuine_vw_brazil_cab_door_rubber_68-79_10330/?id=10330

This format doesn't exist anywhere and never has so I have no idea where its getting this url format from

The user agent details I get are as follows:

IP ADDRESS: 107.22.107.114
USER AGENT: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])

thats a great help cheers

wheres the best place to do an inbound link check?

cheers, i thought as much

We have a test version of a clients web site on a separate server before it goes onto the live server.

Some code from the test site has some how managed to get Google to index the test site which isn't great!

Would simply adding a robots text file to the root of test simply blocking all be good enough or will i have to put the meta tags for no index and no follow etc on all pages on the test site also?

Recently I was asked the question by a client who has some SEO knowledge.....

We built their site and during testing we simply set the home page links to be:

home

rather than:

home

This was mainly to help while in test and dev as they were on different servers to the live site and if we had put the full domain in it kept taking us from dev/test to live.

Is it bad to have just a slash for SEO purposes or is it REALLY that important on home links to have the full domain as a slash takes you back to the full domain?

Thanks

Site is still in testing but as an example the products are for cars and the same product can fit multiple cars, hence why it may be in more than one place

Another option i thought of would be that the other pages I simply add a no index meta tag to so it only indexes the one url.

The reason I say this as from experience of how seomoz crawls sites and some pages which i know are not duplicate content yet i have had many pages for many sites get tagged for duplicate content simply because there isn't enough of a difference on the page.

The product pages wouldb e very much the same as really all that would be different is the breadcrumb trail and possibly some other links 95% of the pages would be the same as they are the same product.

We have an eCommerce site we have built for a customer and the products are allowed to appear in more than one product category within the web site.

Now I know this is a bad idea from a duplicate content point of view, But we are going to allow the customer to select which out of the multiple categories the product appears in will be the default category.

This will mean we will have a way of defining what the default url is for a product.

So am I correct in thinking all the other urls where the product appears we should add a rel canonical to these pages pointing to the default url to stop duplicate content?

Is this the best way?

Many of our e-com sites we build for customers have drop down menus to help the user easily find products without having to click -

Example: http://www.customandcommercial.com/

But this then causes the report to trigger too many on page links

We do have a site map and a google site map

So should I put code in place not to follow the drop down menu link items or leave in place?

cheers

so even though the search could generate lots of extra pages you think I should leave the pages as is?

I have a customer site where you can search for products they sell

url format is:

domainname/search/keywords/

keywords being what the user has searched for.

This means the number of pages can be limitless as the client has over 7500 products.

or should I simply rel canonical the search page or simply no follow it?

so what was it, and how quick was it solved?

That worries me a little as if i now have to wait another week, I cant sit here doing nothing while I wait for it to crawl again and see if my last round of changes has improved things?

Can you force it to crawl more frequently

looks like its only crawled the home page so what would stop it past the home page, if it had errors would it tell me, as i know there are more pages as the last crawl it did over 1500

set up a new campaign

seomoz crawled the site and did about 1500 pages

the last crawl it now only lists 1 page

I can see anyway to see why, the site has not changed so why has the number of pages dropped?

We are a web design and hosting company

http://www.spiralsites.com

But when we register domains for clients, or they take email only or a name is parked we have it using a holding page  which can be seen as follows:

http://www.jcevents.co.uk/

is this a good thing?

Bad thing?

Could it would it should it benefit our rankings?

Should we do more to make it look less like our main site and actually try to use it better to help rank our other services pages?