Robots.txt: excluding URL
-
Hi,
spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course.
What is syntax for disallow these kind of urls in robots.txt?
Thanks so much
-
You don't want to do this in robots.txt. If you serve pages with these parameters, people will inevitably link to them, and even if they're disallowed in your robots.txt file, Google maybe still index them, according to this: "While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web."
This is what the rel=canonical tag is designed for. You should use that to tell Google the page is duplicate content of another page on your site, and that it should refer to that other page. You can read (and watch a video) about that here.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query for paginated URLs - Shopify
Hi there, /collections/living-room-furniture?page=2
On-Page Optimization | | williamhuynh
/collections/living-room-furniture?page=3
/collections/living-room-furniture?page=4 Is that ok to make all the above paginated URLs canonicalised with their main category /collections/living-room-furniture Also, does it needs to be noindex, follow as well? Please advice, thank you!1 -
Worth Dropping www. from Domain Name to Shorten URL
The Moz page grader identified several pages where the only problem is the length of the URL, above 75 characters. The site uses uses the www. prefix, and has for years, since 2001. Is it worth dropping the www. and doing a redirect to https://domainname.com to gain the 4 characters
On-Page Optimization | | FatRodent20130 -
Does having a \ on a URL make a difference?
On our website we have CMS pages which end normally without a /. However when linking from web banners e.c.t it always adds a / on the end. Will this have a negative impact on SEO?
On-Page Optimization | | AHF_Furniture0 -
Need advice on the better URL structure to go with
I am rebuilding our existing website on a new platform and need advice on which URL structure would be the most ideal. The following examples are of a product that we have with a very long page title. Not all of our products have titles this long, but enough of them do to cause some concern. I was also wondering if I should end the url with file type .html or if leaving it out is better. Thanks in advance! OPTION 1. this example just uses the root domain and the entire product title separated by dashes http://ewheels.nextmp.net/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 2. this example uses the crawl path as well as the entire product title http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 3. this example uses the crawl path and just the part number at the end since the folders already contain all the keywords necessary http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/ace-2090aff01silace-20105aff01sil
On-Page Optimization | | elementmotor0 -
To update or not to update news URLs ?
We manage a huge daily news website in my small country - keeping this a bit mysterious in case competitors are reading 🙂 Our URL structure is www.companyname.com/news/categoryofnews/title-of-article?id=articleid In this hyperreactive news world, title of articles change frequently (may be ten times a day for the main stories). The question we debate is : should we reflect the modification of the title in the URL or not ? Example : "Trump says he wants to ban search engines" would have URL http://www.companyname.com/news/entertainment/Trump-says-he-wants-to-ban-search-engines?id=12345678 Later in the day the title becomes "Trump denies he suggested banning search engines". Should the URL be modified to http://www.companyname.com/news/entertainment/Trump-denies-he-suggested-banning-search-engines?id=12345678 (option A) or not (option B) ? In Google News it makes no difference because of the sitemap, but in Google organic things are different. At present (option B in place), Google apparently doesn't see that the article has been updated, and shows the initial timestamp which is visually (and presumably SEOwise) not good : our new news looks like old news. Modifiying the URL would solve that issue, but could, may be, create another one : the new URL, being considered a new article, would lose, the acquired weight of the previous one in terms of referrals, social trafic and so on. Or not ? What do you think is the best option ? Thanks for your expertise, Yves
On-Page Optimization | | yves678901 -
Canonical URL, cornerstone page and categories
If I want to have a cornerstone "page", can I substitute an actual page with a category archive of posts "page" (that contains many posts containing the target key phrase)? This way, if I make blog posts about a certain topic/ key phrase (example "beach weddings") and add a canonical URL of the category archive page to the individual posts, am I right then to assume google will see the archive page as the cornerstone page (and thereby won't see the individual posts with the same key phrase as competing)?
On-Page Optimization | | stephanwb0 -
URL Strucutre
Hi there, Need some advice please on URL structure. I have been doing SEO for quite sometime now, however one thing that always get me is URL structure. I have a decision to make, its either: URL 1 /conditions/allergies/food/ URL 2 /conditions/allergies-food/ Lets say i am optimizing for the key-phase "Food Allergies" what do you think is best practice? I know that this is not a major factor in gaining high SERPs & maybe i'm thinking about it too much, however your input would be really helpful. Kind Regards,
On-Page Optimization | | Paul780 -
Include the company/domain name in page titles and urls?
I know this isn't something that I would use site-wide but I'm wondering if it helps or hurts me to use my company name (also my domain name) in pages below the homepage. As an example, let's say I'm Home Depot. In the category pages off the homepage should I use Page names and urls like Home and Garden Supplies or Home and Garden Supplies at Home Depot? Or does it hurt me to reuse my company/domain name on multiple pages?
On-Page Optimization | | kdieruf0