Is it possible to have the crawler exclude urls with specific arguments?
-
Is it possible to exclude specific urls in the crawl that contain certain arguments - like you can do in google webmaster tools?
-
You will need to do a block in your robots.txt based on a match:
**User-agent: ***
Disallow: /variable=I think the above would work.
The other approach would be to add a line in your page code itself to generate a 'nofollow' robots tag when that argument is present.
This link should help you, by the way:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Numerous duplicate destination URLs from within one menu - potential impact for on-page SEO?
Hello all What is your evaluation in regards to a number of links (different anchors) targeting the same destination URL from within one and the same menu (on the same website)? Keeping it brief: Think of a top menu drop down entry, that needs to feature the alphabet (each letter has it's own sub-entries). However, the actual letter itself is not represented by a page (it has no URL either). So far so good. However, when testing the menu on a mobile device, the letter entries are still treated, as if they were non-existent pages - thus throwing a 404 when clicked. In order to avoid people getting a 404 when clicking on any letter, it would be ideal, if they were directed to any main page (the same destination URL though). However, that would mean 26 times the same destination URL from within that menu. Is this approach potentially bad for SEO, hence there would be numerous duplicate destination URLs in place? Please mind, I am not inquiring for help on how to arrange the actual menu. I am concerned about the impact, identical destination URLs could have on the on-page SEO. Many thanks in advance for your help and input!
On-Page Optimization | | Hermski0 -
URL Structure Category Pages -Current Moz Friday-
Hello,
On-Page Optimization | | _Heiko_
regarding #15 of the last moz friday I have a question: http://moz.com/blog/15-seo-best-practices-for-structuring-urls What would you prefer if the lenght of the URL will be still under 60 characters and you have an example like this: Let's call it a specific page in a category. As I like the old shoe examples: You have a page about red shoes in your shoe category. Which URL would you prefer: a) www.mydomain.com/shoes/red-shoe b) www.mydomain.com/shoes/red Personally I would prefer a) or would you already consider this as spammy? My real example is not that trivial like the shoe example and the categories will be in plural and the specific pages always in singular (like in the example shoes vs shoe). c) would be to put it independently from the side structure on www.mydomain.com/red-shoe - but personally I have the experience that a) or b) will help the rankings of the category page if you have the specific pages in the same subfolder. What's your opinion on this?1 -
URL structure
Hello all, I am about to sort out my websites link structure, and was wondering which approach to our services page would be best. should we have: services/digital-marketing & services/website-design etc or: digital-marketing/website-design & digital-marketing/seo Basically I see digital marketing as the top level category that is the umbrella term for all of our digital services. But would it make more sense to have service to be the main category and digital marketing within that (along with all the other services from web design to seo)? all thoughts welcome!
On-Page Optimization | | wseabrook0 -
Changing URL Structure From Flat to Pyramid Theme
Hello Mozzers, I have an on-page SEO question regarding URL structure. A few months back we hired a full-time SEO person who is working on-page right now and she really wants us to completely re-due our URL restructure from a flat to pyramid style (example below). Current URL structure / page title is: Dog training Collars - K9electronics.com
On-Page Optimization | | k9byron
http://www.k9electronics.com/dog-training-collars/ Small Dog training Collars - K9electronics.com
http://www.k9electronics.com/small-dog-training-collars/ Einstein ET-300TS Mini Dog Training Collar - K9electronics.com
http://www.k9electronics.com/einstein-et-300ts-mini-remote-dog-trainer.html Suggested URL structure / page title change: Quality Dog Training Collars - Lowest Price Guarantee - K9electronics
http://www.k9electronics.com/dog-training-collars/ Dog Training Collars - Small Dog - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/ Einstein ET-300TS Mini - Dog Training Collars - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/einstein-et-300ts-mini I guess you could say we are the poster-boys for Google penalties and have received just about every penalty in the book. Panda, penguin algo penalties and a partial manual action for unnatural links. Several months ago we removed 1000's of in-bound links and had our manual action lifted a few months back and are now hovering around the top / mid 2nd page for all our big terms ...we used to be top 3 for everything. As we were removing bad links, we also completely redesigned the site and removed lots of categories and products and 95% of all our old, low quality content and replaced it with new, high quality content. The site was really slooooow, so we optimized it and moved it to a big dedicated server and tripled page load time. Added rich snippets, Google authorship, increased our FB and other social presences and much more ... I had also considered this URL structure change during the redesign because I had heard and read that it was good to do, but it required redirecting practically all our URL's which I know can hurt the site even more so then it already has been ... Our SEO says that as it sits now, our pages are competing with each other and really seems to think this is going to improve our rankings a lot ...after several weeks. My question is, at this stage in the game, is it really going to help a lot and give us more benefit compared to the 301 redirect link juice loss? Any comments and/or suggestions are very much appreciated!1 -
"On Page" report says 2 rel canonical urls-how do I fix that?
I am reviewing my On Page scores and I'm not getting a perfect score bk of this notice: No More Than One Canonical URL Tag Moderate fix <dl> <dt>Number of Canonical tags</dt> <dd>2</dd> <dt>Explanation</dt> <dd>The canonical URL tag is meant to be employed only a single time on an individual URL (much like the title element or meta description). To ensure the search engines properly parse the canonical source, employ only a single version of this tag.</dd> <dt>Recommendation</dt> <dd>Remove all but a single canonical URL tag</dd> <dd>HOW do I fix that?</dd> <dd>I am using Platinum seo plugin which I have checked "Use canonical urls" and the page in question is</dd> <dd>http://adderalldosage.net/general-adderall-dosage/</dd> </dl>
On-Page Optimization | | ccare7230 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Does google treat all urls equal?
Sorry for the lame title, i couldn't think of a better one. I want to know if google treats this: http://www.domain.com/products/some-product-name the same as it would treat: http://www.domain.com/?products=some-product-name if not, could you tell me the differences?
On-Page Optimization | | adriandg0 -
Switching URL from keyword heaven to actual brand name?
Our client has a site, we'll say it's delicious-lemonade.com (That's an example.) Their brand name, however, is PowerSky. (That's an example, but the point is that it's the name of a technology, and has NOTHING to do whatsoever with being a drink. Someone would never guess what the product was, by the brand name.) The client has this domain name for a long time. The domain name itself is a top keyword for their product. We are building a new site for the client and have planned to use the brand name (PowerSky) as the primary domain, so PowerSky.com, with the product page being PowerSky.com/Delicious-Lemonade. And we are planning to redirect Delicious-Lemonade.com to PowerSky.com. However, we are concerned about the SEO hit the site is going to take. What recommendations can you make in this situation?
On-Page Optimization | | grayloon0