How can I block unwanted urls being indexed on google?
-
Hi,
I have to block unwanted urls (not that page) from being indexed on google. I have to block urls like example.com/entertainment not the exact page example.com/entertainment.aspx . Is there any other ways other than robot.txt? If i add this to robot.txt will that block my other url too? Or should I make a 301 redirection from example.com/entertainment to example.com/entertainment.aspx. Because some of the unwanted urls are linked from other sites.
thanks in advance.
-
Hi Nick,
Thanks so much!
Is there any advantage for shorter URLs over long ones? My site is on IIS6, How can I remove the .aspx extension in IIS6?
-
Since some of the unwanted URLs are linked from other sites, use 301 redirects.
Usually, people want to remove the file extensions like .aspx for cleaner looking, shorter URLs.
You can remove the .aspx extension from all pages with a rewrite like this in .htaccess:
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.aspx -f
RewriteRule ^(.*)$ $1.aspxIf you want to keep the URLs with .aspx, rather than without, I think you will have to use individual 301 redirects.
-
Vipin
My understanding is that if you block anything from Robots.txt that you prevent link juice from flowing for the pages. If the pages in question have links to them then I agree with Highland to 301 redirect them to the correct pages.
shivun
-
301 is definitely the way to go, especially if you have inbound links to the unwanted pages. That will preserve most of your rank and transfer it to the actual page.
As far as blocking goes, you can use a robots.txt or a robots meta tag on your page with a NOFOLLOW,NOINDEX
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Indexing our site
We have 700 city pages on our site. We submitted to google via a https://www.samhillbands.com/sitemaps/locations.xml but they only indexed 15 so far. Yes the content is similar on all of the pages...thought on getting them to index the remaining pages?
Intermediate & Advanced SEO | | brianvest0 -
404 broken URLs coming up in Google
When we do a search for our brand, we are get the following results in google.com.au (see image attachment). As outlined in red, there are listings in Google that result in 404 Page Not Found URLs. What can we do to enable google to do a recrawl or to ensure that these broken URLs are no longer listed in Google? Thanks for your help here! sBqpvtj
Intermediate & Advanced SEO | | Gavo0 -
If I only Link to Page via Sitemap, can it still get indexed?
Hi there! I am creating a ton of content for specific geographies. Is it possible for these pages to get indexed if I only put them in my sitemap and don't link to them through my actual site (though the pages will be live). Thanks!
Intermediate & Advanced SEO | | Travis-W
Travis0 -
How Can Google tell, if a anchor text is exact match
So, I was thinking to myself today. Couldn't Google say everything is an exact match anchor text in reality? Such as, Hyundai in Boston, Or cars in boston? I'm just concerned, that's all. Thanks for your help.
Intermediate & Advanced SEO | | PeterRota0 -
How to get content to index faster in Google.....pubsubhubbub?
I'm curious to know what tools others are using to get their content to index faster (other than html sitmap and pingomatic, twitter, etc) Would installing the wordpress pubsubhubbub plugin help even though it uses pingomatic? http://wordpress.org/extend/plugins/pubsubhubbub/
Intermediate & Advanced SEO | | webestate0 -
Changing Hosting Companies - Site Downtime - Google Indexing Concern
We are getting ready to switch to a new hosting company. When we make the switchover, our sites will be offline for a couple of hours and in some cases perhaps as long as 12 hours while DNS is configured -- should we be worried about Google trying to index pages and finding them unavailable? Any fear of Google de-indexing pages. Our guess was that Google would not de-index anything after just a short period of not being able to find pages -- it would have to be over an extended period of time before GOOGLE or BING would de-index pages -- CORRECT? Just want to gut check this before pulling the trigger on switch over to new hosting company. We appreciate input on this and/or any other thoughts regarding the switch over to new hosting company that we may not have thought of. Thanks, Matt
Intermediate & Advanced SEO | | MWM37720 -
Should I 301 Poorly Worded URL's which are indexed and driving traffic
Hi, I'm working on our sites structure and SEO at present and wondering when the benefit I may get from a well written URL, i.e ourDomain / keyword or keyphrase .html would be preferable to the downturn in traffic i may witness by 301 redirecting an existing, not as well structured, but indexed URL. We have a number of odd looking URL's i.e ourDomain / ourDomain_keyword_92.html alongside some others that will have a keyword followed by 20 underscores in a long line... My concern is although i would like to have a keyword or key phrase sitting on its own in a well targeted URL string I don't want to mess to much with pages that are driving say 2% or 3% of our traffic just because my OCD has kicked in.... Some further advice on strategies i could utilise would be great. My current thinking is that if a page is performing well then i should leave the URL alone. Then if I'm not 100% happy with the keyword or phrase it is targeting I could build another page to handle the new keyword / phrase with the aim of that moving up the rankings and eventually taking over from where the other page left off. Any advice is much appreciated, Guy
Intermediate & Advanced SEO | | guycampbell0 -
Rel= Author : Google suggests that we include this. Can i link to my FB profile
There have been several reference on how Google is trying to build author profiles so that it can effectively measure the authors authority based on the Rel=Author. I quite liked this idea , but my site is a user generated content website. So how can i leverage this ? . One thing which i was thinking was if its a good idea to link to the customers Facebook profile , plan to collect this during sign up.
Intermediate & Advanced SEO | | ShoutOut0