How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL Keywords
I am doing SEO on our eCommerce website and read that I should include keywords in the URL The original URL is: http://thegiftlinks.com/personalized-wedding-glass.html
On-Page Optimization | | abdulw
Title page: Wedding gift Dubai - Anniversary gift Dubai - Personalized Wedding Glass
Meta Data:
Wedding gift Dubai - Anniversary gift Dubai - Personalized Wedding Glass
It is great for a wedding gift and anniversary gift for friends and family members. If I will include the keyword to the url it will be like this
http://thegiftlinks.com/personalized-wedding-glass.html/Wedding-gift-Dubai is this the correct way to include keywords in the URL? Thanks0 -
Can we use "top 10" in title?
Hi All, This is a query regarding title tag.I have a listing page with list of hotels in each city. To improve CTR, I used "Top 10 hotels in Boston" as title even though I am showing 25 hotels per page. The page takes about "hotels" but I didn't use "top 10" term anywhere in the page. Is this called keyword stuffing? Should I use "top 10" somewhere in the page as my title says so? Thanks for your help!
On-Page Optimization | | Avinash_12340 -
How would you improve our URL structure?
Hi Mozzers, I have a question about the URL structure on our website (www.ikwilzitzakken.nl). We now have a main category with "zitzakken" (beanbags). We also have different brands, types and colours. Now we have URL's like this: <a>https://www.ikwilzitzakken.nl/zitzakken/vetsak/vetsak-fs600-flokati-zitzak/_381_w_3544_3862_NL_1</a> which seems long and not clean. Please don't look at the query at the end, we can't do anything about that in our CMS. In english this would be: https://www.iwantbeanbags.nl/beanbags/vetsak/vetsak-fs600-flokati-beanbag/_381_w_3544_3862_NL_1 How would you optimise this? We do have good rankings (this one ranks #1 for example), but I think our overall structure could be way better. Would love your thoughts about this.
On-Page Optimization | | TheOnlineWarp0 -
Strange URL resulting a page
Hi, my friend has asked me to take a look at his site. I only know the basics of SEO so I'm learning along the way. He has some duplicate title errors showing in Moz, resulting to this page: https://www.domainname.com/about/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers This URL shows the 'About' page. I have tonnes of pages like this showing with really long URLs that result an actual page. Has anyone seen something like this before? I don't have a clue how this is showing the about page Any advice is greatly appreciated. Thanks James
On-Page Optimization | | Craze_Media0 -
Indexed/Submitted URLS vs Total Indexed
Hello, My site is www.colbysphotography.com. I have Total Indexed 195 while I have 87 URLs submitted and only 79 URLs Indexed. What is the difference and is there a problem? Thanks ahead of time,
On-Page Optimization | | littlecolby
Colby0 -
Keyword in URL?
I have a website that has been live for about 8yrs. I do not have any significant rankings for my main keywords but am now starting SEO on my site. I am contemplating changing the url to contain the main keyword prefixed by my brand name. Any views on the ranking benefits and or CTR benefits.
On-Page Optimization | | Johnnyh
Example:
Main Volume keyword - 'car leasing'
current url - www.bobleasing.co.uk (made up name) thinking of changing to - www.bobcarleasing.co.uk (made up name) Any advice would be much appreciated. John0 -
Can my amount of internal linking seem spammy ?
Ecommerce site. I am optimizing for each producer of products on a separate page. Atm my provider does lack some functions(i cant put in H1 , title and can put text only product pictures) on the product pages - like here http://www.epleskrinet.no/smafolk/M_23 . The are updating within 1-2 months to allow me do this. This has led to actually some of the products themself ranking higher that the producer page. what Ive been doing is to put anchor text and link back to the producer page for all 9 products on this page. Is that a problem or should i just do it like this ? thanks
On-Page Optimization | | danlae0 -
Duplicate product urls
Our site automatically creates shorter urls for the products. There is a rel canonical tag in place, but webmaster tools shows these urls have duplicate title tags. Here is an example: http://www.colemanfurniture.com/holden-desk.htm http://www.colemanfurniture.com/writing-desks-secretary-desks/holden-desk.htm Should the longer url be redirected to the shorter one?
On-Page Optimization | | thappe0