How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing/shortening a url hurt SEO?
Hi all, I am in the process of making small optimization changes to my site. I noticed Moz identified quite a few URLs that could be shortened. I intend to shorten these URLs and create 301 redirects to ensure website users land on the right page. My question is, will this change in URL damage rankings and engagement(assuming the URL remains content relevant)? I have read in some places that when creating URL redirects for a change in domain, people saw a dip in rankings and engagement. I, however, am not intending to change the main domain of the site, but rather the URL slug. Any thoughts?
On-Page Optimization | | annegretwidmer0 -
How can I make it so that the various iterations (pages) do not come up as duplicate content ?
Hello, I wondered if somebody could give me some advice. The problem of various iterations of the clanedar page coming up as duplicate content. There is a large calendar on my site for events and each time the page is viewed it is seen as duplicate content . How can I make it so that the various iterations (pages) do not come up as duplicate content ? Regards
On-Page Optimization | | Tony14Aug0 -
URL STRUCTURE & RE-WRITING
1.) I need a suggestion from you, please help. How should be the Directory/URL structure if I am offering servicves in many cities of UK/US. /<service>/
On-Page Optimization | | younus
/<service>/<state>/
/<service>/<state>/<city>/ OR /<service>/
/<state>/<service>/
/<state>/<city>/<service>/ Thanks for your time.</service></city></state></service></state></service></city></state></service></state></service></service> What exactly mean by 'URL enforce writer' to rewrite the URL.0 -
New bookingsengine url, what would you do?
A client of mine is introducing a new and improved bookingsengine. They're launching it on a different url than the existing one. The existing one needs to stay online a little bit longer for affiliate purposes. The old engine url has a sitelink in the SERPS and ranks well on a few terms. I'm wondering what you would do in this case? They want the new url to rank as quickly as possible also as sitelink of course. Any help greatly appreciated. I have some thoughts of my own of course... 🙂 But to keep the discussion as wide as possible... I'll wait a bit to add m thoughts.
On-Page Optimization | | YannickVeys0 -
Where can I find broken link.
Hi there, I got some 404-errors in crawl diagnostics. How can I get the URLs where the broken links were found?
On-Page Optimization | | fleetway0 -
Maximum length of a URL for good SEO?
Hi there, We have a content database as part of our site and I noticed that the way the database is loaded with new content, it means that the URL’s for these pages are really long, around 100 characters or sometimes more http://www.xxxyy.org/knowledge-base/documents/word1-word2-word3-word4-word5-word6-word7-word8 Is there a suggted maximum character length for a URL? Kind of like for title tag where I max out at 69… Should I truncate the URL’s or at least reduce the numbers of words in them to something less spammy? Does that make a difference? Thanks!
On-Page Optimization | | inhouseninja0 -
Meta refresh - nojavascript url
seomox is telling me that I am getting a page that is not being indexed or crawled and since the crawl status code is 200 and there are no robots the meta-refresh url must be the problem. the meta refresh url is different than the on page report card url as it's the nojavascript url which my developer says should be ok. see his comments below. The is redirecting to http://mastermindtoys.com/store/nojavascript.html only in case if the JavaScript is disabled in the client browser. This is the right way to do it, I don’t understand why this might be a problem, otherwise MM has to implement Noscript pages that have a real content. I didn’t get what’s wrong about accessibility. The code 200 means it is accessible, and yes there is nothing to access if JavaScript is disabled on browser. I think there are no modern retail sites that would do any sensible business with the scripting disabled in browsers.The H1 is really present 2 times and second occurrence can be removed, though I highly doubt about importance of this change.Regarding duplicates – what URLs are considered duplicates? Can you please send me examples?I am not aware of canonical URL problem for MM site unless we consider old .asp links as duplicate links of the canonical product pages. I would appreciate if SEOMoz gave us an example what they mean.I suspect that the page is not getting indexed as a result of this or I'm just not getting a good score. Which is it?
On-Page Optimization | | mastermindtoys0 -
Absolute URLs
Hi, this is a very basic question but I want to confirm, as I remembered it was consider a good practice to use the absolute version of your links when linking to other pages of your site, not for any issue related to passing authority or PageRank, but because if someone scraps your content then they would take the links as well (as if they didn't remove them). Have the practices for internal linking with absolute or realtive URLs changed in any way? Which is the best way? absolute or relative? is there any harm for using the relative version? Relative: Absolute: [](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html)[](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html) [Thanks!](<strong><em>http://www.cheapdomain.com/myfolder/mypage.html)
On-Page Optimization | | andresgmontero0