How to Hide Directories in Search?
-
I noticed bad 404 error links in Google Webmaster Tools and they were pointing to directories that do not have an actual page, but hold information.
Ex: there are links pointing to our PDF folder which holds all of our pdf documents. If i type in , example.com/pdf/ it brings up a unformated webpage that displays all of our PDF links.
How do I prevent this from happening. Right now I am blocking these in my robots.txt file, but if i type them in, they still appear.
Or should I not worry about this?
-
Yes, a visit to example.com/dir should now return a 404 error (if you haven't done any redirecting/canonicalizing). This will increase your 404 count in Web Master tools but it's far preferable to the alternative. If you're not redirecting the robots.txt will eventually work and hopefully the links will just fall out of WMT.
-
My hosting company turned off directory browsing and now everything is how it should be. So to my understanding, if the server sees a file that does not have a index file, it should not be view able and should be forbidden. This shoujld not affect us from an SEO standpoint should it? My hosting company said they disabled all directories in our site, however everything still works, except for the forbidden file directories.
-
Basically it shouldn't really have an affect; those unformatted file listings are literally the web server automatically saying 'here's the files that are in this folder', there's no meta tags, description, on page elements, etc.
If you have these pages and they're ranking well, you generally don't want them to be. The automatic file browsing pages don't have your name, your company, etc. in them, and they're generally pretty ugly. They also theoretically could be 'stealing' juice from your 'real' pages, if your internal structure isn't flowing relevance properly.
Basically what I'm saying is that if these pages are having some kind of SEO effect, you probably don't want them to be since they're so basic.
Also I can't overstate the security concerns that directory browsing might be introducing. If someone can directory browse to where your code lives (.php, .aspx.vb, whatever) they may be able to read it. Code sometimes has important things like logins, passwords, merchant account ids, etc. in it that you definitely don't want people reading.
-
Agreed with Valerie that step 1 is to turn off those directory listing pages - that can be a security issue and you don't necessarily want people to see/access the whole list. Also, make doubly sure you don't have any internal links to that directory (Google crawled it somehow).
Generally, Robots.txt should prevent crawling, but it's not foolproof, and it's pretty bad about removing pages once they're indexed. If you can block the page from browsing and return a 404 for the root page, that should be fine. The other option would be to have the page removed in Google Webmaster Tools. You could request removal for the entire folder, but I'm guessing that you may want the actual PDFs indexed.
-
Will turning of directory browsing affect Search for all directories?
-
I really don't want to 301 redirect them as they are just holding files. This is happening with my includes file too. that holds our header, footer, navigation etc. I can check with our hosting company to find out.
-
I'd create an index.html for the directory, and then redirect it somewhere. This way, you're capturing the inbound links and then rescuing some of the inbound juice.
Otherwise, you can also check out this post for more info on other solutions and modifying your htaccess file to prevent the directory view - http://perishablepress.com/better-default-directory-views-with-htaccess/
-
Blocking it in robots.txt will work to hide it from search engines.
If you want to hide it from users or people to who type in the url, you can simply drop a blank "index.html" in the /pdf folder.
-
I would suggest 301'ing them to their /index.htm or /pdf.htm equivalents. If you don't know, a 301 is a signal to a web browser (or search crawler) saying "this page has permanently moved, please go to (otherpage.htm) instead".
Here's a good SEOMoz article explaining it a bit more:
http://www.seomoz.org/learn-seo/redirection
What might be more of a concern, is it sounds like your web server has directory browsing enabled. This could be a security issue (depending on your web server setup). Generally you don't want to expose directories if you don't have to because it gives a potential attacker insight into your system setup. Here's an example how to do it in Apache:
www.camelrichard.org/topics/Apache/Turn_OffDirectoryBrowsing
And IIS:
technet.microsoft.com/en-us/library/cc731109(v=ws.10).aspx
If you like I can confirm if you have open directories if you give me the link, either here or through private message.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suddenly keywords Disappeared from Google Search Results
Hello Guys Please help me, suddenly all of my site's keywords are disappeared from google search result, most of keywords are no.1 on google but today after 6pm i see the traffic decreasing and when i search my keywords there is no any keywords in search result. Only homepage keyword is showing. Please Help what is Happening with me.
Intermediate & Advanced SEO | | mianazeem4180 -
Sitelink Search Box mark-up when multiple query strings are used
Hi all, I'm looking to implement sitelink search box mark-up in Google Tag Manager in JSON-LD format. This would be popped into the Custom HTML tag and would look a little something like: The above option is great if you have one query string for your search term, but what if you had a URL that triggered two query strings - for example: https://www.example.com/search?q=searchterm&category=all Would you need to amend the code something like the below: Any help would be much appreciated! Cheers, Sean
Intermediate & Advanced SEO | | seanginnaw0 -
Directory links with no follow
Hi I'm researching competitor backlinks & they have a lot of directory links which are no follow - but they rank very well. Is this type of link building even allowed by google? I know they they aren't allowed followed directory links, but will no following them help with rankings?
Intermediate & Advanced SEO | | BeckyKey0 -
Directories that Redirect - Do They Pass Link Juice?
I did some searching before asking but could not quite find what I was looking for. There are valid directories out there that provide business as well as links that provide SEO value. My question is whether or not having a redirect in place negates passing any link juice. When I use Open Site Explorer for Old Monterey Inn, this directory (CABBI) does not show up on their list. However, their website dropped from Google Analytics altogether for some time because of an issue in how they built their site. Their "fix" is this redirect which was integrated a short time ago. I do see traffic in Google Analytics now but wonder about the link juice. Example: <a href="[/redirect?type=website&inn=34211&url=http%3A%2F%2Fwww.oldmontereyinn.com](view-source:https://www.cabbi.com/redirect?type=website&inn=34211&url=http%3A%2F%2Fwww.oldmontereyinn.com)" target="<a class="attribute-value">_blank</a>">www.oldmontereyinn.coma>p> What say you? Thanks to anyone that responds.
Intermediate & Advanced SEO | | ColoradoMarketingTeam0 -
Does alt tag optimization benefit search rankings (not image search) at all?
The benefits of alt tag optimization for traditional SEO has always been a "yo yo" subject for me. Way back in the day (2004 to 2007) I believed there was some benefit to alt tag SEO. However as time went on I saw evidence that the major search engines were no longer considering alt tag SEO as a ranking signal. However I later had the pleasure to work on a joint project with a high end SEO firm in 2011/2012. My colleagues fully believed that alt tag optimization was still a very important strategy for traditional SEO at that time. Is there any evidence available that alt tags still help with traditional SEO nowadays? I'm fully aware of the benefits of optimized alt tags and image search. However could optimized alt tags be one of those ranking factors that Google removed due to abuse and later quietly resurrected?
Intermediate & Advanced SEO | | RosemaryB0 -
Are prices shown in search results good for e-commerce sites?
Hello here. I own an e-commerce website (virtualsheetmusic.com) and with the fact we have implemented structured data for our product pages, now our search results on Google appear with pricing information whereas most of our competitors don't have that information displayed (yet). I am wondering: Do you think is that good? What side effects could that cause? Less CTR? Less bounce rate? Less traffic? Any thoughts on this issue are very welcome. Thanks!
Intermediate & Advanced SEO | | fablau0 -
Directory and Classified Submissions
Are directory submissions and Classified Submissions still a good way to create backlinks? Or they are obsolete methods and should be discontinued?
Intermediate & Advanced SEO | | KS__0 -
Foreign languages and first page listings on search engines
I have a customer that has asked me to find out the best way to move forwards with foreign languages. He sells Jukeboxes in the UK and has found recently that a large number of his customers are coming from abroad due to the financial climate. So he wants to appear on the first page of google in Russia, Arabic countries, French speaking countries and China. Whats the best route ? Add pages to his current .co.uk site or look at purchasing domains for each country. Then submitting them to each google. So Google.ru etc. Thanks
Intermediate & Advanced SEO | | onlinemediadirect0