Needs clarification: How "Disallow: /" works?
-
Hi all,
I need clarification on this. I have noticed we have given "Disallow: /" in one of our sub-directory beside homepage. So, how it going to work now? Will this "Disallow: /" at sub-directory level going to disallow only that directory or entire website?
If it is going to work for entire website; we have already given one more Disallow: / at homepage level blocking few folders. How it is going to handle with two Disallow: / commands?
Thanks
-
Hi vtmoz,
You've received some great responses! Did any of them help answer your question? If so, please mark one or more as a "good answer." And if not, please let us know how we can help. Thanks!
Christy
-
If you have concerns, I strongly recommend using Google Search Console to test URL use cases against your existing robots.txt file and before you do potential edits.
-
The directive that is literally "Disallow: /" will prevent crawling of all pages on your site, since technically, all page paths begin with a slash. Robots.txt files can only live at the root folder (subdirectory) of a site, so if you want to disallow a folder, you'll need to specify that with a directive like "Disallow: /folder-name/
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need to 301 redirect www.domain.com/index.html to www.domain.com/ ?
So, interestingly enough, the Moz crawler picked up my index.html file (homepage) and reported duplicate content, of course. But, Google hasn't seemed to index the www.domain.com/index.html version of my homepage, just the www.domain.com version. However, it looks like I do have links going specifically to www.domain.com/index.html and I want to make sure those are getting counted towards my overall domain strength. Is it necessary to 301 redirect in the scenario described above?
Web Design | | Small_Business_SEO0 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
How to do a non-spammy "doorway page"?
Hi there, ISSUE: I have a client who wishes to use a "doorway" page, but not in a spammy way. He would like to have a nice crisp URL for use in ads/brochures. The page is strictly a landing page (just with a separate URL). DOORWAY/LANDING PAGE WILL BE: Non-spammy -- There will be no attempt to optimize the landing page/no attempt to get the page to rank. Strictly a vanity URL -- he likes the way a separate website looks in ads as opposed to a landing page on the existing website (i.e., www.websitename.com/landing page) WHAT I'M TRYING TO DO: I'm basically trying to figure out what the best things to do to protect his other sites (which are very high quality valuable sites which rank well) from getting punished. STEPS I'M CONSIDERING: Robots no follow Separate hosting server Different person's name on a private domain registration Adding additional pages, so it's not a 1-page "doorway" Many thanks in advance to anyone who would share their experience and help me protect my client in the best way possible. I've told him there are risks, but he still wants to go ahead. MC
Web Design | | marketingcupcake1 -
Does it do harm if you add a rel="canonical" tag on a page that doesn't need it?
If a page is clearly unique and there is obviously no canonical tag needed, does it hurt anything if one has been added?
Web Design | | jaychow0 -
Should I nofollow my blog "read more" links?
I have a standard blog index page with entries formatted as such: Blog title Blog excerpt read more link The blog title is linked to the blog post (duh). The "read more" link is also linked to the same blog post, but I put a rel="nofollow" tag on that link because I don't want the SE's thinking "read more" is relevant anchor text. Now, is what I'm doing right? Won't having the read more link as nofollow result in some sort of conflict considering the blog title link is set to follow? Will a nofollow link have a cascading effect on any matching "follow" link? I've frequently read that "read more" should be nofollowed, but I've never once seen anyone address the conflict of having a follow and nofollow link on the same page pointing to the same link. Which one wins? Thanks.
Web Design | | bluekite770 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0 -
How does the "first link" rule work with the "reasonable surfer patent" when it comes to the main navigation for a website?
In trying to figure out navigation for a new website, I am struggling with the first link rule vs. the reasonable surfer patent where the first link rule implies that Google "counts" the first link to a page including navigation, and the reasonable surfer patent that implies that navigation links carry less weight than body copy links. What is the best solution for creating main navigation so that it doesn't take away from the body copy links?
Web Design | | cindyt-170380