How to block search bots in crawling my site except for homepage?
-
Thanks in advance!
-
Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png
-
You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt
or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though
Hope that helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any idea why Google Search Console stopped showing "Internal Links" and "Links to your site"
Our default eCommerce property (https://www.pure-elegance.com) used to show several dozen External Links and several thousand Internal Links on Google Search Console. As of this Friday both those links are showing "No Data Available". I checked other related properties (https://pure-elegance.com, http:pure-elegance.com and http://www.pure-elegance.com) and all of them are showing the same. Our other statistics (like Search Analytics etc.) remain unchanged. Any idea what might have caused this and how to resolve this?
Intermediate & Advanced SEO | | SudipG0 -
Moving career site to new URL from main site. Will it hurt SEO for main page?
For one of our clients we are building a career site and putting it under a different URL and hosting service (mainly due to security concerns of hosting it under the same host and domain). almost 100% of the incoming traffic to their current career section (which it is in a sub-folder) receives traffic for branded keywords (brand + job/career/employment), that is, there are no job position specific keywords. The client is now worried that after moving the site, the inbound traffic to the main site will be severely affected as well as the SERP results. My questions are, will the non-career related SERPs be affected? I don't see how will they be but I could be wrong If no, how could we reassure her that the SEO to the main site wont be affected? are there any case studies of a similar case (splitting part of the website under a new URL and hosting service?) Thank you for your help. PS: this is my first post so please forgive me if this has been asked before. I could not find a good response.
Intermediate & Advanced SEO | | rflores0 -
Seo flash site
Hey. Would hear whether it is possible to SEO a website which is flash site cms?
Intermediate & Advanced SEO | | Agger0 -
Zip Code Blocks the Search Engines!
I have a site where when you visit the product pages, it asks for your zip code. This is obviously blocking the bots from crawling the site. I know you can basically tell the bots how to ignore the zip code feature but I am not exactly sure how to do this. Any help would be appreciated
Intermediate & Advanced SEO | | lhawk0 -
Sitewide Vs HomePage Links For Network of Sites
I wanted to site wide link a few sites together as they are sort of in the same network of ownership and wanted some advice. 1X PR1
Intermediate & Advanced SEO | | upick-162391
2X PR2
2x PR3 Would it be best to just get home page links before the footer, the links will be within a paragraph of text OR Just site wide link them in the footer with a heading of "Our Shopping Network"0 -
ReLaunching a very old site
Hi, I am in the process of re-vamping a website that hasn't been touched for years and whose rankings slowly dropped. Any best practice in how to do it making sure that there's not any more loss and - hopefully - it could go back to the old glory? The website is http://www.nlp-world.com Thanks
Intermediate & Advanced SEO | | pdmonline0 -
How to best utilize network of 50 sites to increase traffic on main site
Hey All, First off I wanna thank everyone who has responded to all my previous questions! Love to see a community that is so willing to help those who are learning the ropes! Anyways back to my point. We have a main site that is a PR 3 and our main focal point for lead generation. We recently acquired 50 additional sites (all with a PR of 1-3) that we would like to use as our own little back linking campaign with. All the domains are completely relevant to our main site as well as specific pages within our main site. I know that reciprocal links will get me no where and that google is quickly on to the attempted 3 way link exchange. My question is how do I best link these 50 sites to not only maintain there own integrity and PR but also assist our main site. Thanks All!
Intermediate & Advanced SEO | | deuce1s0