How to block search bots in crawling my site except for homepage?
-
Thanks in advance!
-
Robots.txt exclusion is definitely the easiest way to go. The URLs within the site might still be "indexed", but they will not be crawled and if they ever showed up in a search, they would look like something like this: http://i.imgur.com/xU6mDYA.png
-
You can use robots meta great guide here - http://moz.com/learn/seo/robotstxt
or if you wanted to be a bit different you can hide content behind a login screen or flash etc. things robots cant get through, I'd recommend the robots though
Hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Merging Niche Site
I posted a question about this a while ago, but still haven't pulled the trigger. I have a main site (bobsclothing.com). I also have a EM niche site (i.e shirtsmall.com). It would be more efficient for me to merge these site, because: I would have to manage content, promos, etc. on a single site. In other words, I can focus efforts on 1 site. If I am writing content, I don't have to split the work. I don't have to worry about duplicate content. Right now, if I enter a product URL into copyscape, the other sites is returned for many products. What makes me apprehensive are: The niche site actually ranks for more keywords than the main site, although it has lower revenue. Slightly lower PA, and DA. Niche site ranks top 20 for a profitable keyword that has about 1300 exact match searches. If you include the longer tail versions of the keyword it would be more. If I merge these sites, and do proper 301s (product to product, category to category) how likely is it that main site will still rank for that keyword? Am I likely to end up with a site that has stronger DA? Am I better off keeping the niche site and just focusing content efforts on the few keywords that it can rank well for? I appreciate any advice. If someone has done this, please share your experience. TIA
Intermediate & Advanced SEO | | inhouseseo0 -
Noindex search pages?
Is it best to noindex search results pages, exclude them using robots.txt, or both?
Intermediate & Advanced SEO | | YairSpolter0 -
Why is my site not getting crawled by google?
Hi Moz Community, I have an escort directory website that is built out of ajax. We basically followed all the recommendations like implementing the escaped fragment code so Google would be able to see the content. Problem is whenever I submit my sitemap on Google webmastertool it always 700 had been submitted and only 12 static pages had been indexed. I did the site query and only a number of pages where indexed. Does it have anything to do with my site being on HTTPS and not on HTTP? My site is under HTTPS and all my content is ajax based. Thanks
Intermediate & Advanced SEO | | en-gageinc0 -
How is my 301 redirected site stealing rankings from the main site?
Hello, I have a site, drhobelt.com, that 301 redirects to the main site, drhonow.com. Not only is drhobelt.com still indexed, but it recently stole rankings from drhonow.com for "decompression belt" related terms. What could be causing this? How do I reclaim the rankings for drhonow.com? Thanks for reading!!
Intermediate & Advanced SEO | | DA20130 -
Changing the relevance of the homepage
One of my new clients is hell bent on changing the content of their homepage. They are one of the world's largest resort companies. The site is graphics-heavy (with embedded text), and barely contains any content. I haven't started any of the on-page optimization yet, but when I do, it will be a major overhaul. Despite the poor on-page of the site, they are getting great rankings and a ton of traffic due to number and quality of their backlinks and domain authority. My concern is this: they want to change the homepage and make it into a "vacation sweepstakes" type of page. Their logic seems to be that they will generate a lot of interest on the site and get people excited about winning an expensive dream vacation, which is all fine and dandy, however, my feeling is that this will change the relevance of the page. So, instead of pitching their ownership-based program, now, they will be promoting vacation contests. So here's the stupid question: would this have the potential to negatively affect their search engine results or the Domain Authority? I'm thinking of suggesting to them a less drastic approach. Perhaps something like the lightbox sweepstakes overlay on marshallsonline.com. At least, this way, we can keep the current homepage and improve on it, rather than going into another niche. Any feedback or suggestions on this is greatly appreciated!
Intermediate & Advanced SEO | | ollan0 -
I have a general site for my insurance agency. Should I create niche sites too?
I work with several insurance agencies and I get this questions several times each month. Most agencies offer personal and business insurance and in a certain geographic location. I recommend creating a quality general agency site but would they have more success creating other nice sites as well? For example, a niche site about home insurance and one about auto insurance. What would your recommendation be?
Intermediate & Advanced SEO | | lagunaitech1 -
Are Links from Homepage safe?
I have the option of getting a link from 2 websites home pages. My site is a financial service website. I want to know if it is safe to do so. The first site is a hotel site, so their content has no relation to my site And the second is a Google News website about finances They have offered me a section in the sidebar to have a sentence or 2 about my site with a link. Can you tell me would these be safe to get? I don't want to be hit by any penalty
Intermediate & Advanced SEO | | JohnPeters0 -
Niche sites: how to optimize them?
Dear SEOmozzers, I am focusing on those keywords I find using the "Keyword Analysis" tool that are not too competitive (among 20%-30% competitiveness). I then buy domains that include those keywords. The sites are in Italian and targeting the Italian search engines, so the competition is lower than it would be in the US. Basically, I'd like to build niche sites and I'd like to ask a few questions that I hope somebody with a good experience in this field can answer: How do I optimize a niche site. Specifically, how do I go about link building? How many backlinks should I get to see some results? How long does it take for a typical niche site to start appearing in the search engines for a certain keyword, after launching an effective link building campaign? Please kindly provide any recommendations you believe to be important when building niche sites. For example, is there a company/professional you know that specializes in this field and who is trustworthy/reliable? Thank you very much for your help. All best, Sal
Intermediate & Advanced SEO | | salvyy0