Robots.txt question
-
What is this robots.txt telling the search engines?
User-agent: * Disallow: /stats/
-
Oh - and it's affect the domain negatively.. when cleaning up your site directories via robots.txt. Its actually better as I explained below
-
Hey Mark,
It's good practice to disallow access to any folder/content you don't want indexed as well as anything that has any security involved (login's, databases etc).
It will also keep the most important pages from the domain in front of the search spiders eyes, while keeping poor content out of the indes. This helps the domain on a site authority level provide valuable content and information to users.
Lower ranking pages, can cause the domain to be pulled down by serarch engines (Google and Bing have attested to this already) as they want businesses to focus on high value content - which leads to better user experience.
Cheers!
-
Thanks- wanted to make sure all was copacetic there. I'm assuming that it's good practice to disallow access to stats and won't impact the site negatively?
-
Assuming that this is the entire contents of this file: It says that no robot (search engine spider, other crawler, etc.) should visit or index anything in the /stats/ directory or any directories inside of it.
More info available here: http://www.robotstxt.org/robotstxt.html
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Subdomain Ranking Question
Hi All - Quick question that I think I know the answer to, but I feel like I've been going around in circles a bit. My client is launching a new product and wants us to build a microsite for it (product.clientname.com). My client really dislikes their brand website, and wants to use paid media to push their audience to this new microsite. However, they also said want it to rank well organically. I feel uneasy about this, because of the subdomain vs. subfolder argument. I believe that the product will also be listed/featured on their main brand website. What is the best way forward? Thanks!
Technical SEO | | AinsleyAgency0 -
Help with google news application url question
Hi, i am going to be applying to have out site in google news but i have come across the below and not sure how we do this. I use joomla and our site is www.in2town.co.uk and the page we are including is http://www.in2town.co.uk/latest-news-headlines Article URLs. To make sure that we only crawl new articles, please make sure your URLs are unique with at least 3 digits, and are permanent. can anyone please let me know how i do this with the url please
Technical SEO | | ClaireH-1848860 -
How to solve the meta : A description for this result is not available because this site's robots.txt. ?
Hi, I have many URL for commercialization that redirects 301 to an actual page of my companies' site. My URL provider say that the load for those request by bots are too much, they put robots text on the redirection server ! Strange or not? Now I have a this META description on all my URL captains that redirect 301 : A description for this result is not available because this site's robots.txt. If you have the perfect solutions could you share it with me ? Thank You.
Technical SEO | | Vale70 -
Canonical question
I have at least three duplicate main pages on my website: www.augustbullocklaw.com www.augustbullocklaw.com/index augustbullocklaw.com I want the first one, www.augustbullocklaw.com to be the main page. I put this code on the index page and uploaded it to my site: http://www.augustbullocklaw.com/canonical-version-of-page/" rel="canonical" /> This code now appears on all three pages shown above. Did I do this correctly? I surmise that www.augustbullocklaw.com is pointing to itself. Is that ok? I don't know how to take the cononical code off the page that is the page I want to be the main page. (I don't know how to remove it from www.augustbullocklaw.com, but leave it on www.augustbullocklaw.com/index and augustbullocklaw.com) Thanks
Technical SEO | | Augster990 -
The Ultimate Question
This is probably going to look bad, but Im going to do it anyway. Ive owned my own business for over twenty five years. Due to the economy and a health crisis, my business has come to an end. Im left pretty much with nothing except my family, which is on shakey grounds as well. anyway. Ive been working on the IM, SEO thing for about two years. Ive made some money, but nothing that would make a difference to my situation. I now have a site in a competitive niche. The site is rated A by the app here on seomoz. But my site is lacking in authority backlinks. I write my own posts, about two a day, and honestly, it is of much better quality than those ranking . My site stays right around #11. Every other day, the site goes to #7 or 8. When that happens, I make between 300-400 a day. Then it falls back to 11. So the on site is good, rated A here. My site is speedy Not enough backlinks The content is good I was foolish enough to believe the G story about backlinks not being important. Concentrate on the content. All of that crapola. The sites that are ranking well have all the typical xrumer links. Totally not related links. Junk. And from what Ive seen, I believe the sites ranking 1-3 are making about $1000. a day. So you can imagine what being this close, after losing so much is doing to me. So.... Right now I am just writing content, pinging, social bookmarking and submitting rss feeds daily. Does anyone have any real information on how to bump my site up in rankings. Please, no stories about guest blogging, or attempts to sell me your SEO service I know I would do it for someone else and I have many times in the past. If you can help me, please reread, and contact me. I wish you all the best. Sorry for the melodrama.
Technical SEO | | vansy0 -
Canonical Question
Our site has thousands of items, however using the old "Widgets" analogy we are unsure on how to implement the canonical tag, and if we need to at all. At the moment our main product pages lists all different "widget" products on one page, however the user can visit other sub pages that filter out the different versions of the product. I.e. glass widgets (20 products)
Technical SEO | | Corpsemerch
glass blue widgets (15 products)
glass red widgets (5 products)
etc.... I.e. plastic widgets (70 products)
plastic blue widgets (50 products)
plastic red widgets (20 products)
etc.... As the sub pages are repeating products from the main widgets page we added the canonical tag on the sub pages to refer to the main widget page. The thinking is that Google wont hit us with a penalty for duplicate content. As such the subpages shouldnt rank very well but the main page should gather any link juice from these subpages? Typically once we added the canonical tag it was coming up to the penguin update, lost a 20%-30% of our traffic and its difficult not to think it was the canonical tag dropping our subpages from the serps. Im tempted to remove the tag and return to how the site used to be repeating products on subpages.. not in a seo way but to help visitors drill down to what they want quickly. Any comments would be welcome..0 -
Should I add my blog posts to my sitemap.txt file?
This seems like it should be an obvious no, just because of the amount of work that would entail, and then remembering to do it every time I make a post, but since I couldn't find anything on Google about it and have never heard anyone mention it, I figured I'd ask.
Technical SEO | | UnderRugSwept0 -
Question concerning a 302 Redirect
Hi! I've already done some research on redirects, but I still have a question concerning a 302 redirect implemented at the homepage of a website. The Website www.domainA.com has a 302 redirect to www.domainA.com/content/.... Also all subsequent pages have the /content/ directory in their URLs: e.g domainA.com/content/products First thing I was wondering about, was the use of a redirect to a new site using an additional directory /content/... Why would anyone do this? Would it be enough to replace the 302 with a 301 redirect, or would you recommend to change the entire structure and eliminate this /content/ directory? The most logical structure would be www.domainA.com/products/.., and not www.domainA.com/content/products, right? Second thing: Given that 302 means temporary redirect, what are the actual implications when redirecting from domainA.com to domainA.com/content? I've heard that 302 redirects don't pass linkjuice and are detrimental for the site's rankings... What are the actual implications concerning the example above (302 redirect from domainA.com to domainA.com/content ? Would be great to get some advice about the first problem and maybe some insights about the second one concerning 302s in general. Thanks in advance! Cheers, Chris
Technical SEO | | adwordize0