How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
Can anyone suggest a AHREF's alternative?
Hello Everyone,
Competitive Research | | Juratovic
Now that AHref's has killed their keyword analysis I am on the search for a substitute.I just can't see paying $80 per month for their link analysis only even-though I do find their link index to be much much much more updated then any other company even SeoMoz (sorry Rand and company). Does anyone have any suggestions? SemRush is the only one I have seen to come close. Thank you! Pete0 -
Another how the *%#^ is this site ranking question
I saw a question posted by someone a while back asking how a certain (in their opinion crappy) site was ranking in the top then. It happened that there were some good reasons for that site ranking. Well.... I have stumbled on a site that seems to be ranking for (almost) no reason at all: relatively low DA/PA very few inbound links (none seem to be that special) thin content The only thing I can think of, is that the site has the keyword in the domain name. But looking at the search results, there are other domains with exact match keyword in URL and somewhat stronger metrics that don't rank.
Competitive Research | | inhouseseo0 -
Working through the slides on my first seo project, I'm trying to determine the most popular content on competition sites.
From the video, I understand I should be looking for unique opportunities. But is there a baseline for product qualification, ie., a min. PA,LRD, Inbound LInks, social shares, etc. I guess I'm asking what I should be looking for in the data provided on each competing site? What determines THE most popular and unique product?
Competitive Research | | AhmadS0 -
Why do you think this site can't reach a better ranking?
Hello, My name is Pablo Lagos. We have a hosting company in Chile since 2006. All the time we have been ranked in the first page for the keyword "hosting" in google.cl. Some years ago, we used to be on the first 3 results. Since the last year our site went down to 7 and higher places no matter we do. When I run the link analysis, we have the highest punctuation in all items. I would really appreciate your comments. The site is in spanish: www.ilihost.cl Thanks.
Competitive Research | | pabloandres0 -
Sites With Duplicate Content Ranking Way Higher
The site I am writing about is easendtorontohomes.com I don't get it. I see that some of my competitors have more inbound links and have been around longer. Their SEOmoz trust etc is also higher. But I don't get how many could possibly be ranking higher since they're sites are all template based site via Web Tech Design and the content is duplicated from one to the other. The other weird thing is that they do everything wrong - they stuff their keywords tag, some of them use the same keywords/meta data on every single page, some of them don't even have meta tags, some of them don't even have a blog with ANY original content. Apart from listings EastEndTorontoHomes.com has TONS of original, well-researched, and keyword rich content. I just don't get it. Is it possible that google has some sort of relationship with that company? Or am I totally missing something? Some of these sites include: tinasmith.ca, mikeclarke.com, teamkassen.com
Competitive Research | | annasus0 -
How due I take the advanced report that i have exported as a excel file and recreate your online color coded version with the data.
In the keyword difficulty screen how do I recreate your beutiful color coded spreadsheet in the exported excel file based on the advanced reports.
Competitive Research | | bmcfarlane0 -
How does a site get to no 3 in Google with no KW in their links?!!
Hello everyone, my first post - ahhh I'm investigating a niche and there is a site that should have no right being there in my view. It's no. 3 Google UK for 'company formation' with a small site with 65 weak links from only 7 domains and hosted in the US. But more importantly, the Open Site Explorer says there is not 1 link with that term in its anchor text. This I find crazy and makes me suspicious. But before I go back to my client saying "oh they must be black hat" I would like your expert views. I'm not sure whether to tut or congratulate them and for the first time I'm not sure what reasons to give for their amazing performance! What's your views?
Competitive Research | | GOYMedia480