How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html) -
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
I need Ecommerce Site who have implemented J-Son Schema for Blog
Hi All, Just I need few sites who have implemented J-son schema on there blog. Looking for ecommerce blog. Hope you guys share with me good site. Thanks in advance.
Competitive Research | | amu1232 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
Keyword Search - another site bypassed me and they are not ranked well...
I was trying to optimize the keywords "music for kids" . This is a no brainer because my URL is "musicforkids" My On-Page Report Card for "music for kids" gives my site a "A". The on page report card for the other site gets an "F". But the other site comes up number ONE in the organic Google search and my site comes up number three. The two sites above me receive a crappy on page Keyword report for the key words "music for kids" but they still come up above me for that search term. What's the deal?
Competitive Research | | musicforkids1 -
Keyword based link problem on site
So I think I might have identified an issue with a site that I'm trying to get ranked for a specific keyword but, wanted to get some opinions before I started making some big changes on the site. On my homepage I have the keyword that I would like to be ranked for in the title lets say "Blue Widgets - Company Name', also on the home page I have some descriptions of our services including the keywords. I also have a couple of the keyword based links within in the content, navigation and footer. But these keyword based links all point to another page on the site: blue-widgets.htm. If I really want my home page to rank for the keyword "Blue Widgets' should all of these links point to the home page instead of the sub page? I know there are a great number of other factors that contribute to rankings but looking at my competition, this is something that they seem to be doing. The keyword based links within the content, navigation or footer all point to the homepage. I also have a higher Domain Authority than some of the sites that rank higher than me so I'm not sure if building more links is the answer. Of course I always want to build natural links but these sites don't seem to be doing that either. Any comments, suggestions or input would be greatly appreciated.
Competitive Research | | TRICORSystems0 -
How can I estimate a domain's overall organic search traffic - any tools?
Most of my analysis revolves around looking at rankings for specific keyword phrases that I've identified as important/relevant. But it'd be nice to be able to look at a domain and get a sense for how much organic traffic they get overall. If they're not ranking for the keywords I'm researching but have a lot of organic traffic that would be a nice signal to me that they are probably targeting other phrases more or have a big brand presence or something. Any suggestions? Thanks! Jeff Gibson
Competitive Research | | jeff.gibson0 -
Would buying the domain 'howtoget-a-girlfriend.com' help organically
for the exact search 'how to get a girlfriend' in google? How-to-get-a-girlfriend.com and howtogetagirlfriend.com are taken, so I just wanted to see if this was close enough to help google.
Competitive Research | | insitegoogle1 -
Can't Grasp Why Pages rank Higher?
The first result "Bankruptcy on IRS" is the search term. Why does the first url rank higher in google. The second one, the IRS.gov page beats them in PA, DA root domains links. The title meta has bankruptcy near the front. unclefed does have the IRS keyword in the title, but an I missing something here? What are the other factors, that are most obvious. Sure one can have bad links, and other negative criteria, but these are pretty decent sites that probably don't engage in much in seo, let alone bad SEO. Sure link text and mix of links can help, but am I missing something here? Actually what I think I really need IS A CHECKLIST OF WHAT TO CHECK IN WHAT ORDER WHEN COMPARING WHY ONE PAGE RANKS BETTER THAN ANOTHER. Appreciate all discussions. Thanks in advance. http://www.unclefed.com/AuthorsRow/Daily/Fwdcsea.html http://www.irs.gov/businesses/small/article/0,,id=98701,00.html
Competitive Research | | joemas990