How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem on my site
Hello, thank you for guiding me
Competitive Research | | kaajshop 0
I had a problem with the speed of the site. I do not know how to solve this problem
this page:https://kaajshop.com/cat/700 -
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
Can anyone suggest a AHREF's alternative?
Hello Everyone,
Competitive Research | | Juratovic
Now that AHref's has killed their keyword analysis I am on the search for a substitute.I just can't see paying $80 per month for their link analysis only even-though I do find their link index to be much much much more updated then any other company even SeoMoz (sorry Rand and company). Does anyone have any suggestions? SemRush is the only one I have seen to come close. Thank you! Pete0 -
How did this little site rank #2?
Keyword: "adoption agency" Site: http://www.opendooradoption.org These guys are ranked #2 for this very competitive term in all US regions. They're beating Bethany (and my company) who are orders of magnitude larger in both company size and link profile. Their domain authority is 40, and they have less than 200 links pointing to them with "adoption agency" in the anchor according to OSE. Their on-site is pretty much nonexistent. Does anyone have any theories as to how they accomplished this?
Competitive Research | | AdoptionHelp0 -
How a site only with Frame can be at #1 google in a competitive keyword?
Guys, I need help. I have a company in Brasil and due to lack of budget, I'm doing the SEO for my self. And for two years Im reading everything that I can to do my best in my website. I worked very hard, but the #1 in the most important keyword for my company is a very bad website only with frames. What they are doing? Could anyone give help? Keyword: "acampamento"
Competitive Research | | Naghirniac
Google Brazil (www.google.com.br)
#1 website: www.acampar.net My website: www.acampamentoaguiasdaserra.com.br Thank you in advance Miguel0 -
Is SEM rush accurate enough to validate it's price?
I have been doing manual keyword research for some time now, but I am looking for something that can dig deep. I used to use the SEMrush API when i was a raven tools subscriber and it was great for identifying organic rankings for domains. Has anyone had a good experience with this software strictly for keyword research?
Competitive Research | | JordanGreve0 -
In Open Site Explorer, what does it mean when a linking page does not contain any reference to the URl entered?
When running Open Site Explorer on a particular URL, I get a list of linking pages. Many of these pages have a high Page Authority. I am assuming that this is a list of pages that presumably link to the URL I entered. First, is this correct? Next, when I click on an entry in the list I don't see any reference to the URL on the page, even viewing the page source. What does this mean and why is the link in the list?
Competitive Research | | jkenyon1