How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best SEO way to benefit from your competitor's shutting down?
Hi ! Our main competitor is shutting down (website will be offline) at the end of the month and we are negotiating with them to buy their domain name: the idea would be to take advantage of their good rankings in the SERPs to redirect traffic to our pages (we're planning to crawl their site or get their sitemap and redirect th category / product pages to ours). The question is: for how long this strategy will be useful: days / weeks / months? (= for how long their pages will continue to appear in the SERPs from the day we enable the 301 redirections to our site?) Thanks in advance for your help! And if you have better suggestions, we're up to hear them of course.. 😉 Cheers
Competitive Research | | Kuantokusta0 -
Why is this site ranking higher?!
I'm trying to figure out why www.lvimmigrationattorney.com would be ranking higher on several keywords than www.immigrationlawnv.com. It doesn't make sense to me because the latter site has a higher page authority and mozrank, as well as more followed links. The domain authority from the linking sites to each site are pretty close. I'm sure I'm missing something, but I can't put my finger on it. Any idea?
Competitive Research | | atstickel120 -
Feed sites replacing my pages in SERP
I have a problem! Sites like:
Competitive Research | | pycckuu
http://lynk.ly/
http://www.yousaytoo.com/
http://www.stumbleupon.com/
http://friendfeed.com/ Are replacing my posts even if i am searching for an exact title of the post. For example this search query:
Christmas Party Style Sets With Black Cocktail Dresses Original post here:
http://richclubgirl.com/christmas-party/christmas-party-style-sets-with-black-cocktail-dresses/ How do i fight this issue? I want to stay on these sites somehow! Thanks in advance!0 -
What is a desirable range for a keyword's difficulty rating?
Hi all! I am new to the world of SEO and just getting started. As I am doing keyword research I am consistently finding difficulty ratings in the 40%-50% range. Is this considered a "par" rating or should I be looking for "a longer tail" rating? Thanks
Competitive Research | | jclayton180 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
Tools to search google for sites?
Hi, I am sure I read it on here. But is there a link to a resource whereby I can search Google for sites that accept things like guest bloggers, offer do follow links, do not no follow comments etc... It used things like "search term" blogs, eg "guest bloggers" websute Anyone help me out? Cheers Will
Competitive Research | | YNWA0 -
Manta inbound link in OSE - competitor site yes, my site no, why?
Our site has had a link back to our site from Manta for a long time, but it's not appearing in our OSE results. One of our competitors similarly has an inbound link from Manta and that link DOES appear in their OSE results. I'm not certain whether that matters for us. Does it? But in any event, why would the one link appear but not the other? Thanks! Tim
Competitive Research | | tcolling0 -
IP Address Of Site
Hi there, really new to seo and loving being able to research everything on seo moz. I have a question about the ip address of my site: I use a company that sells templated real estate websites (I'm a realtor) and I'm noticing all of their websites show as having the same ip address. There are probably 100's of sites on their server targeting the same keywords as I am, would this affect my ranking at all? I'm just considering switching to a Wordpress site with a unique IP address, so I'm curious to find out how the seomoz members would answer this question. Cheers, Adrian
Competitive Research | | adrianaldous0