How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Non ecommerce site with high DA, being outranked by lower DA sites with ecommerce - help?
I have a website that we look after, up until January this year the rankings were really strong - the site has a much higher domain authority than the competitors, much more links and much more relevant content (all measured with moz's tools). About 6 months ago the rankings started to dive for national search terms. Now one thing ive noticed is that nearly all of the other sites in the search sell products online where we have no such function - could this be correlated - has anyone else experienced this?
Competitive Research | | rhysc1 -
Blocking Competitive Analysis Tools in robots.txt.... Worth it?
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all. My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
Competitive Research | | davidwaring0 -
Why Won't Google Update My Title?
I have waited plenty of time. Google has cached several pages with the updated title placed in the <title>tags.<br /><br />However search results, continue to show otherwise.<br /><br />I read the following (https://support.google.com/webmasters/answer/35624?hl=en)</p> <p style="padding-left: 30px;"><em>If we’ve detected that a particular result has one of the above issues with its title, we may try to generate an improved title from anchors, on-page text, or other sources. However, sometimes even pages with well-formulated, concise, descriptive titles will end up with different titles in our search results to better indicate their relevance to the query. There’s a simple reason for this: the title tag as specified by a webmaster is limited to being static, fixed regardless of the query. Once we know the user’s query, we can often find alternative text from a page that better explains why that result is relevant. Using this alternative text as a title helps the user, and it also can help your site. Users are scanning for their query terms or other signs of relevance in the results, and a title that is tailored for the query can increase the chances that they will click through.</em></p> <p>The reason I want to change my title, is because there seems to be a relevancy issue (as pointed out my other community members here.) Google is having trouble recognizing understanding what our site is about.<br /><br />So instead of a title that reads, "Felix And Fingers: Dueling Pianos" (as Google continues to use) I prefer "Dueling Pianos - Felix And Fingers"<em> </em> I don't believe Google is recognizing us correctly as a dueling piano company.<em><br /></em>Google doesn't seem to like that. Any idea why or how I might go about getting this updated?</p></title>
Competitive Research | | osaka730 -
Best way to find the highest ranking pages for a message board?
What is the best tool or method to find the highest ranking posts and content on a message board or forum? What is the best tool or method for finding the highest ranking keywords for a message board or forum?
Competitive Research | | Hondaspeder0 -
Can an https: impact (not provided) keywords in Google Analytics? Meaning if I make my site https: will it provide any more keywords. If not how can I get this data?
I have a site that I'd love to have more keyword data on. Can making my site an https: site give me more keyword data than leaving my site an http: site? Thanks!
Competitive Research | | mattdinbrooklyn0 -
Why do some sites perform great but show no data on the tool bar?
Hey everyone, Been a paid member for 3 days and this is my first time posting 🙂 I have a quick question about competitive analysis with the MozBar. I do not want to point out the exact website (unless that is something cool to do on this forum), but there is a site ranking for a single keyword in the local 7 pack in slot one, which is also slot one for the SERP entirely. When I go to look at the page in more detail, the MozBar is showing no DA, PA, anything at all, and the Open Site Explorer is showing no links internal or external. I know that the page has to have at least some of these elements populated in order to be ranking for a term this competitive, so why is the bar not showing anything? The bar works GREAT on other sites below this one. Thanks for any help!
Competitive Research | | tonyflorespsa0 -
Tool to compare which links competition has but i don't?
Is there a tool that I can use that will show me a list of my competition's links that I don't have?
Competitive Research | | ballhogjoni0 -
Tool for finding what keywords a competitor ranks for?
Does anyone know of any good tools that display what keywords a competitor ranks for? I have many competitors that I know get a lot of traffic, but I'm not entirely sure where the traffic comes from so it would be nice to plug in their url and get a general overview of what keywords they rank for and what positions.
Competitive Research | | shawn810