How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Many SEO tools donot allow me to check my site with https?
Hello All, Many seo tools I am using in that few tools do not allow me to test my site with https but it force me to check with http and then give report. So my query is do these tools give accurate report with http same like https? Thanks!
Competitive Research | | wright3350 -
How can I find out the exact match visitors for a competitor's website?
Hi, I'm trying to find a way of finding out the estimated number of exact match visitors that a competitor's website has got. I know that SEO agencies can find this out, and there must be a way of doing this for myself. Thanks
Competitive Research | | MThacker0 -
Why is this site ranking so well
I just took over an SEO campaign for a company we recently did a website for. The old company created individual landing pages for each city they service, but they didn't have any original content except replacing the city name. We are in the process of creating completely unique content per page. My main issue is that there is a company with no link profile, duplicate content on every page, no social media presence, etc - but consistently ranks for city + fence company The company we are starting to work with is: http://betterdecksandfences.com/ The company that is ranking for everything is: http://www.alcovyfence.com Any ideas?
Competitive Research | | BeardoCo0 -
How can stale, poor quality sites rank above fresh, high quality sites?
Dear Community, How is it that a Google search for temperature sensors shows the second result as site **A **(http://goo.gl/zRS1L) in a higher positionthan site B (http://goo.gl/3Xy2O) which is on the 2nd page of the SERPs. Site A blatantly does not follow Google's webmaster guidelines and recommendations for design, content or code. Site B scores an 'A' for on page analysis using SEOmoz, it also has a much higher number of root domains in and Moz rank, we have checked that site B has only quality inbound links, and anchor text is natural, it follows Google's guidelines for content and code diligently, and it is on an older domain. We are satisfied that site B has not been penalised by Google, e.g Penguin algorithm updates. We have also done a website verification with Google. Site B has fresher content compared to site A. Site B also has a sitemap yet site A doesn't. We're satisfied that we can rule out the following Algorithm Factors: Penguin algorithm updates Google Notified penalties Dodgy Inbound links Dodgy Anchor text Website verification with Google Fresh vs. static content Sitemap present Server Speed Server location Domain age and about 170 other algorithm factors Does anyone have any ideas as to why site A is ranking higher than site B in Google.co.uk? Thanks in advance. CD.
Competitive Research | | chichesterdesign0 -
Tools to search google for sites?
Hi, I am sure I read it on here. But is there a link to a resource whereby I can search Google for sites that accept things like guest bloggers, offer do follow links, do not no follow comments etc... It used things like "search term" blogs, eg "guest bloggers" websute Anyone help me out? Cheers Will
Competitive Research | | YNWA0 -
Crazy SEO question (maybe I'm missing something?)
OK - so one of our customers just called us and told us an interesting story: A local SEO company called her yesterday to try to sell their services to her. She's in the process of starting SEO services with us, so she told them she wasn't interested. The sales guy told her that they were better (without even asking who she was currently using) and asked her for a term that she'd like to rank higher for. She said she'd like to rank higher for "spray in bedliners northern ky" and he said "Gotcha, call you tomorrow" He called back just now and told her to look at Google. She's now ranking number one for that term. He didn't have access to her site, so he wasn't able to change anything on her site. He won't tell her what he did, and told her it was legitimate - but it seems to me that with only off-site tactics, it'd be nearly impossible to white-hat her site to number one overnight... Any ideas what he's doing? First of all, we want to be able to tell her what he's doing, because she's curious. More importantly, we want to be sure he's not doing anything black-hat that's going to hurt our client's site. Thanks for your help, Mozzers!
Competitive Research | | Greg_Gifford0 -
Keywords in URL structure for Large Site
We are creating a national real estate website which will contain 2mil+ listings. Our aim is to compete with the National leaders in the field. We are trying to lock down a url structure and it basically comes down to if we should put a major keyword in the url structure or not. The structure would have regional keywords naturally. Its the addition of a more descriptive key word which is in question. Our domain name has no keywords in it. For the sake of this example, we would be targeting "city real estate"... oursite.com/real-estate/state/city/ or oursite.com/state/city/ Here is what the big sites use as an example: http://www.zillow.com/homes/for_sale/New-York-NY/ http://www.trulia.com/NY/New_York/ http://www.trulia.com/real_estate/New_York-New_York/ http://www.realtor.com/realestateandhomes-search/New-York Id love to not have to throw keywords in there and be as clean as possible but i also dont want to shoot myself in the foot. The big boys do add additional keywords in the url, does that mean we should? Id much rather be a leader than a follower but again, I dont want to mess this up from the start and these guys have probably tested this (or have they?). Input would be greatly appreciated.
Competitive Research | | cobbsfriedman0 -
Twitter as a website's #2 ranked linked page?
A site I'm researching on open-site explorer has a #2 link with page authority of 52 and Domain authority of 97, and that link is the site's twitter page. No other sites I've researched have had their twitter page show up in it's link rankings like this, can someone explain?
Competitive Research | | TheSquareFoot0