How do I disallow crawl on a directory when it's a prefix to my site's URL?
-
I am trying to disallow our media repository (hosted elsewhere, but appears as a directory on our site) from being crawled by robots but it is not a subdirectory of the site, it's a prefix.
So I need to disallow: mediabank.mywebsite.org
Not: mysite.org/mediabank
What would I need to put in my robots.txt and/or the other host's robots.txt to make this happen?
Thanks!
-
Hey there! Tawny from Moz's Help Team here.
You'll want to add a robots.txt file for that subdomain, and then add a Disallow command to that robots.txt file. So, using your example, you'd want a file like mediabank.mywebsite.org/robots.txt that had a Disallow command for any robots you don't want crawling that subdomain.
For all user-agents, that would look something like this:
User-agent: *
Disallow: /That would stop any user-agents from crawling any pages on that subdomain.
I hope this helps! If you've still got questions, feel free to send us a note at [email protected] and we'll do our best to sort things out for you.
-
Hi,
Please check this old thread on the same topic @ https://mza.seotoolninja.com/community/q/block-an-entire-subdomain-with-robots-txt
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links between my two sites
I have a branded site (corporate site) that has links to my blog site. I am trying to locate all the links between the two. Most importantly from the BRANDED site to my BLOG. I've tried the inbound link but it shows I have none. What or how am I searching wrong?
Moz Bar | | Bwaples0 -
Learn how to use MozBar to analyze competitors' schema markup. Get you Daily SEO Fix!
Schema markup helps Google and Bing identify what your (and your competitors') website pages are all about and as a result, helps search engines better index them. Better indexing can lead to better rankings for relevant keywords and phrases and herein lies the opportunity. In today's Daily SEO FiX, Jordan shows you how to use MozBar to analyze the schema markup of the competition and optimize your own site and pages for rich snippets. If you don't have MozBar, no worries! You may download it for free here. This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Moz Bar | | kellyjcoop1 -
URL not returning a page successfully
I am trying to use the on page grader however it is not working with my website. The url is: http://www.britishhardwoods.co.uk/ Can anyone help?
Moz Bar | | British_Hardwoods0 -
Why Moz Rank tracker's results are different from results search in browser?
I would like to perform a competitor rank analysis in Google.gr . I have noticed that the results from the rank tracker for Google.gr are different than the results in Google.gr from a machine located in Greece with no Google account logged in. Could you inform me why such a behavior occurs?
Moz Bar | | thmavri0 -
408 errors in crawl diagnostics
Best community, The Crawl Diagnostics Report of Moz gave our website a lot of 408 errors like below: <dl> <dt>Title</dt> <dd>408 : Error</dd> <dt>Meta Description</dt> <dd>408 Request Time-out</dd> <dt>Meta Robots</dt> <dd>Not present/empty</dd> <dt>Meta Refresh</dt> <dd>Not present/empty</dd> <dd>-----------------------------------------------------------------------</dd> <dd>The report has diagnosed a lot of these (around 320), even though we cannot reproduce the error (we cannot seem to find it ourself). </dd> <dd>2 questions relating to this: </dd> <dd>* Can you (the people of Moz) reproduce the errors manually? </dd> <dd>* Is it possible that it is a bug in the spider of Moz itself (too many spiders crawling at the same time)?</dd> </dl>
Moz Bar | | arjen.koedam0 -
Site Crawler Tool by the Company Formerly Known As SEOMoz
Moz had a tool I used that would crawl my site and send me a report of all pages, all errors, 301s 404s 505s, and a whole plethora of stuff. I used it to fix pesky errors quite a bit. Does this still exist? Was it replaced or am I just not finding it in the new design?
Moz Bar | | KJ-Rodgers0 -
Moz not crawling opencart product pages
hi, i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products. any idea of what can be wrong? im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1). thanks
Moz Bar | | marlvass10 -
What happened to moz Crawl Test? Is it moved in the redesign?
Love the new design of moz.com! And the mentions tool is terrific. However, I use the Crawl test a ton and cannot find where to access it now.
Moz Bar | | jessential0