Why did Moz crawl our development site?
-
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues.
What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further.
How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again?
Thanks!
-
@multitimemachine a noindex tag only really applied to Bing/Google other crawlers etc.. You said you blocked (via wildcard) all robots, are you sure you've not gotten e.g. meta robots that might be different?
[email protected] might be your best bet for a quick resolution for 'cleaning' the report though I'm still slightly lost as to how your main domain and dev/staging were confused as normally there is a subdomain in the way from my experience, even stranger as bots can't by-pass passwords unless it's your sitemap.xml?sorry I can't get you a direct response but without seeing the site or similar it's hard to diagnose though I'm sure the team at Moz can point you in the right direction .
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro Tools
Hello I ran into a error while using Moz Pro Tools Crawl Site feature. Stating that my wordpress website could not be crawled. When contacting moz they sent me this screenshot stating the reason for this error is because of the odd ip address highlighted in yellow. Only time I've seen this particular ip-address is during local development. If anyone has any advice on how to fix this or what may have caused this issue. I feel this maybe effecting the site's overall search visibility. ednqnL7
Moz Pro | | willakawillow220 -
How to quit moz 30days free?
dear moz: my english is so poor!!haha!! my user id si heroiamhero,my email is [email protected]. 2weeks ago i made a free on moz.com but i found it is not very useful for me ,so i want to quit free use ,please tell me how to quit moz 30days free,thankyou very much.
Moz Pro | | heroiamhero1 -
Twitter and G+ MOZ not tracking
Our company site TW and G+ accounts are not being tracked by MOZ although they are connected with MOZ. Can someone please look into this?
Moz Pro | | US-Chemical-Storage0 -
White Label Version of Moz Pro?
I may have missed it, but is there a white label option for Moz Pro? I have a few clients that would like to login and have a look at competition, rankings, etc. Perhaps something like WebCEO or MySEOtool?
Moz Pro | | TysonDowns0 -
Campaign Crawl
I have a site with 8036 pages in my sitemap index. But the MozBot only Crawled 2169 pages. It's been several months and each week it crawls roughly the same number of pages. Any idea why I'm not getting fully crawled?
Moz Pro | | JMFieldMarketing0 -
So many problems with my site.
Hi all.I was shocked when when i run a campaign and the warnings and recommendations about my site are so many.I know nothing about web design and the person who design it is asking me what are these problems and where did i get all these? any solution this are the problems 1.5XX (Server Error)
Moz Pro | | jubba
2.Duplicate Page Content(875)
3.Duplicate Page Title(875)
4.Overly-Dynamic URL(1048)
5.Too Many On-Page Links(60) and this is just a few of the problems.0 -
Site metric tool
Is there a tool on here (or anywhere else) where you can upload a list of sites (say 500 sites) and be given things like mozrank on it?
Moz Pro | | thefresh0 -
How often is Open Site Explorer updated?
Or perhaps my question is actually how often does the seomoz bot crawl the web? I ran some benchmark link counts for a number of websites about this time last month and I noticed when I went to update them for this month the data is the same. Thanks.
Moz Pro | | WCR1