How to turn off automated site crawls
-
Hi there,
Is there a way to turn off the automated site crawl feature for an individual campaign?
Thanks
-
No problem!
Thanks for the suggestion!
I will pass that on as feedback
Eli
-
Hi Eli,
Thanks for your response.
I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.
Thanks
-
Hey!
Thanks for reaching out to us!
Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!
If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:
User-agent: rogerbot
Disallow: *I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help fixing a duplicate content issue for my website. The moz crawl is show OMG my website with https:// and https://www. But I have never used the url https:// so I don’t understand why moz is showing this
Moz is showing my url with two different starts. Https:// and then the one I use https://www. The problem is I don’t think I have ever used the url without the www. at the start. How do I fix this?
Moz Bar | | jdp_uk0 -
Moz Crawl - 804 : HTTPS (SSL) error encountered when requesting page.
Got an issue sending a Crawl Request to https://www.usernamebuddy.com/ " "804 : HTTPS (SSL) error encountered when requesting page." I have tried to recrawl several times now same issue keeps occurring. I cannot see an error when I access the site am I missing something, if so how can I diagnose the issue and sort the problem? I have reviewed the source and cannot use any http: resources.
Moz Bar | | GrouchyKids0 -
Why isn't the Moz bar data populating for Yahoo sites?
The Moz bar isn't populating information for Yahoo homepage or it's verticals (i.e. homes, autos, finance, etc.), but I can get this data for other portals like AOL or MSN. I'm specifically looking for PA, mR, and DA information, but instead I get a generic "Search Profile" bar with no page/site-specific data.
Moz Bar | | AllieBell
Is there a reason Open Site Explorer data isn't populating for this particular portal?0 -
Why is 410 (Gone) being classed as a high priority issue in crawl diagnostics?
Are high priority issues have suddenly soared by over 100 because Moz is classing 410s as high priority.
Moz Bar | | Melissabraz
Google doesn't class these as so serious, so we were wondering if anyone knows why Mos does?0 -
Suggestion for Improving the Crawl Report on Canonicals
This came up in the answer to a question I gave here http://moz.com/community/q/canonicals-in-crawling-reports#reply_222623 Wanted to post here to put it in as a suggestion on how to improve the Moz Crawl reports Currently, the report shows FALSE if there is no canonical link on a page and TRUE if there is. IF you get a TRUE response, this shows up as a warning in your report. I currently use Canonical to Self on almost all my pages to help with some indexing issues. I currently use the EXACT function in excel to create a formula to see if my canonical link matches the URL of the page (as this is what I want it to do). I can then know that the canonical is implemented properly, or if I need to manually check pages to make sure the canonical that points to another page is correct. I would like to suggest that the Moz crawl tool does this. It can show FALSE is the canonical is missing, TRUE if the canonical is present and SELF if the canonical points to the URL of the page it is on. I think for the most part this would be much more actionable information. I would even suggest that TRUE would need to be more of a high priority alert, and SELF can't do any damage, so I would leave that info in the CSV but not have that as a warning in the web interface. Thanks for listening!
Moz Bar | | CleverPhD0 -
Crawl Diagnostics: Exlude known errors and others that have been detected by mistake? New moz analytics feature?
I'm curious if the new moz analytics will have the feature (filter) to exclude known errors from the crwal diagnostics. For example, the attached screenshot shows the URL as 404 Error, but it works fine: http://en.steag.com.br/references/owners-engineering-services-gas-treatment-ogx.php To maintain a better overview which errors can't be solved (so I just would like to mark them as "don't take this URL into account...") I will not try to fix them again next time. On the other hand I have hundreds of errors generated by forums or by the cms that I can not resolve on my own. Also these kind of crawl errors I would like to filter away and categorize like "errors to see later with a specialist". Will this come with the new moz analytics? Anyway is there a list that shows which new features will still be implemented? knPGBZA.png?1
Moz Bar | | inlinear0 -
Blocked Production Site from Search Engines - How to get it Crawled by Moz Crawler
I have an 'under development' site hosted, (which is an exact replica of live site as working on to add new functionalities & modules) - but its password protected, excluded from robots.txt (Disallow) & also marked noindex on all pages in the index - so that Googlebot & other Search Engines can not crawl the site At present the development work is almost 95% completed., Now - feel like to crawl the site through SEOMOZ Roger Bot - to know the errors and all indexed urls by Rogerbot. What's the best way to get Moz Bot crawl the site - but simultaneously continue it blocking its access to Search Engines I have gone through - https://support.google.com/webmasters/answer/93708?hl=en, it says a) Save it in a password-protected directory. Googlebot and other spiders won't be able to access the content- But this way Moz will also not be able to crawl the site b) Use a robots.txt to control access to files and directories on your server - However it also says - It's important to note that even if you use a robots.txt file to block spiders from crawling content on your site, Google could discover it in other ways and add it to our index. c) Use a noindex meta tag to prevent content from appearing in our search results - It also says that a link to the page can still appear in their search results. Because we have to crawl your page in order to see the noindex tag, there's a small chance that Googlebot won't see and respect the noindex meta tag Password Protected thus seems the best way to continue blocking. However, continuing with it will also block Moz bot to crawl the site. Any suggestions Thanks
Moz Bar | | Modi0 -
Moz not crawling opencart product pages
hi, i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products. any idea of what can be wrong? im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1). thanks
Moz Bar | | marlvass10