Is it possible to block Moz from crawling sites?
-
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking.
I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....!
So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious.
-
Thank you both for the responses... I guess it was just a matter of time in my case. It looks like the keywords crawl/report comes in a few hours "after" the initial crawl email from Moz... not sure why, but about 15 minutes after I posted this question, I received the email titled "New Rankings and On-Page Reports Ready" that included the keyword rankings!
Thanks again
-
Here are the most common reasons why only one page is crawled. If this doesn't answer it, send us a note at [email protected]. Thanks!
-
Ok, here is the helpful resource regarding Roger Mozbot. http://moz.com/help/pro/rogerbot-crawler
This might help!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Not Cached By Google
Hi My name is Apoorwa and i have my own website, My website is not cached by google, Why is this problem happening with my site.can somebody help me please? its urgent....this is my site - http://www.holifestival.org/Please assist me.......
Moz Pro | | Packersmove0 -
How to Learn a Lot from Moz Q&A
You can learn a lot from Moz Q&A without asking any questions. The way to do that is to watch for topics that you are uncertain about, then subscribe to that question by clicking the "Email Updates" button. Then, when people reply to that topic, you will be able to read what they have to say and will likely learn many things (and see a few entertaining arguments 🙂 Give it a try. You might subscribe to more questions than you think and learn many things that will improve your skills and expertise.
Moz Pro | | EGOL8 -
How do I find out which pages are being indexed on my site and which are not?
Hi, I doing my first technical audit on my site. I am learning how to do an audit as i go and am a lost. I know some page won't be indexed but how do I: 1. Check the site for all pages, both indexed and not indexed 2. Run a report to show indexed pages only (i am presuming i can do this via screaming Frog or webmaster tool) 3. I can do a comparison between the two list and work out which pages are not being indexed. I'll then need to figure out way. I'll cross this bridge once i get to it Thanks Ben
Moz Pro | | benjmoz0 -
Why is my MOZ report only crawling 1 page?
just got this weeks MOZ report and it states that it have only crawled: Pages Crawled: 1 | Limit: 10,000 it was over 1000 a couple of weeks ago, we have moved servers recently but is there anything i have done wrong here? indigocarhire.co.uk thanks
Moz Pro | | RGOnline0 -
I am trying to find inbound links for one of my site urls. My question is does SEOMoz able to track all internal links as the Open Site Explorer shows 0 internal links?
It shows 0 internal links when I am pretty sure we have multiple internal links.Should we use absolute urls or relative urls for internal links?
Moz Pro | | SulekhaUSLLC0 -
Why is my crawl STILL in progress?
I'm a bit new here, but we've had a few crawls done already. They are always finished by Wednesday night. Our website is not large (by any means), but the crawl still says it's in progress now 3 days later. What's the deal here?!?
Moz Pro | | Kibin0 -
Is it possible to change the order of my campaigns?
I've just upgraded from a PRO to a PRO PLUS account (as if I need more caffine!) and wondered if it were possible to change the order of the campaigns? They seem to be in alphabetical order - which is no problem, but I'm not actually running all the campaigns - I'm only monitoring a few potential sites, so don't want to see them at the top? it's only a niggle, not a gripe, but wondered it it were possible - or if not, can I show more than 5 campaigns at once rather than scrolling. As always, thanks in advance. G.
Moz Pro | | Gordon_Hall0 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750