Amount of Pages Crawled Dropped Significantly

[jeffmace]

I am just wondering if something changed with the SEOMoz  crawler.  I was always getting 10,000 or near 10,000 pages crawled.  After the last two  crawls I am ending up around 2500 pages.  Has anything changed that I would need to look at it see if I am blocking the crawler or something else?

[KeriMorgret]

Hi! This is probably a question for the help desk team. If you send them an email at [email protected] and let them know your campaign name, they'll take a look at it and help you figure out what happened.

[jeffmace]

i also want to add that I ran a test crawl and it actually returned 3200 pages, so I am not sure why the site is only doing 2504 for the last two weeks.

[jeffmace]

i did a site update about 6 weeks ago, but was crawled a few times with about 9,000-10,000 pages crawled. It was the last two crawls that it dropped to 2504 pages.  I don't use .htaccess, nothing in robots.txt so I am not sure what could be doing it

[OptimizeSmart]

you may  be blocking the mozbot. Do you remember the exact date when it started happening? If yes then you can identify and revert the changes. Also check your robots.txt and .htaccess file for any abnormal changes during this time period.