Websites First Crawl - Over 2 Hour Suggested Wait
-
Hello SEOMoz!
We recently signed up for a free trial and on the pro dashboard it states the following.
"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."
It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.
Is this normal?
Thanks
-
Thanks!
Last time I checked the robots.txt was there. Anyway I have rcreated this.
Will try creating a new campaign and seeing how long it takes.
Update: worked fine after around 10 minutes crawl!
-
Hi Louis,
You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.
I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.
To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.
User-agent: *
Disallow:Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt
Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:
- After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
- Or wait until your next crawl
- Contact the help team at [email protected]
Hope this helps! Best of luck with your SEO.
-
Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.
I just started one on a 30 page site, so I expect it will show results tomorrow.
-
So this means I will not get any data under crawl diagnostics for a week?
Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.
-
The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Could my Crawl Error Report be wrong?
HI there, I am using Yoast SEO plugin on a wordpress website. I am currently showing 70 Med priority crawl errors 'missing meta description' on my Moz pro account. This number of missing meta descriptions has increased over the last 6 weeks. But every single page / post / tag / category has both the title and meta description completed via Yoast. I requested a google bot to crawl the site a few weeks ago as thought it perhaps wasn't been crawled and updated. Any idea what the issue might be? Could the Moz report be incorrect for some reason? Or could something be blocking Moz / Google from seeing the Yoast plugin?
Moz Pro | | skehoe0 -
"link_count" column in Crawl Diagnostics report
On the Crawl Diagnostics report, does "link_count" represent external (links to this URL), internal, both, or what ?
Moz Pro | | GlennFerrell0 -
Find a 4xx or 5xx link referenced in an SEO Crawl Report
So I just got the Crawl Diagnostics report for a client site and it came back with a number of 4xx errors and even 1 5xx error. So while I can find the URL that has the problem, I cannot find the pages that have the links pointing to these non-existent or problematic pages. Normally I would just search the database for the site, but in this case I don't have access to it as the site is on a proprietary platform with no access other than to the CMS. Is there anyway to get the linking URL from the report? Thanks!
Moz Pro | | farlandlee0 -
SEOMOZ Crawl Test
Guys I really have an issue that i know have but cannot see if that makes sense. Basically 3 months ago i did a site wide 301 from economyleasinguk.co.uk to www.economy-car-leasing.co.uk Every thing looks good get all the correct header responses , all canonicals work perfectly , Google webmaster tools is updated fetch as google bot shows the old site is 301 I tried the seomoz crawl test today on the old domain and got this message Oh no! Looks like the page you were trying to access is temporarily down which at first thought ok because the site was not there it wont do it on an old 301 domain, however i tried it on a domain i know has just been 301'd and i got this message The URL http://www.site1.com/ redirects to http://site2.com/. Do you want to crawl http://site2.com/ instead?
Moz Pro | | kellymandingo
Would you like to:
Continue with www.site1.com
Continue with site2.com I really do not know what to do, its either the redirect script is missing something however its doing what it should or the server is a problem but again its doing what it should so why would SEOMOZ not be able to crawl the old URL like it example site above. Now the strange thing is Open Site Explorer does see the 301 and asks if i want to check the new URL instead Ps the redirect is done using PHP redirect which i am asking him to change to a htaccess as its now on a apache server and was wondering if this could be an issue, all pages go to correct pages as requested Thanks in Advance1 -
How can i get seomoz to crawl a campaign on demand
hi how can i get seomoz to crawl a campaign on demand instead of on a weekly basis? For example i have corrected some error warnings and on page elements and would like it to re crawl the site sooner to see how the corrections have worked? thanks
Moz Pro | | Bristolweb0 -
Pages Crawled: 250 | Limit: 250
One of my campaigns says: Pages Crawled: 250 | Limit: 250 Is this because it's new and the limit will go up to 10,000 after the crawl is complete? I have a pro account, 4 other campaigns running and should be allowed 50,000 pages in total
Moz Pro | | MirandaP0 -
Started my first campaign - keyword rankings do not match
Hello, I started my first campaign on SEOmoz and my keword position do not match the results I receive by using google search or by using another SEO program. For example - Ingersoll Watches should be on position 6 on google.com - SEOmoz shows me Ingersoll Watches is on 16. Why does SEOmoz show different results? Thank you for your attention!
Moz Pro | | JasonMGreen0 -
Crawl complete, but nothing changed?
Hi everyone, According to my account, the crawl diagnostics were completed yesterday. However, the duplicate page titles that it mentions aren't correct. The changes that I implemented several days ago are not being shown in the report. When I click the duplicate page title links, the latest date in the graph is 3/26. However, it says the crawl was completed on 3/30. Does it take a few days for the reports to match what the crawl actually discovered?
Moz Pro | | beeneeb0