Rogerbot did not crawl my site ! What might be the problem?
-
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore.
Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems.
What might be the problem?
thanks for any tips!
Holger -
I am running into this problem as well. Does anyone have an answer. I was a customer a year ago and had no problems. Stopped for a while and then came back, my site was never able to be crawled again. A few months later I did make a few changes on my site, and just tried again today, but it only gets 1 page. Once before they told it had to do with something like http 1.0/1.1 which makes no sense. Not sure why google, bing and the rest can crawl the site, but this service can't. Hopefully someone has an answer. I looked at that link that is mentioned and I do not feel that any of those are causing the issues.
-
Hi there,
Thanks for reaching out! I went ahead and started a ticket for you to do some digging. I will run a few tests on my end to see where it is causing the issue. I will get in touch you ASAP
Thanks again for reaching, hope I could talk to you soon!
Best,
-
Thanks Keri,
But its not about these common problems, because Rogerbot crawled the properly before. May be a ticked will help to find the cause.
Holger
-
Hi! Here are some common reasons why only one page is crawled. If this doesn't solve it, opening a ticket with the help desk would be the next step.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why did Moz crawl our development site?
In our Moz Pro account we have one campaign set up to track our main domain. This week Moz threw up around 400 new crawl errors, 99% of which were meta noindex issues. What happened was that somehow Moz found the development/staging site and decided to crawl that. I have no idea how it was able to do this - the robots.txt is set to disallow all and there is password protection on the site. It looks like Moz ignored the robots.txt, but I still don't have any idea how it was able to do a crawl - it should have received a 401 Forbidden and not gone any further. How do I a) clean this up without going through and manually ignoring each issue, and b) stop this from happening again? Thanks!
Moz Pro | | MultiTimeMachine0 -
Crawl Diagnostics 403 on home page...
In the crawl diagnostics it says oursite.com/ has a 403. doesn't say what's causing it but mentions no robots.txt. There is a robots.txt and I see no problems. How can I find out more information about this error?
Moz Pro | | martJ0 -
Open Site Explorer and link numbers
I know this question has been asked many times in this forum but I still can't work it out. Why does this link: http://www.opensiteexplorer.org/links?page=1&site=www.bookpal.com.au&sort=page_authority&filter=&source=external&target=subdomain&group=0 Which is showing all links, external, to pages "on this sub domain" show 1,935 external links but this link: http://www.opensiteexplorer.org/links?page=1&site=www.bookpal.com.au&sort=page_authority&filter=follow&source=external&target=subdomain&group=0 which is exactly the same but this time shoing followed + 301 links, says "showing 1 - 50 external links) but won't show the total links (and I know the mouse-over on the question mark says it's won't show the total links, but I don't understand why it can't show the total links when it could show the total links when I requested to see "all links" instead of just "followed+301" links.) but it actually lists 700 links (14 pages, 50 results each page). I know the link list is limited to 25 links per domain but then it means you can NEVER know the total link count unless you download the full report. This makes using OSE to know numbers of links (internal, external, or otherwise) impossible. And if anyone uses the API, why the API (external+follow) returns 1,451 links? I'm sure it's an ongoing issue with people trying to get their head around all of this and I've never really been able to. Any insight would be much appreciated!
Moz Pro | | eatyourveggies0 -
How long is a full crawl?
It's been now over 3 days that the dashboard for one of our campaigns shows "Next Crawl in Progress!". I am not complaining about the length... but I have to agree that SEOMoz is quite addictive, and it's quite frustrating to see that everyday 🙂 Thanks
Moz Pro | | jgenesto0 -
Where is the best place to add links on my site?
If I'd like to put links to other sites in my site, is it better to have a page named "Our Helpful Links" etc. instead of just adding them to the bottom of an existing page like I've seen on some sites? I'm asking because I'm wanting to make Google as happy as possible and still add them. Just in case it helps to look at the site yourself to give advise its; http://www.allstatetransmission.net If you see anything else there that I should work on feel free to be hard on it, I value any criticism. Thanks, Jeff
Moz Pro | | allstatetransmission0 -
How long should the weekly crawl take
Mine started yesterday afternoon and it's now almost 11pm on Sunday. 30+ hours and still not finished (and no progress indicator). 438 pages quoted as being crawled. That's not normal - right? I have made a bunch of changes based on last weeks crawl so I have been eagerly waiting for this to finish But 30 hours?.... Thanks. Mark
Moz Pro | | MarkWill0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0 -
Crawl Issues
My website - qtmoving.com - has 26 articles and when the SEOmoz did a crawl it only found 13 articles. Can someone please give me some insight as to why not all pages are being crawled.
Moz Pro | | CohesiveMarketing0