Why wont rogerbot crawl my page?
-
How can I find out why rogerbot won't crawl an individual page I give it to crawl for page-grader? Google, bing, yahoo all crawl pages just fine, but I put in one of the internal pages fo page-grader to check for keywords and it gave me an F -- it isn't crawling the page because the keyword IS in the title and it says it isn't. How do I diagnose the problem?
-
Very glad to see you got it working!
You can mark the question as answered to let others know it is fixed.
-
Thanks. The robots.txt file was the problem. It originally (yesterday) excluded rogerbot (by default) and then I remembered that and put it in as rogerbot but that didn't work. So I changed it to RogerBot and that didn't work. Today I removed the robots.txt file completely and it worked. Then I put it back with rogerbot and it is working.
It APPEARS that maybe it read the robots.txt yesterday before i put in rogerbot and for some reason didn't read it after I put it in. Will never know but it is now working.
Thanks for the help!
-
I know in robots.txt any URL's are case sensitive, I am not sure about user agents (bots/crawlers) but you do have RogerBot spelled with a capitol "B", changing it to lower case (Rogerbot) may fix the issue.
Another thing to test would be to simply remove the mass exclusion just to see if Rogerbot somehow is being blocked by it. Let me know how it goes.
User-agent: * Disallow: /
-
Hi sure, thanks. This page shouldn't have a speed issue but maybe you can see what the issue is:
www.qjamba.com/local-coupons/wentzville/mo/all
Thanks.
-
Hi Theodore,
Last time I looked at this issue for another community member they had a site that had huge images and slow script. This decreased the load time of the page and Roger just got frustrated. Rogerbot is not as sophisticated as the huge Search Engines crawlers and can easily be put off.
As Martijn asked, for us to help we really would have to look at the site to pick out possible issues.
-
Hi Theodore, could you share the specific URL with us so we could help you diagnose what the issue could be?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | | MyFamilClubLtd1 -
Adding keywords to landing pages
Hey Forum, I have a quick question. So I have added in my keywords to track etc and want to then add the keywords to a specific landing page how do i do that? Is it manage and track and add the keywords and URL there then they appear on the landing pages? Thank you,
Getting Started | | TAT1001 -
Crawl rate
How often does Moz crawl my website ? (I have a number of issues I believe I have fixed, and wondered if there was a manual request to re-crawl ?) Thanks. Austin.
Getting Started | | FuelDump0 -
Where are all of my on page links coming from?
I am trying to reorganize my site and with that my home page. I want to follow the 100 internal link rule. However when I run a Moz crawl test it says that there is too many on page links. I than went to Moz open site explorer and it said there is 237 links. When I exclude in bound links that still leaves me in the mid 100 range. After counting them my self I can only come up with 75 on page links. Does anyone know how I can find what Moz is referring to? And how I would be able to find where this links are? I might be missing something simple or maybe I don't understand exactly how the Moz tools work. That is why input from this great community would help. Thanks!
Getting Started | | LovingatYourBest0 -
So the page-grader is giving my site an A, but it is ranking below some websites that the grader gives an F to. What is the point of the page grader?
Basically, I am new to this..very new. In fact, my field was neuroscience, and now I work in marketing.. I just started using seomoz, and for one website that I build. I followed all of the guidelines moz has to offer. On the grade section, I got an for a specific keyword. However, when I rate my site for that keyword, it does not even rank in the top 50. The keyword is even in the domain! Also, some sites that rank in the top 25 have an F rating for the same keyword. Why do they rank higher?
Getting Started | | Meier0 -
How to make site pages appear higher than homepage
Hi, our site sells a mixture of clothes, for example jackets, hats, scarfs and gloves. when somebody searches for 'hats in Chicago' our main website would appear. How can we make it so that our webpage with our hats appears? Thanks
Getting Started | | danieldunn100 -
Campaign.crawl-seed.bad-response ???
Hello Guys I have just tried to set-up a new campaign for this site http://www.emsababies.co.uk/ HoweverI keep getting this error ?? campaign.crawl-seed.bad-response Anyone know what I am doing wrong ?? Cheers James
Getting Started | | BlueNinja1