Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
-
Site URL is https but keep getting and error that homepage (http://www.flogas.co.uk/) can not be accessed. I set up a second campaign to alter the target url to the newer https://www.flogas.co.uk/ version but still getting the same error! What can I do? I want to use Moz for everything rather than continuing to use a separate auditing tool!
-
It will probably help if you delete the following line from your robots.txt file:
User-Agent: rogerbot
Disallow: /Dirk
PS You also block /js/ for all bots - Google doesn't really recommend that
-
I just check and see that http is 301 to https. But https works.
Can you mail this issue to [email protected]?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What does the external links column mean in the crawl report , thanks
Hi, Ran a report for www.dare2b.com report, and it showing 34780 external links. What does this mean Thanks Jeff
Moz Bar | | jefffox0 -
Cannot crawl website with redirect intalled on subdomain url
Hi! I want to crawl this website : http://www.car-moderne.ch. I tried a got back the crawl just for that one url (not for all the pages of the website). This single line cvs says that the status of the http://www.car-moderne.ch is 200, but in fact it is a redirect 301 to http://www.car-moderne.ch/fr where the live home page is (actually the Moz bar sees the 301, not the 200 as the single-lined crawl does). How can I proceed in this case (a 301 redirect being installed on the subdomain url) to still be able to have a full-fledged juicy cvs with all the broken links, duplicate content, etc. Thank you for your help! Pascal Hämmerli
Moz Bar | | Ethos_Digital0 -
I'm getting, "you're not using the rel="canonical" META attribute" in my crawl diagnotic
I'm running a campaign crawler through Moz on this particular page: http://www.henley.ac.uk/executive-education/leadership-and-management-programmes/ but I'm getting a notifcaiton from Moz saying, "you're not using the rel="canonical" META attribute" I don't understand what this means!! Has anyone else had this problem, or can they help me understand what this means and how to fix it? Oh, and Happy Thanksgiving from the UK! Virginia
Moz Bar | | blackboxideas0 -
I got a 404 in the Crawl Test Tool Report
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens? <colgroup><col width="1535"></colgroup>
Moz Bar | | IMforYou
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page | Best regards, Jos0 -
Why does it show (in Moz) that we don't have any meta descriptions on our site when we do?
We have over 1,330 pages that say we are "Missing Meta Description Tag" when I spot check all of them have meta descriptions? Can you please explain to me why Moz is picking up that we do not have meta descriptions when we do. Our website http://www.betheboss.ca Please help. I would like a accurate measure of meta descriptions that are missing.
Moz Bar | | BeTheBoss0 -
No Keyword in URL, but it is there
Hi,friends. I have grade A on all pages but system shows like no keyword in url, but it is there. for example i have Latvian keyword: bīstamo kravu pārvadājumi (with Latvian characters like Ū, Ā) I have url with exact keyword in it: http://vervo.lv/lv/kravu-parvadajumi/transports/bistamo-kravu-parvadajumi So why moz dont see that keyword, it does not understand special characters?
Moz Bar | | Liva0 -
Will the videos for the 2013 Moz Con be available for purchase in a bundle like they were in 2012? If so when? Thanks!
Will the videos for the 2013 Moz Con be available for purchase in a bundle like they were in 2012? If so when and how much? I would like to purchase. Thanks!
Moz Bar | | larahill0 -
Dupe content report showing in 'Errors' section when surely should be in 'Warnings' section ?
Why is the dupe content info showing in errors and not warnings ? Since if dupe content can get your site penalised (as per Panda) or worse banned, surely it should be in that section of reports ? Cheers
Moz Bar | | Dan-Lawrence
Dan0