Unsolved Crawling error emails
-
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
-
@Mughalzadahas1 did you able to solve th issue
-
@DTashjian said in Crawling error emails:
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
@DTashjian said in Crawling error emails:
Recently we start having random error messages about crawling issue:
2024-08-30 edweek:Ok
2024-08-29 marketbrief:Err. advertise: Err, edweek:Err, topschooljobs:Ok
2024-08-23 edweek:Ok
2024-08-22 marketbrief:Err. advertise: Err, edweek:Err
2024-08-21 topschooljobs:Ok, edweek:Ok
2024-08-15 marketbrief:Ok. advertise:OK
2024-08-13 edweek:Ok
2024-08-12 marketbrief:Ok
2024-08-08 marketbrief:Ok, advertise:Ok
2024-08-03 edweek:Ok, topschooljobs:Ok
All for 2024-07 - are OkYesterday I set 2 more crawls for the same sites (edweek and marketbrief) and I get a morning email about original edweek site is ok (still have some problem but crawl occurs and all is fine) but for test crawl for the same site "EW Test" I just got error email.
Also I suppressed ALL email communications and frankly surprised by this email.Can you please check what is wrong with a crawler or stat collection or I don't know who produced the issues.
Hi,
I had a similar issue with my site and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance!
-
@DTashjian
I had a similar issue with my site and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance! -
Hi DTashjian,
I had a similar issue with my [site](link url) and resolved it by first reviewing the specific error messages to understand the problem. I then verified that the new crawl configurations were correct and matched the settings of the working ones. Ensuring that the test environment was properly set up and permissions were correctly configured was also crucial. Additionally, I compared logs to identify any discrepancies and made sure the crawler software and libraries were up to date. I suggest trying these steps to address your issue. Let me know if you need further assistance!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not working
Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh0 -
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
Haven't received an update on site crawl issues more than a week
Hello, my account has be scheduled to have the next updated report on 1st March. However, up till now, the latest data I have for our site crawl issues is made on 21st Feb. May I know if there is any issue related to this? Any way that i can draw the data for this week?
Product Support | | Robylin10 -
Are these MA dashboard issues errors or ccorrect ?
Hi As you know i had spotted some bugs with the new MA dash and now on a more recent report update for a project see the below messages so pls confirm if they are bugs or genuine so i know to inform clients dev to attend to or not ? (PM me for the client/project name if you need it): Crawl Error Moz was unable to crawl your site during the last campaign update!We encountered the following issue:904 Response from Server Timed out Without Completing Page Request Crawl Issue Found: 500 Errors More than 5% of site pages served 500 errors during the last crawl
Product Support | | Dan-Lawrence0 -
Adding 2nd billing email address not working
Hello, I'd like to add a second email address for billing, when I click the 'click to add' address button I am given a pop-up and can enter the address. But, when I click submit the page does nothing. Anyone else had this problem? Any way around it? I don't want to change my main log-in details. Thanks,
Product Support | | Davinia22
Davinia0 -
Moz Email Subscriptions
Hi Mozzers! We recently had someone leave our company who managed a lot of the Moz stuff we were doing and I have now taken over their role. I have been forwarded their emails and they keep receiving information from Moz such as the "Monthly Moz" and "Upcoming Mozinars", which is great, except I want those emails to now be sent directly to me. I cannot for the life of me figure out how to subscribe to these emails through our Moz profile - please help! Thanks! Sam
Product Support | | BlueLinkERP0