Unsolved Moz crawler not working
-
Hi
Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt
can the support team help ?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not crawling on my site
Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh0 -
How get rid of 403 crawl error?
My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?
On-Page Optimization | | emrekeserr30 -
Unsolved How do I cancel this crawl?
The latest crawl on my site was the 4th Jan with a current crawl 'in progress'. How do i cancel this crawl and start a new one? I've been getting keyword ranking etc but no new issues are coming through. Screenshot 2022-05-31 083642.jpg
Moz Tools | | ClaireU0 -
Unsolved /%25s
Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page https://kaydee.net/blog/ But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.
Moz Pro | | kaydeeweb0 -
Interested in testing a new Moz product for Content Strategists?
We're building a new product for content professionals that helps with website auditing, content performance measurement and content discovery/curation. If you're a content strategist, blogger, editor, copywriter, SEO with a content focus or social media manager that curates content we'd love to hear your thoughts. Interested in a sneak peak of what's cooking? Either respond to this post or send me an email at [email protected]. Be sure to include your job function and the type of company your work for. Thanks all!
Moz Pro | | JayLeary5 -
Moz metrics
This discussion is strictly theoretical... I won't hold anyone to their answer. If I have 2 websites that are identical in every way and let's say the domain authority for both is 40, and I 301 redirect one site to the other, what would the DA become? Same question for single pages, both with a PA of 40. If I 301 redirect one page to the other, what does the PA become for the remaining page?
Moz Pro | | AMHC0 -
Why cant I hook up facebook to SEO Moz Campaign Tool?
Greetings from the digital epicentre otherwise known as Wetherby Uk 😉 I want to track sentiments ie mentions, slurs fliratations of a brand on Facebook. So to start out with I thought 'Id hook up SEO moz social tracking service as illustrated here: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/cant-add-facebook-url.jpg But whilst i could add my twitter account I caount not add my favebook page:http://www.facebook.com/david.honan.98I added david.honan.98 in the url box and a number of other versions but none successfully hooked up. Am enetering the wrong url or is SEO moz socail media facebook plugin jinxed? Thanks in advance, David
Moz Pro | | Nightwing0 -
Sudden decrease in Moz Page rank
Hello, We have a serious issue with 404s and recently saw our Moz Page rank fall from 53 to 47. 1. OSE Inbound links no longer shows any of our Linked In posts, did Linked In stop passing juice? 2. Does SEO Moz reduce your ranking when there's a sudden increase in 404s? 2a. WP Yoast SEO - I accidentally checked the box on this plugin to "Strip the category base (usually /category/) from the category URL" which basically caused all of our blog post categories and Datafeedr categories to disappear. Didn't realize till too much time had passed that I accidentally clicked that box. Datafeedr is a plugin for our estore that parses the data feeds from affiliate vendors and allows you to create a saved search that auto updates old products every 3 days. I had a no index/follow parameter on the category items, but seeing the # of 404s continue to increase, I temporarily removed this parm last week to see if it reduces this now static number of 404s. Google Webmaster tools started showing a ton of soft 404s that kept increasing, while SEO Moz didn't show any of those 404s. I didn't pay much attention to GWT since Google kept saying it won't affect our rankings, and nothing was showing up on SEO Moz. Last week a fraction of those 404s showed up and I am not sure if that's what lowered our Moz rank or what looks like a possible delinking from Linked In and a higher ranking complimentary website directly related to our field itsallaboutyoga. Looking at the Moz graph of "Total Linking Root Domains Over Time" all of our competitors took a similar % hit since between June and the end of July, so I am thinking its more wide based than fat fingered mistake. I fixed # 2, (have to still figure out what to do with most of those 404s, thinking of submitting a request to Google vs 1,000s of 301s) so in doing my review of this sequence of events and using it as a learning experience, where would I assign max destructive value as a percentage? A. Ignoring GWT soft 404s in favor of SEO Moz campaign reports B. No follow from Linked In and related industry site C. Datafeedr, thousands of indexed products through Datafeedr that are no longer available mostly due to WP Yoast SEO fat finger error. I did have the D. WP Yoast SEO, "Strip the category base (usually /category/) from the category URL" E. Global Google algo change Cheers, Michael
Moz Pro | | MKaloud0