My page cant be crawled by MOZ
-
Hi everyone,
I have a page named zita.vn. Im using Moz to do SEO for the page, but when I use On-page grader in MOZ, I got an error that the page cant be accessed. Additionally, I got an issue with page crawling.
Google search still crawl my page normally.
Please help.
Thanks a lot.
-
Hi there,
I have been checking my site for a week, but I havent seen any new updates about my site in backlink. Im sure that since last week there are a few sites pointing a link to my website but I havent found it on Moz.
Please explain it for me.
Tks a lot,
-
The useragent is: rogerbot
-
Hi Kristina,
Thank you for your answer.
Currently to protect our site from being hacked we need your user agent to add it into our whitelisted, so you will be able to crawl our website.
Regards,
Phuong.
-
Hi there! Kristina from Moz's Help Team here.
When our crawler attempts to access your site, we are running into this error: http://screencast.com/t/CmprKKb7a "403 Forbidden".
This normally means that you're blocking us on a server level. When I ran your site in a header checker I was able to see a 200, but when I curled your site from our servers it returned a 403.
The bummer here is that I can't do anything on the support side to fix this.
This 403 error is coming directly from your server, so at this point the best thing to do will be to send this information to your web dev. They'll need to make sure our crawler (rogerbot) is whitelisted, so our tools can start working hard for you.
As always, you can contact our team on issues like this in the future by emailing [email protected], but hopefully this helps point you in the right direction towards a solution here.
Thank you,
-Kristina -
I'm seeing a 403 response for your website:
wget zita.vn
--2016-08-02 11:46:09-- http://zita.vn/
Resolving zita.vn... 107.167.188.173
Connecting to zita.vn|107.167.188.173|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.zita.vn/ [following]
--2016-08-02 11:46:09-- https://www.zita.vn/
Resolving www.zita.vn... 107.167.188.173
Connecting to www.zita.vn|107.167.188.173|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2016-08-02 11:46:10 ERROR 403: Forbidden.If Moz is seeing a 403 they're not going to be able to crawl.
-
there is a possiblity of this issue is due to TLD, as i believe Moz crawler do not support to all TLDs as of yet. this could be the only reason comes to my mind. However, i reckon to raise a suppert ticket for more certian answer for this.
-
Maybe you've set a number of threads too high and pause between requests too low? Some hosts will block IPs that send too many requests. How much is too many depends on your host.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz is not showing traffic stats..;(
I have connected my analytics account in moz. But it is not showing my traffic stats.
Moz Bar | | daniyal_ahmed0 -
Site Crawl report show strange duplicate pages
Beginning in early in Feb, we got a big bump in duplicate pages. The URLs of the pages are very odd: Example URL:
Moz Bar | | Neo4j
http://[email protected]/dir/page.php
is duplicate with http://website.com/dir/page.php I checked though the site, nginx conf files, and referral pages, and could not find what is prefixing the pages with 'http://firstname.lastname@'. Any ideas? The person whose name is 'Firstname Lastname' is stumped as well. Thanks.0 -
How does a non-traditional TLD impact Moz's crawl test?
I have a client who moved from a .com to .academy domain 6 months ago, and their current crawl tests are coming back with a universal page authority of 1, along with 0 indexed backlinks. The previous version of the site had an average page authority of 35-40, the site architecture and content are nearly identical, and there are no other errors or red flags in the crawl report that would hold back their organic rankings. In fact, looking at the site's analytics account, I can see dozens of sites that provide current and properly functioning backlinks, non of which are listed on the crawl test. So the question is - is Moz currently unable to properly crawl a .academy (or any other non-traditional TLD) site, or is there some deeper issue with the site's SEO that I'm not seeing? Thanks!
Moz Bar | | ThinkAOR1 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
URL inaccessible for On Page Grader
I am trying to use the on page grader however it is not working with my website. The URL is as follows: https://capbeast.com. I have been trying to read in older posts to see if https is now supported or not but have not found anything. I know there is no robots.txt issue as I am able to run the crawl test on our website fine. Is the issue on my end in regards to configuration or is due to DDos attacks? Any help would be appreciated. Thanks
Moz Bar | | MisterStitches0 -
On Page Grader not working on the fly
On Page Grader has this lovely little re-grade page button next to the URL and term field but it's never worked on the fly for me. What's going on here? You discover something critical, fix it and you want the satisfaction of re-grading to test it and get that A but it doesn't work. I can see the changes on the site so why not??
Moz Bar | | wearehappymedia0 -
I am doing the On-page SEO for a website that's never had any SEO done before. I will start with the Pages. Is it necessary to do SEO/Keywords for older Posts?
Using the On-Page Grader to perform SEO on pages of WordPress website. This website never had any SEO done before. Should I go back and perform the same work on the Post Pages?
Moz Bar | | Joseph.Lusso0 -
What happened to moz Crawl Test? Is it moved in the redesign?
Love the new design of moz.com! And the mentions tool is terrific. However, I use the Crawl test a ton and cannot find where to access it now.
Moz Bar | | jessential0