Site crawl only shows homepage
-
Hi everyone,
A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage.
Does somebody have an idea why the other pages of the website are not showing up?
Thanks,
Jens -
Hi SEOchris,
Thanks for your answer. I checked the robots.txt file, changed the User Agent to Googlebot in Screaming Frog, but non of these gave new insights. For now, we don't have access yet to the server log files, but when we have, hopefully they will tell us more.
-
The crawlers may be blocked inside of the robots.txt file. In Screaming Frog change the User Agent to Googlebot and recrawl the site. If that doesn't work you'll have to check the server log files inside of the hosting account for the site. Those may tell you what the crawl doesn't work.
-
Hey Jens,
Thanks for reaching out to us!
Would you be able to head over to this link https://mza.seotoolninja.com/help/contact and drop us a quick email. This will allow us to investigate further (and for that, we would need to access your Moz Pro account).
Looking forward to hearing from you!
Eli
UPDATE - The Campaign was set up for us to only crawl a specific subfolder as opposed to the whole domain. All sorted now
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 1-page 301 status error but httpstatus.io says its 403
I am trying to run a site crawl for my website and MOZ is only resulting in 1 page crawled with the home page URL Status Code of 301. However when I run it in httpstatus.io it is giving me a 403 status error. Im curious as to why MOZ is saying its a 301 and httpstatus.io is saying 403. Is there anything I can do in MOZ first to get the site crawled before asking my developers to look into the 403 error?
Moz Bar | | JohnConover0 -
Keyword Explorer shows wrong data
Hi all, Trying to figure out what is going on with the keyword explorer. All searches are made in the EN-US region. Example keyword: app localization
Moz Bar | | Fred.Visnevsky
Moz shows me that volume is 0-10, which is Wrong. Actual volumes are in thousands per month. Another keyword - translation service
Moz shows the volume as 2480 when actual volumes are in tens of thousands. I can give many more examples like this - what is going on? NZGzX2p.png vzIwMoe.png1 -
Is the update site crawl feature following robot.txt rules?
I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
Moz Bar | | jamestown0 -
External Links not showing in MOZ?
Hello:) I have a ton of external links on my site, but there not showing on MOZ? The reason i'm slightly concerned is because my top competitor has 80 and I show as only having 3? Does any one know why this would be? Also, my traffic on MOZ is a lot lower then my traffic report on Google Analytics? Is that normal, & which should I trust more? Here's my Site: Www.ThumannAgency.com Thank you in advance:)
Moz Bar | | MissThumann0 -
Q&A Question showing as answered when it is not
Yeh, that question at http://moz.com/community/q/onpage-grader-says-site-unreachable is not actually answered even though someone has marked it as such. Don't know how to mark it as not. Just annoying, as it may prevent someone else chime in with new information now I have updated it with some more info myself as I am just trying to get this resolved.
Moz Bar | | TheWebMastercom0 -
Crawl Diagnostics - nofollow - reducing duplicate pages
Hi I'm looking at a crawl diagnostic report, I can see I have many duplicate pages, the reason for this is that when a brand filter is applied to a page. IE
Moz Bar | | chameleondm
www.mysite.com/mycategory - lets say this is the product listing page
www.mysite.com/category/mybrand - and this is the same page but with a brand filter applied
www.mysite.com/category/myotherbrand - and this is the same page but with a different brand filter applied I had intially appendeded the meta title, description and keywords with some extra content if a brand filter was applied, because the page on the whole does have different content. IE I would have a custom meta information, H1 tag and products on that page just for that specific brand.
However I am wondering if these two pages are really just competing with each other as lots of the content will be the same. Should I scrap that approach and use either nofollow on the brand filter link, or simply use a canonical. Thanks, James1