Site crawl only shows homepage
-
Hi everyone,
A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage.
Does somebody have an idea why the other pages of the website are not showing up?
Thanks,
Jens -
Hi SEOchris,
Thanks for your answer. I checked the robots.txt file, changed the User Agent to Googlebot in Screaming Frog, but non of these gave new insights. For now, we don't have access yet to the server log files, but when we have, hopefully they will tell us more.
-
The crawlers may be blocked inside of the robots.txt file. In Screaming Frog change the User Agent to Googlebot and recrawl the site. If that doesn't work you'll have to check the server log files inside of the hosting account for the site. Those may tell you what the crawl doesn't work.
-
Hey Jens,
Thanks for reaching out to us!
Would you be able to head over to this link https://mza.seotoolninja.com/help/contact and drop us a quick email. This will allow us to investigate further (and for that, we would need to access your Moz Pro account).
Looking forward to hearing from you!
Eli
UPDATE - The Campaign was set up for us to only crawl a specific subfolder as opposed to the whole domain. All sorted now
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | | AllAboutShapewear2 -
We Launched a new site and Rogerbot is still reporting on links/errors from the old site, is there a way to clear those out?
We are mostly a Branding agency, and have not put a lot of effort into SEO for ourselves... SEO tends to take a backseat to design most of the time, making it a little difficult for me at times when it comes to SEO. We recently launched a new site, http://Roninadv.com/ and the developer and I have done quite a bit of work to make it work well for Google. I was really looking forward to a new crawl report from Roger, but alas, It's like Roger crawled the old site? The new site has been up since last Monday. Is there a way to clear out the old errors? Do I just need to give roger more time?
Moz Bar | | PaulRonin0 -
I got a 404 in the Crawl Test Tool Report
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens? <colgroup><col width="1535"></colgroup>
Moz Bar | | IMforYou
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page | Best regards, Jos0 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
Keyword Difficulty Showing ONLY Bing Search Volume (Exact Match)
Hi I am using the "Keyword Difficulty" tool and selecting "Google US". But the report that gets generated shows "Bing Search Volume (Exact Match). Is there any way to get "Google Search Volume (Exact Match)" being shown in the report? Regards
Moz Bar | | rholt0 -
404 Crawl Diagnostics Report MOZ
Hi, I keep getting 404's appear in the Crawl Diagnostic error warnings. How do I find out which pages are linking to these 404 pages? How is MOZ finding them? thanks Ben
Moz Bar | | bjs20100 -
Why do the crawl diagnostics indicate duplicate page content among blog postings hosted by WordPress?
Does anyone know why the crawl diagnostics indicate duplicate page content regarding the blog we are hosting on WordPress? And does anyone know how to fix this issue? The content is not, or does not appear to be duplicate.
Moz Bar | | AndreaKayal0 -
Open site explorer linking root domains
My company has been trying to increase the number of linking root domains for a specific page on our website using our PR company and press releases that are sent out and linked back to this page. This is working nicely, but the number of linking root domains is still not increasing under the "linking root domains" tab. I am noticing the correct links to this domain under "fresh web mentions" though. I know this tool can take a bit to update, but it has been quite some time and I still only see the one link.
Moz Bar | | isret_efront0