Why isn't the Moz crawler getting all of my item pages?
-
I am stumped and Moz is being terrible to work with.
This site has about 40k pages
39,800 of them are item pages roughly.
Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.
I do not know which item pages they are missing.
The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.
The site is built on Magento 2 and uses the Amasty Layered Navigation extension.
Does anyone have any ideas?
-
Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.
-
Hi Tyler, are you setting your pagination pages as nofollow? That might be causing the issue. Any chance you can post the URL and we can try crawling the site to see if we hit any issues with a crawlers like screaming frog?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I 'sign in' to the Moz Bar?
It's installed, I can see links etc as highlighted - but it won' t let me "sign in". This 20 second video explains: https://www.screencast.com/t/3kEjQFkTHZv Suggestions? Or shall I just ignore? Paul Barrs
Moz Bar | | PaulBarrs0 -
Accuracy of Moz Keyword Tool
I understand that no search tool is entirely accurate, but I'm seriously beginning to doubt the accuracy of the Moz keyword tool search volumes for keywords. For example: I entered "largest police departments" and it says 0-10 per month in the US, despite the fact that that term is directly responsible for 200 hits per day. Many alterations of that keyword come up with 0-10 or not determined. This is not a small error -- and it repeats over and over for other search terms that I have already ranked for in the US. It's gigantically inaccurate in a massive scale.
Moz Bar | | Luke_Proctor0 -
On Page Grade & Page Optimization Tools not working
Hello! I have an issue with my on page grader and Page optimization. Over the past two weeks i get the attached errors. I searched the forums but after contacting our developer I am in a bit of a standstill. Could someone help on what triggers this? Would highly appreciate it. Best,
Moz Bar | | pranbg
Pran c5CSsHK c5CSsHK0 -
Increase Moz authority above 26
Hi, I have been using Moz for 12 months and found some very useful features. My site’s score steadily increased from 18 to 26. But I seem to have now ‘hit a wall’ and my Moz ranking is static. My question is: Where on Moz.com can I find what next steps I need to take to improve my Moz score? Thanks in advance. Matt
Moz Bar | | wonderdome0 -
Performed a Moz Crawl Test - Says I have 107 External Links on Homepage??
Hello Mozzers! Exactly at the title suggests, I performed a crawl test on one our sites and the report says we have 107 external links on the homepage and another 34 on one of our internal category pages. On both of these pages I can only find 7 external links, anyone know why the crawl test is saying this? And if Moz is finding these external links could google be doing the same and punishing our site for the high number of external links? Any response appreciated! Richard
Moz Bar | | Richard-Kitmondo0 -
I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
The website is www.bigbluem.com and is a wordpress site. I'm getting the following error: 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag But what is weird is the domain it lists below that is http://None/BigBlueM.com Any advice?
Moz Bar | | TumbleweedPDX1 -
Who and how does one get in Fresh Alerts?
Who and how does one get in Fresh Alerts? This is such a great tool! Thank, Moz! I would like to use this more often and to a better advantage. Can someone help me understand what criteria the tool uses to choose who it and what it picks up? Why would someone's personal family gathering turn up in my Moz Fresh Alerts("Minneapolis home buyers"? http://mydesultoryblog.com/2014/07/having-a-great-time-with-katelyn-and-drew-in-wayzata-mn/ My Desultory Blog Desultory thoughts on a variety of subjects … Having a great time with Katelyn and Drew in Wayzata, MN It seems completely random when and which of my blog posts show up in Moz Fresh Alerts. For example one that did ("Minneapolis real estate sellers"): "5 Critical Shifts in the Twin Cities Housing Market" http://www.homedestination.com/real-estate-blog/4-critical-shifts-in-the-twin-cities-housing-market Jeannie
Moz Bar | | jessential0 -
Moz showing warnings for each dynamic link despite canonicalization?
As you can see in the attached image, Moz is showing a warning for each dynamic URL despite a rel=canonical tag. Is this by design? If so, it is frustrating seeing as it is really just the one page with many links . . . 9h5oDmr.png
Moz Bar | | BlueLinkERP0