Thanks for your replies, they were very helpful.
After watching and reading I have decided that I need to implement rel="next" and rel="prev" in such a way that we handle multiple filters (facets) and sorting options so "to essentially pretend that only one version of the order/sort variable exists in every situation, and knock out the rest", that way Google will pickup rel=next sets for each facet on its own.
The video made it clear that big view-all pages aren't great if there is a chance they will take time to load.