Links from more authoritative sites are more important for improving rankings than links from related sites. There are tons of examples of sites that rank well that have almost no relevant links. One such example that illustrates this is pitchingmachines.net, which currently ranks #1 for the term pitching machine.
While this site has the benefit of being an exact match domain, its link profile is full of links from sites that are unrelated. 70% of their live links come from a paid blog network where the sites cover a wide range of topics (but all have PR), 5% are from article submissions (again, sites that aren't topically related), 4% are blog comments and another 3% are blogroll listings (from unrelated sites). That's 82% of their links coming from sites that aren't topically relevant.
Another 6.5% of their links are forum profiles, web directory submissions, sidebar links, and links lists. That means that at best, roughly 10% of their links are legitimate contextual links from related sites. In reality, it's less than that, as most of those links appear to be from another blog network -- I just haven't been able to identify which one yet.
Is there risk in this approach? You bet. If Google decided to stop allowing the blog network they're getting links from to pass authority, the rankings would tank. Worse yet, if Google decided to penalize sites getting these types of links, they may never be able to recover their rankings. Other than the blog network links, though, I think the risk is minimal. Blog comments, directory submissions, blogroll links... those are all types of links legitimate sites might be expected to have.
So, don't stress out too much about obtaining relevant links. Tons of sites rank without them.