I see, Anthony provides some good insights below.
Removing inbound links from other domains thus effects your overall inbound links which then can have an impact on your natural ranking. Restoring them will probably fix the immediate issue. BUT! There is a fine line here that should be discussed.
You don't want to become a victim of link manipulation this can be done by buying a crap ton of domains making some generic content and point them back to one another or just one, creating an internal link ring or link funnel. This will eventually get you penalized. Now, owing a handful of sites and pointing them to one another is perfectly acceptable. An example being Pepsico pointing to Pepsi, Mt. Dew, Taco Bell, Quaker Oats, Tropicana ect.. these are all companies they own. In fact our company owns 3 branded websites, but we were careful only pointing to relevant pages when it suits the user's experience. You may have been benefiting from a form of this, which is why your rank dropped whence you removed the links.
Hope that makes sense.
Don