I would think punishment is worse for civil cases under the hypothetical libertarian form of government, because the damages can be set for an arbitrary high amount, and the debt collections for the losing party renewed in perpetuity. Think frivolous lawsuits, which cost businesses and individuals billions of dollars to fight. While the government offers venues (courts and judges) to mediate these lawsuits, it’s individuals and businesses that initiate them.
The government, on the other hand, has limits (10 years for IRS to pursue back taxes, etc) and can’t just sue for an arbitrary high amount like many private entities can. Everything is itemized. Also the government has pretty generous payment options for the defendant. And the bureaucratic sloth makes such lawsuits take forever, and the collection process even longer.
Being sued by the government is like stepping in a ring against a 300+ pound heavyweight that may occasionally land a hard punch, but is otherwise too encumbered by his girth to do much, and just nods off in the corner most of the fight, only to be occasionally aroused by his cornerman. Being sued by a private entity, on the other hand, is like going against a welterweight that will bloody you before you even lift your gloves. He won’t stop fighting until he loses or the ref calls the fight.
On the other hand, punishment for criminal cases may be worse under the federal government than the libertarian one. For example, the three strikes law, which some argue is too harsh for nonviolent recidivists. Also, issues pertaining to wiretapping and snooping private emails. Maybe the punishment for these criminal cases under the libertarian ‘system’ would be lesser. The government pursues certain criminal matters (chid engagement, homeland security, homicide) quite aggressively, with very long sentences, although the trials can take a long time.