Economic Expertise and Moral Mathematics

Ryan Avent, following up on Ezra's post on the policy prestige of economists in general and benefit/cost analysis in particular, writes:

[I]f we can’t use cost-benefit, how do we make these decisions? How in hell do we figure out which trade-offs are sound ones and which are damaging to society on net? To the economist, and indeed to many policymakers, eliminating cost-benefit analysis is like depriving them of language — how can you discuss the problem without it?
In fact, we really have no other way of wrestling with these issues. One can argue by moral imperative — that we don’t have the right to impose serious costs on others — but we still must determine how much compensation or preventative action those others are owed, and from where the resources to pay or take action should be drawn. These questions involve trade-offs, which must be weighed in some fashion, and so again we find ourselves turning to economists.

Why do we turn to economists again?
I agree that it is impossible to think intelligently about policy without some minimum of economic literacy. But the economist has no competence whatsoever to tell us, say, the appropriate discount rate to apply to future costs and benefits, to take one important example. I've heard philosophical arguments to the effect that the discount rate for future welfare should be zero and that the discount rate should approach infinity as we consider the welfare of furture beings with whom there is no possibility of reciprocity. The funny thing is that I think people get the implications of discount rates wrong, and that both zero and infinity point to more or less maximizing growth. A zero discount rate plus a basic grasp of the relationship between technology and growth plus a reasonable projection of the current trend of technological progress implies an obligation to maximize economic growth rates with no concern whatsoever to avoid the incidence of future externalities of current activity. This is an economic argument, but it is also something rather more. Likewise, an infinite discount rate implies that we should do the best we can for our children and grandchildren, and leave it to our grandchildren to worry about their grandchildren. If we're doing something now that might hurt people none of us will coexist with 100 years, then so what?
Of course, neither of these arguments will convince Ryan or Ezra. I'm sure we disagree both about the discount rate, which is not itself an economic question, and about the implications of the discount rate, which is only a partially economic question. And I think it gets even worse: economists have almost no competence whatsoever in telling us what counts as a cost or a benefit. That's pretty important, isn't it?
So why do we give so much weight to the opinions of economists?