On Non-Magical Government Investment

It's pretty frustrating for libertarians to argue about government investment in science and technology because one is constantly confronted with the problem of the seen and not seen. One is bludgeoned with every government initiative that ever happened to pan out while all the wasted trillions and the private investment therefore foregone is lost to memory. 
My position is not that government investment in technology has zero returns. My position is that on average it does worse than returns to private investement. This should not be controversial. It is the consensus view of economists who study innovation and growth. If you think average returns to government-directed investment are higher than average returns to private investment, then you really do believe that the state has special generative powers. And you should formalize your findings, collect your Nobel Prize, and forever change the world. Or you should chill out about how awesome the Internet is.
I do want to distinguish between government spending on the development of particular technologies and government financing of basic scientific research. I'm convinced that a lot of valuable basic research would not be conducted without state subsidies, and that much of this research is the basis for later technological innovation that leads to increased growth. So here's one area where I think well-conceived government spending can pay its way by boosting growth. Despite ample motivation to be persuaded, I've remained unpersuaded by most libertarian arguments to the effect that scientific research without obviously marketable future applications would be sufficiently funded. There is a lot of waste, and some truly objectionable politicization, in government grant-making. But my sense is that, on the whole, much of American science policy is a good deal.
However, I get skeptical pretty quickly as we move downstream toward engineering and the development of technological applications of science. Here's where I see government subsidies responsible for a huge amount of misallocated human and financial capital.  People who think that we will tend to do better rather than worse when the government tries to pick winners in technology really do bear the burden of proof here and should stop simply assuming that landing a man on the moon has made us better rather than worse off. 
Last night, Kerry and I were reading Louis Menand's excellent essay “The Last Emperor: William S. Paley,” about the long-time CBS chief. In the middle of the piece, Menand argues persuasively that almost all of the television technologies that reached American households by 1990, aside from satellite transmission and the VCR, were available in the 1950s, but that regulatory collusion between incumbent businesses and government stifled innovation. “What we might have had for the last forty years is what, almost everywhere we have only had since around 1990: a mixture of local and national programming and commercial-free pay services on a hundred channels — and all in living color,” Menand writes. When government picks winning technologies it creates, as a matter of course, vested interests that will seek to, and often succeed in, creating barriers to further innovation — even if the “winner” the government picked turns out to be a loser. If you generalize the case of the TV lobby — which prevented or delayed innovation at every turn — to hundreds of other heavily regulated industries, one can start to see how government can have a systematically dampening effect on the pace of innovation.
Consider ethanol. Here's Michael Levi in an excellent article on “green jobs” in Slate:

For many environmental advocates, of course, these discussions [of whether green initiative will on net increase employment] are of secondary importance; what matters most is that green jobs will help the planet. They'd be wise to be careful there, too. Indeed, the most successful green jobs program to date is one that no environmentalist wants to brag about: the conversion to corn-based ethanol. A recent United Nations report estimated that the heavily subsidized U.S. ethanol industry provides employment for 154,000 Americans, about five times as many as the wind power industry and nearly 10 times as many as the solar industry. That goes a long way to explaining why, despite mounting evidence showing that corn ethanol is a failure (some would say a disaster) on the environmental front, U.S. policy appears to be on cruise control. At its base, corn ethanol is not a green policy so much as a jobs policy—and its success in that respect has made it almost impossible for the government to change course. 

And this is just the way it works. How much money has been sunk into this? Lots. That's money that could have been spent more productively, but wasn't. So we're poorer. And all the tens of thousands of folks right here in Iowa working in corn ethanol are misallocated human capital. So we're poorer. These are skilled, hardworking people whose diligence and effort is, thanks to the government, making the world worse. And the case of ethanol is no anomaly. It is completely typical.
Judging from some of my comments, one would think all the government ever does is land men on the moon and invent the Internet, and therefore libertarians are blinkered idiots. Now, I truly don't see the point of the moon landing, which strikes me as nothing more than a 20th-century version of grotesque pyramid-building waste. The Internet, like many other things based initially in government projects, probably helps account for the fact that returns to government investment are above zero. But it is truly hard to honestly identify the relevant comparison when trying to tote up the net benefits of government investment. What has been foregone in the process? (And don't forget to include the casualties of regulatory sclerosis that so often accompanies winner-picking.)
We cannot glimpse the nearby possible worlds in which the government did not for many decades help incumbents in telephony and television block basically every new innovation. Could we have had something like the Internet earlier had regulation not so effectively locked out any new thing that threatened well-connected interests dependent on the status quo regulatory dispensation? Menand writes that “[t]here were  subscriber-supported cable systems for radio as early as 1923, and television networks have always used coaxial cable, leased from phone companies, to transmit their pictures to broadcasting stations.” I don't know if this is true, but I have no reason to doubt it.  A well-developed early cable infrastructure might have been able to interface with emerging computer technology in inventive ways that we simply cannot imagine. Who knows what the present might have been had government gotten out of the way? I don't and neither do you. To simply assume that the value of the seen is greater than the value of the foregone unseen is a most elementary intellectual error. Yet some of you seem very proud of yourselves when you make it.  
Now, if you think Obama's centralized push toward a “green economy” doesn't assume one or two great leaps forward, then you should be clear about the fact that the very considerable centralized pushing up until now has taken us to a point where the cost of a unit of energy produced by “alternative” sources is still remarkably high relative to the cost of a unit produced by a carbon-based source. And it looks like this is going to be the case for a good while into the future. The green transition will either require a massive temporary increase in the cost of energy through some combination of taxes and subsidies, or some major leaps in green energy generation that swiftly brings prices in line with prices of energy from coal, natural gas, oil, and so forth. Subsidies may accelerate breakthroughs, but they can just as easily draw a huge amount of money and talent into dead ends, such as ethanol, and create heavily-invested corporate interest groups who will seek to block more promising breakthoughs in areas the government overlooked when first passing out the lucre. 
Also, if advocates of a centralized push toward a green economy (which you have to admit is a pretty radical and romantic thing to even think plausible) aren't counting on big technological leaps borne of subsidies, then they should be more open about the fact that their plan is really just to make energy incredibly expensive until incrementally developing green energy sources finally become competitive with carbon-based sources. Ten years? Twenty years? Thirty? And they need to explain why they think government winner-picking in green technology is likely to have a better record than government winner-picking generally. 
I predict green energy will pan out and that we'll have incredibly cheap, clean energy within my lifetime. And I think we'll get there faster, with much smaller costs to growth and human welfare, if the government continues to subsidize basic scientific research, but stays out of subsidizing technologies. I sincerely wonder why it is so clear to so many people that the big government push is the less risky path.