Relatively Minor?

So, I continue to be annoyed with Jonathan Chait's book. Here's the sort of thing I have in mind. Page 19:

From 1947 to 1973, the U.S. economy grew at a rate of nearly 4 percent a year — a massive boom, fueling growth in living standards across the board. During most of that period, from 1947 until 1964, the highest tax rate was 91 percent. For the rest of the time, it was still a hefty 70 percent. Yet the economy flourished anyway.
None of this is to say that those high tax rates caused the postwar boom. On the contrary, the economy probably expanded, despite, rather than because of those high rates. Almost no contemporary economist would endorse jacking up rates that high again. But the point is that, whatever the negative effect such high tax rates have, it's relatively minor. [emphasis mine]

My response to this was: huh? Two things Chait either does not understand or takes pains to ignore: (1) the relevant counterfactual and (2) the compounding nature of economic growth. He concedes that these astronomical rates put a brake on growth — and for about a third of a century (i.e., the '47 -'73 boom plus the following downturn until the '8os cuts.) So what would the growth rate have been with much lower tax rates? I don't know. Depends on how much lower, obviously. But, other things equal, higher. (Anyone know of a rigorous estimate?) And Chait agrees.
Now, if it had as much as 1 percent higher on average, that's going to be close to the difference between GDP per capita tripling rather than doubling over that time span — which is to say, the difference between what we got and twice as much. Obviously, an additional doubling of GDP is not minor. Even were the increase in growth from lower taxes only a couple added decimals on the average growth rate, over 30 years it still would have added up to something much more than “relatively minor”. Does Chait think that something like a 20 or 30 percent higher average income in the lowest decile would have been “minor” for the poor? What if that amount was larger than all the money actually spent on poverty over that period?
U.S. GDP per capita is now about 30 percent higher than much of Europe's. There is good evidence that this has to do with incentives to work, definitely including tax rates. Is this difference “relatively minor”? What do you suppose Chait would need to know to admit that the effects of high tax rates on growth and well-being are “major”?