Jurisdiculous

Some libertarianish and conservatives types sometimes like to think of the territory of the U.S. as a big piece of real estate over which citizens have a kind of shared property right. There are lots of things that are wrong with this way of thinking, especially for people who think their philosophy is grounded in a strongly moral conception of property rights. Perhaps the primary flaw in the national-territory-as-collective-property schema is that very little U.S. territory was gained legitimately. Mostly it was gained in the way political territory is generally gained: war, theft, and purchase from thieves. Some worry about the fact that the Louisiana Purchase was unconstitutional. But why not worry instead about the far more troubling fact that the French state could not have been the legitimate owner of large swathes of land already owned by natives? Can James Polk's brinkmanship really be principle that determines that I have a stake in what happens in in Portland, but not in Vancouver?
I understand that this is the way the territories of kingdoms, principalities, and states are formed. Colonial criminality drew the map. That's the way it is. No turning back. But shouldn't we be long past the idea that these traces of regrettable history have truly weighty moral significance–that the Treaty of Guadalupe Hidalgo or the Gadsden Purchase somehow create deep moral facts about where some human beings should and should not be able to go?