It's like
cratcheyty ol joe said, the vault experiments which concept was first introduced in Fallout 2 were suppose to be very subtle, a system here that didn't work as it should or had a secondary function that was not mentioned, a population of groups that could not perhaps easily live together, a vault command hierarchy with flaws, or experiments in creating a new type of human society as mentioned in Van Buren in which children were separated from their parents and re educated by the Vault's teaching computers to learn the basics of production again such as agriculture that did not relied on advanced technology that would be lost or in short supply.
Only a few experiments were 'wacky' or plain ridiculous such as the Vault in which gas would be released that would make people hallucinate or turn them psychotic, the one with very bad entertainment files, or very low quality tasteless food.
As for colonizing Mars vs colonizing an Earth like planet in another star system, well the problem is resources and back up.
If we had developed the right space infrastructure today we could support a moon or Mars base, and perhaps eventually a colony.
The problem with such installations is that they won't be self reliant for at least several generations before the right infrastructure has finally build up and there is a abundance of resources in reserve.
Think of the bases on the South Pole/Antarctica. Only a handful can provide their own power (I think a Russian base has geothermic energy), but they can not grow their own food, or in only a small supply, manufacture replacement parts, or synthesize medicines.
Outside the base you can not even grow crops or hold lifestock unless you want to feed the scientists penguins, seals, and krill.
Now you can still send the people on the South Pole perhaps all this manufacturing equipment needed for such purposes but it still does not make the climate more suitable for habitation.
The same goes for Mars, it would still depend on supply ships being send several times during a year, and the colonists would be forced to live as much underground as possible due to the cosmic radiation (Mars has a much weaker magnetic field than Earth does). It can only be colonized with continuous support of Earth.
An Earth like world in which you can at least breathe the air, or perhaps only need to filter it minimally, has water that also only require minimal filtering, has gravity as close to Earth normal as possible, soil that is capable of supporting the plant life you bring along, or perhaps even has plant life that is not poisonous to a human and can actually be digested would be a much better choice.
I like to think the Enclave has a starship hidden somewhere, either in orbit around Earth or in some secret rocket silo.