They do let just anyone use it, though. Even in the games where it's been retconned in that Power Armor requires training, your companions (including such high-tech combat experts as a Dead Horses tribal and a junkie sex slave) can slip into any old suit of Power Armor they trip over (if they ever find one, which is 999,999% unlikely), and The Brotherhood doesn't attack you (or even comment on it, which is a game thing) when you show up on their doorstep with your partner decked out in full T-51b. In NV, you can get PA training from the Remnants and go to your first encounter with The Brotherhood in full (and highly distinctive) Enclave armor and no one so much as bats an eye.
In-lore, they clearly state in Fallout 1 that any idiot can plug himself into a suit of Power Armor and operate it with no more difficulty than his own arms and legs. In Fallout 2, your born-and-raised tribal scion can use it as soon as he finds it (on a ruby red/sapphire blue moon), as can his buddies the obese repairman, the senior citizen, the rotting ghoul, the spastic 120-lb teenage junkie, and the borderline retarded farm-town siblings. As far as procuring it, yeah, the BoS and the Enclave mostly have the market cornered, but every single game except the first has had multiple suits of Power Armor you could procure without ever even coming into contact with either faction. In NV, even the Great Khans will sell you hard-used sets of T-45d.
There are no two ways around it. Power Armor Training is a flimsy retcon implemented to try and maintain game balance (and, possibly, try to make Power Armor feel more like a reward), and considering the comparative weakness of PA in the Gamebryo Fallouts it doesn't make a terrible lot of sense. I find it telling that in Josh Sawyer's New Vegas mod-- essentially, the lead designer's definitive version of the game, if he'd had free rein and a full development cycle-- eliminates the training requirement entirely, replacing it with a training bonus that allows the armor to carry its own weight more efficiently.
Power armor is unlikely to be found UNLESS you join the BoS, so it is simply not a viable suit of armor.
Viability comes from availability. I would rather, as an average wastelander, use combat armor that I can more easily repair than power armor that is rare and I might be unable to fix if the MECHANICAL parts in it wear out over time.
Not to mention that in-game, and likely in-lore, not many raiders (who would be the main enemy of a wasteland wanderer) would be using weapons and armor that need power armor, so wearing one is simply a waste.
This might be one where we have to agree to disagree. Two of the cornerstones of the Fallout franchise that have remained unwavering are the preternatural durability of select pre-war construction (guns, buildings, stealth boys, Power Armor, et al) and the fact that the PC has never been just your average wasteland jerk. Even if the courier you're roleplaying has worked their entire life as a farmhand and letter carrier, they've got more raw potential than others, and they just so happen, through luck or tenacity, to discover things and get involved in situations that other people wouldn't (/haven't/don't).
Even if I don't quite see eye-to-eye with you on where you're coming from, though, I do agree that Power Armor should probably be hell for the player to acquire in future games, and I'd add to that that it could probably use a complete rebalancing.
As to rarity, the BoS definitely isn't the only game in town-- They're descended from a single garrison of pre-war troops, and similar groups (and perhaps the odd arsenal or two) are bound to have survived, but at this point, any caches that haven't already been claimed are probably inaccessible or impossibly well-hidden, and any factions in possession of the armor that haven't already died out or dispersed would probably guard it jealously. Incidental PA finds (Deathclaw Promontory prospectors, Brotherhood/Enclave corpses, Great Khan armory, F3's medic armor, etc.) should probably be in a state of dismal repair and unusable without intensive servicing and rare components, and locations containing preserved suits (aside from high-tech faction locales, which would be nigh-suicidal to steal the armor from if it were given a proper rebalance) should be impossibly well-sealed, -concealed, and/or -secured. Unfortunately, the abstract "pay-to-repair" mechanic and the fact that "challenge" seems more and more a dirty word to the AAA game industry make both of these ideas unserviceable at present.