Despite their games following a fairly standard formula for a long while now, Bethesda did experiment with Fallout 4, probably far more than they would with TES.
They made the game even more FPS-focused, far more than TES could ever be. They changed the dialogue system. Introduced the voiced protagonist. Added lot more "sim" crap, settlement building and what not. Added a tie-in free-to-play mobile game. In short, they experimented a bit, with the game and with the general approach to the IP.
The changes between FO4 and FO3 are far greater than the changes between TES V and TES IV, and FO4 is a black sheep of the company right now. It sold well, but its reception wasn't as positive as they hoped it would be. Even among their fans, FO4 boasts a weird reputation. Plus, add that Beth fucked up on several instances - texture pack, for example, and above all, paid mods - many people still have a soar taste in their mouth, and will have for a long while.
All in all, Fallout is currently in a weird place. I'd say many Bethesda fans favor Fallout 3 (even FNV) over Fallout 4 and that Bethesda doesn't want more negative reception at the moment. On the other hand, FO4 sold well and some of the changes were widely praised. They will probably keep making TES games in a vein that Skyrim was (with technical improvements and voiced protagonist), but when it comes to Fallout, they will either keep experimenting more or will make a jump back to FO3.
If they do the former, Fallout will probably drown in a cesspool of free-to-play additions, even more cosmetic DLCs and greater FPS focus. And I wouldn't be surprised if we get an introduction of some sort of multiplayer (similar to ME3). To be fair, I'm surprised Bethesda hasn't gone in this direction already. I guess their previous games were not enough FPS-focused for that. FO5 will likely be.
BoS vs Super Mutant deathmatch? Why not. It wouldn't be the first single player sandbox game to have a multiplayer in a "closed" environment like an arena.
As for the other option, Bethesda might take a step back and make FO5 to be closer to FO3, to satisfy their older crowd. I don't see this likely however. I believe they will keep making TES games for their older fans, and Fallout will go down in history as an FPS.