The followers that stay with you are controlled differently than tagalong NPCs for quests... it's all done with dialogue conditions.
I've succeeded in making follower max based on charisma, right now I'm adding in dialogue for my Magnetic Personality perk, which gives +1 followers and lets you recruit followers regardless of your karma.