No hate at all, as long as it is tied into the story and serves a purpose besides being a trophy room.
Hell, I had a castle of my own in Might and Magic 4 and they even bothered do implement it in the games progression.
Lets not forget that you could become a landlord in Baldurs Gate 2.