Having not played Fable 2, I cannot say whether it's a better game or a better RPG than FO3. That said, some places have rules against (or just tend to shy away from) giving a game wins in multiple categories. Perhaps they felt that either 1) FO3 is a better game, but they'd already given it RPGotY or 2) Fable 2 is a better RPG, but it deserves GotY, so RPGotY had to go to someone else. Not saying that's the case, it's just a thought.
As for GTA IV's review score being higher, but it not winning, that's a really naive way to view things. There have been plenty of cases to demonstrate that scores are inconsistent and inherently flawed. There have been cases of sequels getting higher scores then the original, in spite of the reviewer stating outright that it isn't as good as the original. The reverse is true, too, where a sequel gets a lower score than the original in spite of the reviewer preferring the sequel over the original. Review scores are entirely subjective, and any reviewer's score of a game could change from day to day. You cannot simply place a number on something as opinionated as "is this game good" and expect that to be the only metric by which it is measured.
