It should be judged based on the totality of the original Fallout games. I've seen precious few direct Fallout 3-to-Van Buren comparisons; those that have popped up have only been to compare the cheeseball storyline to that of VB's possible storyline; not without its share of goofiness, obviously, but not to the degree of "What?!?" in the FO3 story line.
If you don't know the story line yet, seriously, look through the thread on the News Comments section talking about the leaked FO3 YouTube vids. Spoilers start around page 10 or so.
As it is, as a Fallout game, Fallout 3 really really sucks. And even as a follow-up to Oblivion, it doesn't seem that good. Even the reviews that have come in with "9 out of 10" have talked about the lackluster AI, dull environments, poor dialogue, etc.
So, just talking about it as a Fallout game: it's bad. Van Buren was certainly going to be much better, and acting like all the posters around here are doing is comparing FO3 and VB is... at best misleading.