I find this thing a whole mess, first it was radiation and wild FEV, then it became radiation, then it became instant transformation when exposed to radiation (but as some fan theory goes, only if you have a gene that makes Ghoulism possible. They even wrote a whole theory on how the Vault beneath Bakersfield was set up to test how many people would become Ghouls)
This is my theory, Ghoulism (in Fallout 1 and 2, I can not speak for FNV) is the result of radiation and certain other mutagens/factors that are rare. (they were present in the Bakersfield Vault, the Reservation/Los Alamos, and what used to be Iron Mountains/Cheyenne mountain that resulted in a population of glowing ones. We can assume that these elements only existed in these bunker complexes).
It also takes years before the transformation to becoming a Ghoul is completed and it is more likely that a person dies long before that happens.
Though it leaves a person practically immortal (though not really immortal, cell decay has just been really slowed down now), it also leaves them in a very weak and fragile state. (ghouls can take less punishment than a human, and they are definitely not berserker powerhouses).
Glowing Ones are Ghouls with radioactive material trapped in their bodies. It doesn't hurt them but they can irradiate anything near them, causing radiation sickness and other related disease. As a result they are even more shunned than regular Ghouls.
Unlike FO3, FNV (I will see that as the result of reusing FO3 assets) Glowing Ones do not have radiation based attacks in which they can actually channel energy and radiation towards a target.
Ghouls and Glowing Ones are no longer hurt by sources of radiation, in fact they might even benefit from being in its presence. (if it heals them or if they need it I leave open for debate)
Ghouls are not zombies or mindless in general, though of course severe radiation and other factors (physical or mental) can damage or 'eat away' at their brains' cognitive functions. (a ghoul can just as well go mad from severe trauma as a human does)
There is also no reason for Ghouls to hang out at places like graveyards and such. Sure there are loners but like regular humans most Ghouls are still social creatures longing for company and acceptance. (plus they do need to eat, drink, sleep, and undergo medical treatment to make sure their condition does not further deteriorate)
Last, Ghouls are rare. There are not millions of Ghouls running around, survivors from the Great War with most of them having become feral. Only a handful of places that produced Ghoul populations.