Let me put it this way. There are roughly 10 towns within the open, that could reasonably fall victim to Raiders. There are roughly 16 Raider bases, most of which have several dozen raiders between them. These people could not survive purely by attacking caravans unless trading caravans are frequent enough that they could reasonably feed around 16 groups of raiders until the next one comes along. Either these gangs of Raiders are competing with one another, at which case why don't the larger ones just get rid of the smaller ones to make life easier for themselves, OR they are one big gang, in which case why don't they flatten or conquer the entire wasteland?