Subj. (Well, atleast authors of fonline should know, because it works there)
I know that it's based on wall orientation, and checks are done by tile_in_front_of/tile_to_the_right_of, which uses tile_coord, which returns hex coords.
And it kinda works, but there's some scenery, like billboard...