AbullSinCara is right as far as I know. I think people get confused about two different interfaces. The date and time correctly display at all times when you're travelling on the world map, and this is likely what people are thinking of when they recall "the time working before." You don't need the PipBoy clock at all since when you're using the PipBoy you're on a mission map, and you can press 'D' at any time to get the date/time in the log window.