A phobia is an INVOLUNTARY thing, it's not something rational and generally not something you can control. But regressives do so love to redefine words so, like other words (racism for example), it's lost all meaning now.
So yes, homophobia, REAL homophobia, not just 'a dislike of gay people', is an irrational fear of gay people. Prejudice, on the other hand, is a voluntary, wilful thing. There's nothing counter-intuitive about it, if you're not trying to redefine words to fit an agenda/narrative (talking in general, not specifically you, just FYI).