I think that most of what we associate with gender is purely arbitrary. When figuring out who we refer to as "He" or "She", or who wears what clothes, or what our expectations of someone based on their gender is, what genitalia they have is mostly irrelevant since all that stuff is mostly to do with culture and language, so we might as well as a society answer those questions based on what best empowers the individual.
There is of course biological sex, which is a reality, however even most of the stuff associated with that can be changed. If you've ever known anyone on Hormone Replacement Therapy you'd probably realise just how much about them changes. Lots of things such as weight, facial structure, development of secondary sex characteristics, even the way a person experiences their own sexuality are largely hormonal.
I think this is where the core misunderstanding regarding trans and non-binary people emerges. Lots of people assume that when people talk about gender identity they are trying to magically change their chromosomes and genitalia, when in reality it's mostly just people who are very aware of the limitations of what can and can't be changed about themselves, and using that to construct a personal identity they feel is more suited to them as individuals.