The issue gets more clouded by Section 1 of the 14th Amendment, which states:
Section 1. All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States and of the state wherein they reside. No state shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any state deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.
Not the 15th, which explicitly applies to race. The question lies in does marriage have the status of a constitutionally protected privilege or not?
Well, yes. WHile one of the main bodies of law for civil rights litigation is equal protection, perhaps a more important area is substantive due process. Under this doctrine there are certain rights that are deemed to be fundamental- and that includes the right to family. Since marriage is seen as part of that issue of family, arguably their are other constitutional protections at issue.
Part of the problem is that this thing has become more political and legal- and when that happens bad law gets passed.