This, I think, is the real problem. Too many people feel the need to be superior to someone, whether it be due to race, height, gender or whatever. It may be because they were born with a silver spoon in their mouth (in this case, white privilege) or, conversely, the opposite — being so far down the totem pole that they’re desperate to be better than anyone.
I know that a lot of people don’t really like to think too hard, but it is the tiniest step from Love thy Neighbor to understanding that racism must then necessarily be wrong. Not taking that step is a very deliberate choice.