I feel that the main reason so many people dislike math is math education (starting from elementary school). It focuses too much on memorization, doesn't encourage creativity and makes math feel boring.
Note that this translates directly into math teachers education too, as most of them aren't taught actual mathematics, but rather are taught to perpetuate the same standard curriculum. This is why there's so few "great" math teachers. It's not really their fault.
There's also an extremely harmful stereotype that one needs to be a genius or have some special brain structure to be good at math. It's so prevalent that it discourages most people from even trying.
Nothing could be further from the truth. Most people (as long as they don't have certain neural issues) can learn math and be good at it. It just requires (a) motivation and (b) hard work.
There are also no "innate geniuses" who became great mathematicians without any work or study. No, Ramanujan did not get his formulas from a dream. He actually got proper mathematical education and his papers contain actual proofs. Some savants can do mental arithmetic with huge numbers, but this activity is not the same as "doing math".
In any case, I feel frustrated that so many people are robbed from getting into math, as it's such a fascinating and (!) accessible subject!