The whole issue is cultural at this point. The only reason the confederate flag is even considered mildly controversial is that some backwoods hillbillys put on white hoods and flew that flag as a symbol of white power. Which as history later shows, how successful they were as a group.
As an aside, I have balck friend, who is one of the biggest rednecks that you will ever meet, who rides around with a confederate flag helmet. He does it because he thinks its hilarious. It seems to me that most moderately intelligent people see the flag as a symbol of southern culture. If someone does get upset about it, it usually just means they are looking for something to get upset about, like there aren't enough current events in the world to anger a person.