This criticism doesn’t mean that all white people are the devil, that malice or active racism are necessary. A hermit frontiersman in the 1800s might have had no opinion on slavery or even been against it morally.
But the act of doing nothing is tacit support of the status quo.
An auto union worker in the 1940s and ’50s may have thought segregation was wrong, but if they felt that opposition to anti-lynching bills in the Senate were equally important as economic policy, then their tacit support for a dehumanizing system of oppression is based on racism because it says that mobs torturing and murdering a man, woman, or child with impunity isn’t so important if that person is black.
In the same way, if you say that regularly stopping and frisking black and Latino people without any reasonable suspicion is ‘just one of many issues’, it’s because you think it’s unlikely to affect you or people like yourself, so you don’t care that much.
Malice is not required; apathy is more than sufficient.