Never make eye contact
Keep panning your camera after you take the shot. Point up at the sky like you’re taking a photo of a bird.
Use a quiet camera
Act like a goofy tourist
If you are caught, say you’re sorry and keep walking
Use your situational awareness. The odds of making a good photo are so low, it’s not worth it to take photos of people who look a bit off. The character photos lose their esteem the longer you shoot.
If someone acts aggressive, ignore them and keep it pushing. Never engage with angry people. Telling them your rights or justifying your strange hobby will almost never put the subject at ease. Even if you somehow convince the person your weird hobby is harmless, you’ve just wasted time and energy and likely ruined any kind of momentum you had.
Confrontation is almost always the photographers fault. If you’re having a lot of issues you should work on your approach.