The real problem is with actual Palythoa and zoanthid. Jullian Sprung has a talk on youtube about this. They have found higher levels of palytoxin in Palythoa which typically have rough stalks, brown or green color, and they get super slimy when you touch them. He talked about how he and his tank were poisoned by the mucus of a palythoa. I don't keep they palythoa cause 1) I think they are ugly and 2) they have been know to have higher levels of the toxin.
But you should exercise caution with zoanthids too. Use gloves and eye protection when fragging. Dont use a band saw to cut them. NEVER Boil, Pressure wash, Or try scrubbing them off. Most of the worst cases that I have seen online are because the person did one of those last 3 things.