Some colossi don't try to kill you at all. Phalanx (the 13th) just flies around. Think there are at least a few others. The rest you can argue wouldn't wish you any ill will if you didn't disturb them. And they aren't dormin. They are vessels that keep it imprisoned. Otherwise destroying them would hurt Dormin.