Yep, don't go annealing pre-painted PE, it'll burn the finish right off. Annealing isn't so commonly needed on the pretty thin brass PE of today anyway. The last time I remember having to resort to a lighter's flame was when bending machine gun cooling jackets which were reluctant to hold their desired tubular shape otherwise.
The steel PE included in some kits is another thing however. Older Dragon kits as well as Trimaster's tend to have those. For them, annealing is a must, as they're "springy" and hard, making them sometimes tricky to even cut off from the fret.
As you reasoned, I would too try to avoid using metal tools with prepainted PE to avoid damaging the finish.
I recommend giving the pre-colored parts a coat of flat clear prior to assembly. It'll protect the pre-painted finish from scratching off, something which might happen to parts that require multiple bending (seat belts for example).
I've also noticed that the (at least the older style) pre-painting is prone to re-liquify when coming into contact with superglue (and can then smugde off when touched while still wet). A coat of varnish also takes care of this potential problem.