I don't worry about the ferrules, plastic or rubber. They are cheap enough & I keep supplies on hand of all of them.
The only time pulling wouldn't be acceptable, or possible, is when the fork will not accept the width of the adapter. I found this out when making the pullers and had to redo the fork width.