The problem was that the hole in the rubber washer is smaller than the diameter of the screw. The rubber washer had to be stretched over the top the of the screw so just putting in on top won't work. The stretch fit over the screw keeps the screw from falling out of the hosel when removing the shaft adapter so it is more of a "keeper" than a "washer."
After fighting with it and getting it on I thought of another way of doing it. I have lots of small brass tubing from other hobbies. All I had to do is to stretch the washer over a piece of piece of tubing with an ID just bigger than the screw. Stick it down the hosel over the screw the use a piece of tubing next size up to slip over the tubing down in the hosel and push the sucker off.