We've been leaving it in all the time and have not seen one putt not go in because of the flag stick being left in. It does speed up play because now the last guy putting does not have to have a guy waiting with him and picking up the flag stick and putting it back in.
This past year our course put in new and different metal cups. They're supposed to be put in an inch below the surface, but sometimes they were only 1/2 inch below the surface. It happened to me twice where a putt at normal speed hit the back of the cup and bounced back out.
My only issue with leaving it in is I have shorter fingers and a longer palm and at times squeezing my hand down and getting my ball out takes me more than one try. Other than that I have seen no issues with leaving it in.