Eventually, the rubber valve hushers that sit between the valve stem and the shim bucket will harden and crack, and then make their way through the oil returns to the sump.
Once in the sump, they will usually get caught by the mesh screen of the oil pump pickup.
Here you can see a mass of old hushers that was inside the oil pump pickup:
I can't recall hearing of these hushers ever getting into the oil pump and causing significant damage, but they can obstruct oil flow into the pump, which can also be a problem.
It's perfectly safe to not use hushers at all, you will just have a slightly increased amount of valvetrain noise. In a fresh motor you can hear the difference, but in worn motors I'd say it's virtually impossible to hear the difference over the other mechanical noises present.