Well without the seal there is nothing to really support the kicker shaft, so it could be part of your problem.
Also to clarify, the idler gear is just a bushing, it does have a tendency to wear out, and when it gets back it will pop the snap ring off and that D washer will get digested by your motor and can cause some problems.