Mine occassionally does that too. I find that the connectors from the switch to the motor housing are a little loose, and tightening them up sometimes brings it back.
Other than that, I'm not sure, perhaps the clutches in the power seat transmission nead to be cleaned?