I have just noticed when I tried to move my passenger seat up it won't move, other times it works just fine. I have to fiddle with it back forth a few times and it'll move. Everything else works, tilt back and front, height, and moving back. Is the whole switch going bad or could it be something else?

 

If you don't hear any grinding or motor whine when it doesn't move for you, then it is probably that direction of the switch going bad. Should be able to find a new switch relatively easily.
If you're willing to remove it yourself you could also just see if there's some sort of debris blocking the contacts inside the switch.
Maybe just get one of those cans of pressurized air and see if blowing out the switch does anything first.