Spooky pony, this is getting annoying!?

Sounds like hes doing what every horse does. I cant tell you how many horses i have trained that will do great and get worse its a roller coaster. You need a lot more ground work. He needs to learn that its easiest to just listen to the person. If he is spooky i would try some more desensitizing techniques along with sensitizing by controlling his direction in a round pen or lung line. Basically he needs to build respect for people. You cant do that unless you can control his feet, then through his feet his whole body. More ground work.