It all depends! His exercise intensity, his fitness level, and his metabolism all factor in. I second Lucinda's suggestion to ask an equine nutritionist, or even just a knowledgeable trainer that you trust. If he's being ridden once or twice a week, hay might be enough for him to keep his weight. You could also look into a calming supplement--Smartpak has really great options for supplements.
He might just be a high-strung horse.

I agree with Lucinda and Amy. Also, be sure your horse is not wormy and his teeth are in good shape as those 2 things can make it difficult to keep weight on. Generally these lean horses respond well to adding FAT to their ration; it helps them put on weight without adding the carbohydrates that hot them up. In the old days we used to add vegetable oil but I would suggest going with a fat that you can get in a pellet like Visit Link

His teeth are great and is on a regular deworming schedule. So he is healthy I talked to my vet and he said that what I'm giving him is perfect for now :)

You can always add some vegetable oil to the mix. I agree with the others and be sure that his teeth and parasites are well controlled.

Well one of my horses I had got hot with alfalfa pellets, or any type of grain. So I stuck with just Timothy Hay and he did good on it, plus he kept his weight. The only thing is it bruised my wallet. So it was kind of expensive but atleast he was a more pleasurable ride. Oh I also cut some fresh grass on top of that.