Have you ever gotten on this horse without a saddle? I am not suggesting this, only wondered if the saddle is your problem or your riding is messing the horse's mind. Only you can handle this horse? is a huge red flag that the drama here between you and the horse is due to poor, inconsistent, or inept handling on the ground and under saddle, sorry. Until your horse is calm, listening to you and not annoyed by your tack or riding? You won't be teaching him anything worthwhile and he has a lot to learn - or you do. The advice to get help from a trainer is your best bet. I don't think either one of you needs to revisit riding at this juncture. If you aren't using a helmet, do. There are so many ways to get killed by horses and this is a very quick and reliable fix for one of them; even on the ground.