Keep it and input your effort

Don't think you should dump him for the things that you can still work on. Don't expect him to be perfect. You can help him to make it better. Just input your effort as well. It's not easy for a relationship to last a few years. It's part of your assessts already. You don't want to loose it.