Give it up

If it's not related to your field and you feel it's tough to learn, you should give it up.
It's not helpful to your current work if it's not related to your field.
If you change to another job which is relevant to signal processing, it will be hard for you to excel in that new job since you are not comfortable with those algorithms.
Maybe, spend more time with your two kids :-)