Don't think it's a good idea.

Kids have to be with at least one parent, for one thing. In your case, if some difficult situation turns up, you will be able to put your hands on it easily, for grand parents, however caring they are, are not good at handle unusual situations.