In this way, you are at the mercy of the whims and judgement of

This is not an issue about saying "yes" or "no" to your boss. It is about how much trust you can gain from her/him. If s/he does not trust you, you are the one as you said.;)