[Edit] In this way, you are at the mercy of

the whims and judgement of your boss!

This is not an issue about saying "yes" or "no" to your boss. It is about how much trust you can gain from her/him. If s/he does not trust you, you are the one as you said.;)