You can get lots of trouble this way

If agent A showed you several houses, and then you bought a house through agent B, what would agent A do?

He can sue you for part or all of the commission. If B does not want to share, then you would have to pay out of your own pocket.