may not be a problem as long as you do not sell it.

but When you have to sell it, you will have big problem. most people don't like it. only very small group of people like this kind of house.