Just say that you would prefer not talking about

non-business related stuff during working hours. However, You woule be "glad" to satisfy their curiosity after work but you never show up after work.