follow jinjing

f(x) = a(x-2)(x-b) = ax^2 - a(2+b)x  + 2ab
f(f(x)) = a^3[x^2 - (2+b)x + (2ab - 2)/a][x^2 - (2+b)x + (2ab - b)/a] = a^3*I*II.
Since 5 is the only root of f(f(x)), it must be 2 or 4 multiple root.
So, at least one of the follwoing is true.
I = (x -5)^2
or
II = (x-5)^2.
for either case, we get b = 8. The rest is easy.