Your are right that

It's not fair to compare these two funds based on just 5 years of their performances. Nobody is going to dispute that Buffet is one of the top investors in the world in the last 50 years. But for average people, what we care more is the performances of our portfolios. BRKA may return just 10% a year in the remaining years that Buffet can still manage it, and still ends up as the greatest fund of all times. But if SP500 returns 15% a year in the same period, which one would you pick?