Hence “on the whole”. There are exceptions to everything, and that’s built into that very phrase. You’re not saying anything that anyone doesn’t already know.