The problem is that it seems as if only Gandalf was overly concerned with the fate of the world as a whole. Radagast was focused on his personal forest and animals while Saruman stayed in his tower and was eventually corrupted. We don't even know what the Blue Wizards were doing.