Because you can also disagree with the infallibility bit as well. The concept was invented to head off a doctrinal disagreement about something or other. It does not mean the Pope is divinely right about every opinion he holds. Otherwise Catholics would only be allowed to support a single soccer team...