How you remember him doesn't matter. You don't get to discount his identity just to avoid a conflict between your memory and reality. I am sorry to be so blunt, but to know that you are using the wrong pronouns and to justify it as you have is insensitive.