I think it's a cruel trick GRRM plays on the reader by keeping him hidden for an entire book. Your mind is trying to create a 'new' character personality/visualization in your head and then it's hard to reconcile that person with Selmy until a reread