Tony's confusion arises from the fact that he does remember AJ talking about Harvard and West Point over dinner at the end of season 2 - but it was part of his fever dream. I thought it was a nice touch as I also occasionally have odd memories that take me a little while to figure out were part of a 'realistic' dream.