It did come off looking like he was angry. Considering Hannibal's attachment to his sister and the conversation he had earlier with Will about his regrets in regard to Abigail, I do wonder if Will completely misinterpreted just how much Hannibal manipulated the situation.