I had totally forgot that that scene directly followed chronologically to that other scene. Looking back, his hesitancy could totally have been that he wasn't "ready", and was afraid to admit to his wife that he enjoyed it with what's-his-name, afraid she would use it against him somehow. Maybe the plan wasn't for him…