You would be wrong. Heat transfer is proportional to the difference in temperature between the swimmer and the surrounding water.