Speaking as someone who's lifted for years, you can actually see an awful lot of fluctuation in his physique WITHIN each film. Even in Thor 1, there are scenes where he has 20 pounds of extra muscle, and ten minutes later, scenes where it's gone.