Also, as you go up, the ambient pressure decreases, which will cause the balloons to expand more. So, yea, you're absolutely right! That's why high altitude balloons start out looking so "under inflated." The following composite image shows the same balloon at different altitudes, increasing as you move to the right.