Did you talk enough to the old man? After doing the first shrine he will tell you to look around for the remaining three shrines from atop of the tower, if you visit him in his cabin I do think he mentions that you need to prepare for the mountains because they are cold.
If you read his diary in the cabin he will also…