It's very interesting that over and out became the cultural norm, especially since it seemed to start around the time that most writers (like Serling) would have been more likely to be exposed to the military. I think it's either (or some combination of):
-Protocols were different, as you said
-They knew it was wrong…

That is correct (or at least was for the Navy). Since out implies over there's no reason to use both in a medium that discourages verbose messages. I have no idea why the over and out formulation became so popular in fiction, maybe something about the cadence? I also like how each one sounds a bit like it means. Over…