That would only apply if you assume Bert thinks every client who comes into the office keeps track of what secretary is where—people "seeing her from the elevator" would have no idea she had been "demoted."
Also, if he did think that, I have no idea why he would be so indirect about it in his conversation with Joan,…