I don't think he meant the remaining viewers, but the abstraction that is the viewership he worships, loathes, and fears. I think he was saying that there were two baseline relationships in his life - with his father and with his viewers - and now he's down to one.