I think you've added a few elements there that aren't present in the text. All Ned thinks in regards to Rhaegar is that he doesn't seem the type to frequent brothels. Full stop.
The rest of that musing is all you.