And if you reverse the underlying decisions then you may as well kiss your freedom of speech goodbye. The reason speech and money are linked is very simple - if the government can stop you from spending money in distributing your speech then they can pretty much silence you.