The only problem I have with people trashing on the use of "literally" in a sentence is that no ever claims that using "figuratively" is the correct usage, even when it's true. No one is every going to scream "I am figuratively madder than I've ever been!"