My mom—God rest her soul—had a pretty clear-cut rule on what casual means: "Casual does not mean sloppy." It's steered me well.
I've seen some shockingly trashy looking stuff people have worn under the guise of being "casual." You're at work—not a park, not your apartment, not at a bar. C'mon people, show a little…