It’s not that radioactive dirt/dust is especially harmful to your hands, it’s that you’re at risk of tracking it home and having it sit around your house, irradiating you, or potentially inhaling some if it, which is even worse for long term exposure.