Huh, I didn’t think of that. The Chitauri leader/Thanos lackey gave Loki the scepter, though. It could have been symbolic, but I’m willing to assume the Chitauri army was slaved to whoever is granted the scepter, and the mothership has some kind of broadcast system for the Mind Gem’s control.