I have checked my list of Apps with access and Pokemon GO is not on that list, I also do not remember getting asked for this permission for the App while the other Apps on the list did ask for permission.