I get that it’s your colleague and it sounds like a work trip, but it’s totally legit to say “Hey colleague, this is our night of catching up. Happy to get dinner with you tomorrow night and maybe friend can join. I’ll see you at breakfast tomorrow.”