At this point, regardless of how anyone feels about Allen, it is the safest thing possible to say you regret working with him/don’t want to work with him now. There’s absolutely nothing brave about it. It’s like saying you don’t want to work with Bill Cosby.
“I refuse to work with Kevin Spacey!” Wow, congrats. So…