Yeah, the main problem is that Chuck seems to think there's 4 groups of people in this world. Chuck, the law-abiders, the criminals, and his brother. And no matter what his brother does or doesn't do, he's already assigned him his place in that hierarchy.