"Any other character or actor could be replaced with the exception of Neal / Matt."

See I like him in the role specifically because he sets my teeth on edge. It makes it clear as soon as he opens his mouth that the key difference between him and Neal is that he's a total, total dick.