Missed the W&P again, gorram it.
It's just that at the beginning of the movie he's practically hostile towards Simon. And while the two of them have never been best buds, that hostile relationship felt really, really wrong.
All it would have taken was an explanation that Simon and Mal had a serious falling out when he started bringing River along on jobs and it wouldn't have been an issue. They hinted that might be the case, but a little more explicitness that Simon had been more integrated once upon a time wouldn't have hurt.
The events of the Serenity 3-issue comic might also have made Mal less than pleased to have the Tams on board (though how that was any worse than having Jubal Early sicced on them I don't know).