Yeah, I don't know if an angel tapped into angel radio who's esconced in a highly secure bunker saying "Oh noes, they'll find this place no one could find for decades if he's around!" and following up with "I'm taking your brother if you keep your boyfriend" is not intended to tingle your Spidey sense, and having Cas vouch for him is a pretty good vote against him, given his track record.
If he does have a nefarious plan, which was on the table before the ultimatum, getting away from the only other angel on Dean's side would be exactly what he'd want to do.
If he decides to explain it himself to Cas, then I will be swayed. As is, totally consistent with someone who wanted to sneak into the secret hideaway, and is isolating the vulnerable parties from anyone who is more likely to catch him.