Yeah, I don't think Peter would put the ring on unless it was his own. But like you say, ita, it has to be something understandable or the premise of the show is ruined.
Bizarre theory: Kate's the real villian in the piece. She's screwing with Neal, trying to get him to give up his stash - she was all "just tell me where it is OMG!"; even Neal was suspicious of that. And why would she stick by him for almost four years and then leave him four months before he was getting out? Peter is the one who got her to leave Neal, probably using her criminal record to coerce her, because Neal was about to get out of prison and he and Kate and the stash would've been gone. Kate might even have been planning to kill Neal for the money. Peter didn't just arrest her because he wants to use her to get Neal's stash - it's all stolen stuff that legally should be returned to its rightful owners. Maybe he's got Kate on a leash just like he does Neal, and he's just realized Kate is messing with him. Doesn't explain why Fowler is suspicious of Peter, though. Maybe Peter's arrangement with Kate isn't as aboveboard as his with Neal.
Kate really looks like Elizabeth, huh?
Yes, I thought so too. Quick bizarre soap-opera theory #2, based solely on Kate's looks: Kate is Elizabeth's estranged daughter (Kate could be 22 and Elizabeth could have had her at 18, which makes El 40) and Peter is trying to get her away from Neal because Neal is a bad influence. Maybe Kate's involvement is what made Peter so determined to catch Neal in the first place. So, Peter talked Kate into leaving Neal but she left Neal clues to finding her because she's trying to double-cross both Peter and Neal and get Neal's stash. Maybe Kate's been working with Fowler and she's the one who's been in Peter's house to bug his phone and that's how he knows she's messing with him and Peter is now confronting her with the truth. Ta-dah. Whew.