I found Amy more annoying than Rory.
Rory has some wonderful moments.
When he enters the Tardis for the first time (vampires of venice I think) and the doctors ready to be all smug in the face of human incomprehension and Rory's like, oh so yeah. Different dimension inside then? (I'm paraphrasing) When the doctor is surprised at Rory's lack of reaction, Rory explains he did his research and read up on science theory after their first encounter. How cool is that!
[edit, now with extra added quotage!]
The Doctor: It's a lot to take in, isn't it? Tiny box, huge room inside; what's that about? Let me explain...
Rory: It's another dimension.
The Doctor: It's basically another dimen- what?
Rory: After what happened with Prisoner Zero, I've been reading up on all the latest scientific theories. FTL travel, parallel universes.
And when he says "this isn't fair, you're turning me into you" when the doctor asks him to choose between young amy and old amy in 'The girl who waited.'