All anime guys do this thing where they touch their glasses like this, I don't know why.
At the risk of taking a joke too seriously, that's a difference of habit between North America and Japan. In N.A., when our glasses are sliding down our noses, we tend to push them back up by the bridge, while the Japanese tend to push them back up by the side.
It's similar to what was shown in "Inglorious Bastards", when ordering three beers, we would hold up the first three fingers, while Europeans hold up the thumb and first TWO fingers.
Strange, it must be a manga thing so that the face is not obscured in the drawing because in the Anime series I have watched they push them up by the bridge, most likely for easier cell animation even though cells are computer rendered and then laser printed these days by most studios.
Here is the telling thing I have noticed though, if the character is any sort of jerk, or in the process of behaving like one in that scene, they use their middle finger, yet if they are not a bastard, will use three even though it looks weird.