Someone, as I recall, asked me between repeats of a swim workout something sarcastic of the "are you trying to make the Olympic team or something, ha ha ha" kind of question. And, no, I'm not fast, but why should that mean I'm not entitled to do interval workouts (on the track or in the pool) and the like (heh, I need them all the more!)? I just gave him a long, blank look and continued with my workout.
Another time, I was swimming at what was a pretty comfortable pace, and by no means fast at all (as in, I regularly finish at the end of the pack in any swim competitions), and some man in the next lane was scrambling as hard as he could (could tell by the splash level and his panting when he got to the wall) to pass me, which he did just before I reached the wall. When I came to the wall, I smiled at him as I turned to start my next length. Okay, man, whatever floats your boat.
I took these incidents for the displays of macho bravado that they seemed to be... but one rather troubling incident happened when I was running on a track and a woman I guessed to be in her 70s, very angrily said, "You're so much faster than I am, and that's because you're younger. And someday you'll get older too. I wish I could be young and fast like you" (note: again, my speed wouldn't have dazzled anyone), "but I'm not." And she went on in this fashion almost every time I went by her on the track. The troubling aspect of this for me was that I actually felt sorry for her. It didn't seem as if she was enjoying herself, and was taking it out on me, and it made me sad that she seemed so frustrated and unhappy.
Lots of strange responses to runners, that's for sure!