Travis Bickle: All the animals come out at night - whores, skunk pussies, buggers, queens, fairies, dopers, junkies, sick, venal. Someday a real rain will come and wash all this scum off the streets.
Travis Bickle: [Into the mirror.] You talkin' to me? You talkin' to me? You talkin' to me? Then who the hell else are you talkin' to? You talkin' to me? Well, I'm the only one here. Who the f**k do you think you're talking to? Oh, yeah? Okay! [whips out gun.] Huh?
Travis Bickle: Loneliness has followed me my whole life, everywhere. In bars, in cars, sidewalks, stores, everywhere. There's no escape. I'm God's lonely man.
Answer: This is just my opinion. Remember Iris's line "Have you ever tried looking at your own eyeballs in the mirror?" or something to that effect. Well, Travis sees the madness reflected in his own eyes, doesn't like what he sees (as it reminds him of what he has done, what he might yet do).