Continuity mistake: When the suspects are put into a jail cell together after their interrogation, Keaton walks into the cell. He is wearing his suit jacket in the shot where he walks through the cell door, but in the next shot, where he walks to a bench to sit down, his jacket is now slung over his shoulders.
Other mistake: While inspecting the gems from the robbery, Stephen Baldwin is looking through the wrong end of the jewellers magnifying glass.
Continuity mistake: Near the end, when Agent Kujan realizes who Soze is, he drops his coffee cup and it breaks in slo-mo. If you look closely, you can see that the bottom of the cup does not have the red writing on it. (I will not give away what the red writing is, as it may be a spoiler).
Continuity mistake: When the suspects are in the garage after Saul, Saul drops a gun and Keaton kicks it over to McManus. However in the next shot McManus picks up a different type of gun.
Continuity mistake: When we first see the Hungarian guy lying in the hospital he is burned over most of his face, however his eyebrow is still there. In a later scene, his eyebrow is gone but his eyelashes are still there.
Continuity mistake: When McManus and Keaton are clearing out the boat, the time of day outside (as shown by the daylight) changes from shot to shot. It should be dark during the entire scene.
Plot hole: The premise is that the back-story is a fabrication, including the five being placed together in the police line-up and in custody in order to get them to work together. But in the flashback of Keaton in the restaurant, it is Dave Kujan who takes Keaton in. It is inconceivable that Kujan would be complicit in the staging of the line-up (and doesn't fit with the plot). And if Verbal is inventing it, then Kujan is going to know it didn't happen. It doesn't make sense and undermines the plot.
Revealing mistake: In the flashback, when the boy has his throat slit, there's no blood or wound.
Other mistake: The burned guy who's supposed to be Hungarian keeps making bad language mistakes, although his pronunciation is great.
Answer: Keaton doesn't believe in Keyser Soze, and is so convinced that Kobayashi is the one pulling the strings and manipulating them. They figure that if they take HIM out, they are home free.