Other mistake: While inspecting the gems from the robbery, Stephen Baldwin is looking through the wrong end of the jewellers magnifying glass.
Other mistake: The burned guy who's supposed to be Hungarian keeps making bad language mistakes, although his pronunciation is great.
Answer: Keaton doesn't believe in Keyser Soze, and is so convinced that Kobayashi is the one pulling the strings and manipulating them. They figure that if they take HIM out, they are home free.