Trivia: Producers Gale Anne Hurd and Sam Mercer wanted to film the movie at the American Museum of Natural History in New York. However, the museum's administration was afraid that the film would not only cast the museum in an unflattering light, but it would also scare kids away from the museum. They were given permission by the Field Museum of Natural History in Chicago to film there, because they loved the movie's premise.
Trivia: The Kathoga has only five minutes of screen time in the entire film.
Answer: More then likely, the dog was too afraid of the Kothoga to attack it and when the Kothoga saw this, it let the dog live. The other dog was probably unafraid and tried to attack the Kothoga which prompted it to kill the dog; not out of self-defense, but, because there were still people trapped in the museum and the dog was impeding his attack.