Revealing mistake: In the scene, when the bug (actually another Kothoga monster) run out from the box that Margo has opened due to strange noises, there are visible wheels from the puppet of the bug. (00:48:40)
Revealing mistake: When the Kothoga grabs the one SWAT member and starts to rip his head off, it's obvious that the guy is really CGI.






Answer: More then likely, the dog was too afraid of the Kothoga to attack it and when the Kothoga saw this, it let the dog live. The other dog was probably unafraid and tried to attack the Kothoga which prompted it to kill the dog; not out of self-defense, but, because there were still people trapped in the museum and the dog was impeding his attack.