Deliberate mistake: Leaven spends too much time thinking about if 645 and 372 are prime numbers to be believable. But she knows the factors of 649 and the result of 26 to the power of 3 almost immediately. Plus anyone with any maths knowledge whatsoever would know that any even number (or one divisible by 5) can't be prime, so 645 and 372 could be dismissed without even thinking about it.
Plot hole: Kazam is supposed to be a math genius, something that the plot hinges on, but he makes several mistakes when calculating the number of prime factors to find out whether a room is trapped or not. He says that 462 has three prime factors, when it has four, that 206 has four when it only has two and that 563 has two and 911 has three when both are actually prime numbers.
Other mistake: Toward the beginning, when the characters wonder aloud what the rumbling noise is, someone suggests that it's the ventilation system, to which Rennes responds "No vents." However, in the passages between rooms there are panels with ventilation slits in them (which you can plainly see even as Rennes says the line), and it's pretty impossible that the characteristically level-headed and observant Rennes wouldn't have noticed that.
Continuity mistake: Throughout the movie they are told to suck on the buttons from their shirts. In one scene, once Holloway climbs into the next room, Quentin is sucking on his button. He immediately speaks to Worth, muddling some words because of the button in his mouth. All of a sudden, the button disappears from his mouth and is in his hand again, and he then puts it back into his mouth.
Continuity mistake: When Holloway is hanging outside the Cube from the rope of clothes, it would seem that it disappears at the moment she falls. We see her climbing up toward Quentin, and the rope is around her waist. Then they're holding hands face-to-face, and in the next shot when she slips, the rope stays still. Then, as Quentin goes back inside, it's disappeared entirely, and he doesn't take it back in with him.