Continuity mistake: After Loki time jumps back to the interview room he drops the tesseract on floor. When he gets up it's not there or on the table. (00:36:53)
Plot hole: Loki states while on trial that The Avengers should be on trial for traveling through time to change the timeline. How did he know they came back through time?
Suggested correction: He recognized that there were two Tony Starks in the lobby by the smell of their colognes, and combined with all the other unusual shenanigans going on, he correctly deduced the Avengers travelled through time, though he incorrectly thought it was to prevent his ascendancy.
How exactly do you distinguish the smell of cologne as belonging to two separate people? But besides that, it's pretty wild to jump to a conclusion about time travel when it could be that someone else happens to be wearing Axe.
I'm sure he simply recognized his voice.