Plot hole: The ghosts cannot cross the front gate since they died on the mansion's property, however, the Viking and Native American died before the property was laid out, so they should not be limited by the later boundary lines.
Plot hole: Loki states while on trial that The Avengers should be on trial for traveling through time to change the timeline. How did he know they came back through time?
Suggested correction: He recognized that there were two Tony Starks in the lobby by the smell of their colognes, and combined with all the other unusual shenanigans going on, he correctly deduced the Avengers travelled through time, though he incorrectly thought it was to prevent his ascendancy.
How exactly do you distinguish the smell of cologne as belonging to two separate people? But besides that, it's pretty wild to jump to a conclusion about time travel when it could be that someone else happens to be wearing Axe.
I'm sure he simply recognized his voice.