Factual error: The governor of the island claimed the bank safe (roughly 2 meters square on a side and with walls about 13 cm thick) weighed "an imperial ton." A safe of this size would weigh nearly 22.5 imperial tons.
Factual error: The British Ship of the Line is flying a version of the British Union Jack that was not adopted until 1801. The story is supposed to take place between 1720 and 1750.
Continuity mistake: After the bald witch walks towards a wall with stars painted on them, there's a shot of Henry threatening Jack with his sword. The grip on the sword and the position of the arm are not continuous between shots.
Answer: He's probably worried that either the crew will attack or at very least frighten Henry. Or he just doesn't want Henry to see how horribly the curse can affect people.
But because Will is ferrying souls to the after life, like Davy Jones should have done for the full time, he was captain of the Dutchman. There is no curse that would make the men look like sea monsters. They only started to look like that because Jones denied his duties after 10 years of service, when Calipso didn't show up. So the crew would look like normal men, so Will being worried that Henry will be frightened by them, can't be the reason he started to panic.