Factual error: Jack Sparrow says he loves the French because "they invented mayonnaise." The film is set in the 1750s. The egg-based sauce known as mayonnaise was developed in the early 1800s. The Oxford English Dictionary says the word dates from 1815.
Factual error: The governor of the island claimed the bank safe (roughly 2 meters square on a side and with walls about 13 cm thick) weighed "an imperial ton." A safe of this size would weigh nearly 22.5 imperial tons.
Factual error: The British Ship of the Line is flying a version of the British Union Jack that was not adopted until 1801. The story is supposed to take place between 1720 and 1750.
Answer: He's probably worried that either the crew will attack or at very least frighten Henry. Or he just doesn't want Henry to see how horribly the curse can affect people.
But because Will is ferrying souls to the after life, like Davy Jones should have done for the full time, he was captain of the Dutchman. There is no curse that would make the men look like sea monsters. They only started to look like that because Jones denied his duties after 10 years of service, when Calipso didn't show up. So the crew would look like normal men, so Will being worried that Henry will be frightened by them, can't be the reason he started to panic.