Continuity mistake: After the horse race with Sam, Elsa and Sam are talking with bright high sun and shadows on Elsa's face. Moments later the the sun is low on the horizon in clouds. Then later in the same scene the sun is high and the strong shadows are on her face again.
Continuity mistake: When the trio encounter the law officers from the Wyoming Stock Growers Association, it starts snowing while they're talking to each other. After the gun fight, it's sunny.
Answer: Tim McGraw (the actor of James) put it best. McGraw has explained that James has PTSD from serving in the war. It doesn't help that, per McGraw, James was forced to serve and did not actually want to participate in it. This would make the war not only a shameful memory for him, but a painful one at that. Especially during a time when therapy was not a thing and people, particularly the male head of the households, were taught to hold feelings like that back. In his suffering brain, he thought it was best to try to put that memory aside rather than openly talk about it.