Ernest T. Bass Joins the Army - S4-E3
Revealing mistake: When Barney picks Andy up and they get back to the courthouse, they find Ernest T out of the jail cell again and breaking more windows, and in the shot of Ernest T leaping in the air we see it's actually a flipped shot - note the backwards "snack bar" in the window (that store is later when Ernest T tries to break the new glass for the courthouse).
Answer: Those "boxes" are valences that used to be very common, before air conditioning. They allow for windows to be open during rain storms. They permit air circulation, without letting the rain in.