Sadness: You could get lost in there!
Joy: Think positive!
Sadness: Okay, I'm positive you will get lost in there.
Joy and Sadness are stuck outside of the control center. They are trying to figure out how to get back, and encounter maintenance workers who are discarding old memories. The maintenance workers show that they have the power to send memories back up to the control center to be played. Why couldn't they send the core memories that Joy had back up the same way? Better yet, why not use that method to send Joy and Sadness back up to the control center? The director of the film is even aware of the plot hole, and said "Yeah, well then we wouldn't have a third act," before explaining how the idea of recalling memories was added in later, "box[ing] [the screenwriters] in a corner a little bit."