Every fish adjusts at their own speed, just as any other animal with a personality.SpeedFreak said:
The fish will probably "freak out" the first couple times you do something around them that they don't know whether or not they are in danger. Once they learn it is not a dangerous situation, they should stay calm.
When you say "look horrible", do you mean not colored? Or abused? It is not uncommon for (usually) a dominant male or two to start showing color before the rest - at least that is my experience.