Retreat is a special command available to most units that is used to have have the selected squad attempt to remove itself from battle. When commanded to retreat, the selected squad(s) will stop combat and run straight back to your headquarters. During this time, the squad will break and become immune to suppression, take 80% less damage from ranged attacks, 30% more damage from melee attacks, and will gradually move faster over time on the way back to the base.
Retreat should be carefully used when near or surrounded by melee squads, as it does indeed cause melee to deal more damage. Calling for a retreat while surrounded may very well only ensure that your squad dies faster. Use any available ability such as grenades, jumps, or knockdowns before retreating to ensure your squad has a clear path to the base.