I just saved one sim from Grim after she collapsed from freezing. She was on the street in front of my sim's house and my sim convinced her to group up. It would take too long to walk back to my sim...
Maybe one solution is for weather not to affect sims wearing athletic clothing, and for swimwear to also either ignore weather effects or be considered hot weather clothing. Jumping into the pool with your swimwear on should cool sims off!