@MichaelMellDaGrl Hello! That's really annoying! I checked on my PS4 and did not see that issue at first. I asked some friends about this and one of them said that this has happened for her when she went into CAS while one of the sims was wearing Athletic clothes. I directed my mom sim to change into her Athletic outfit and then used Change Sim on a mirror. The Athletic filter showed for the infant at that point. I exited CAS, had the mom change into Everyday and used Change Sim again. This time the Athletic filter did not appear for the infant.
So, double check the other household sims to see if any are wearing Athletic clothes and have them change into Everyday before going into CAS. Please come back and update if that helps because I think this is a bug.
By the way, I also use a mouse and keyboard with my game at times. I was able to clear the filter using my mouse. So, you might try that as a workaround also if you have a plug and play mouse that you can plug into the USB port on the back of the system.
EDIT: I just noticed that if the Infant sim is the active sim when going into CAS, the filter is not applied. So perhaps you could also try selecting the Infant first, then click on a mirror and choose Plan Outifts.