It looks like I have finally fixed the problem.
Using LiveTcpUdpWatch to look for the IP whenever I got into ~200 ms ping servers.
I found it was always an IP in the range of 185.225.209.XXX.
IP range 185.225.208.0 - 185.225.211.255 belongs to US-VPS-NET which is a VPS.net Cloud Service.
So no wonder the ping is high. Why Apex is connecting me to that IP once in a while I have no idea.
Solution :
I created an Inbound rule (in defender firewall) blocking 185.225.209.0 - 185.225.209.255, since the IP's I have seen was always in the 185.225.209.XXX. range.
I also created an outbound rule with the same IP range inbound rule on it's own didn't work. In the outbound rule I blocked any app/program/service not just r5apex.exe
After playing several hours in the last couple of days I haven't been connected to a single high ping game and I have had two code leaf events. Usually I never see code leaf.
So I will keep playing and if I get any high ping games I will update this post with the IP. But for now it seems to have solved the problem.