I'm updating this thread because I finally remembered how to fix this. omg it took me long enough a year later LOL. But I finally searched everywhere for an answer and the old answer had gone missing on any of the old sites like MTS etc. Long gone are the day we can find a fast answer for TS2 problems.
Even though I am using Graphic Rule Maker so TS2UC can see my better video card it would not let me video capture while in game with smooth edges enabled. Setting my video card in it's own panel to allow AntiAliasing didn't work, either.
It's really simple and it took me forever to remember. Literally. :s
Set the 0 in Video Capture to 1 inside the program files not the documents file. This is not even located where the Graphic Rule maker adjusts and changes the Config or CSConfig files. It is located where most people wouldn't look, in User Data inside the program files.
I knew I would finally remember when necessity demanded it. lol
Harddrive, C (if that is where you have TS2UC installed) Program Files x86 (if you have Windows 7) Find The Sims 2 Ulitmate Collection file. If you have Origin Installed it will be inside the Origin Games folder.
Open it to see Fun With Pets listed/SP9 (open the SP9 folder)
TSData/Res/UserData/Config/Video Capture. Open the Video Capture file scroll to bottom where it says Antialiasing change the Zero to 1. Save. We no longer have to do this for every EP or SP with TS2UC. Just inside the Userdata of SP9.
ETA: Sorry I don't know where these files are located in Windows 10. I sent 10 back. :D
