Have you considered playing with your controller config and customisation settings? You'd be surprised how different your car will handle in terms of quicker direction changes and smoother inputs when you play with steering rate, max wheel rotation, steering saturation and linearity. You can even have multiple controller profiles for various track types.
If you simply use the default settings and use it on all tracks, you're not going to maximise opportunity for laptime. Wheel users are not just automatically quicker because they're on a wheel. Almost all wheel users have custom profiles and adapt the settings to suit their driving style. You can do the same on a controller!
Want to make TC Off easier on a controller? Play with your throttle linearity/saturation and give yourself more trigger travel before the car bites. Same with the brakes to avoid locking up without ABS. There is so much you can do! Everyone has different preferences, driving style and reaction times. Default settings are never going to be as fast!