After 4 or 5 there will be no impact to the enemy.
This is slightly overstating. After 10-12 there will be no observable effect. After 20 there should be zero effect at all, observable or not. If you've got 4 stacks of Armor Shred then between those 4 and when you get to effective maximum the best you can hope for is about +8% damage compared to that inflicted at 4 stacks. And that's assuming a tank enemy, with lots of armour to shred. If your enemy is an attacker at 4 stacks of Shred, you might already be at 97%+ of the damage you can ever possibly achieve, even with 20+ stacks.
Whether there is still potential for noticeable improvement depends on how hard you're looking, but also other things -- some characters (like Han and Lando) have high damage variability. Even at 4 stacks the amount of damage variability will be significantly higher than that 3-8% possible remaining improvement in damage.
In other words, at 4 stacks you might not have gotten every last point of improvement you could ever hope to have, but you've gotten enough that natural damage variability might be larger than any remaining improvement. At 5 stacks this will be true even with tanks. So at 5 stacks you can expect RNG to have a bigger effect than another stack of Shred.