While poly count is one thing, texture size is another. If you have lots of clutter objects, all with 4k textures, then you'll notice lag too.
A good example would be in Skyrim- the Riften region has all those little falling leaves, and for some reason, Bethesda made them all high res textures for their size (256 x 256 or 512). Just thousands of these little high res leaves fluttering around would cause framerate drops in that area. If you replaced the leaf textures with a 32 x 32 or 64 x 64 one, there was a huge jump in performance and you couldn't even tell because the leaves were so small.