If you do start fresh, I would put your original downloads folder onto an external drive you can still access. That way if there's something you find you can't live without, you can grab it without having to worry about the site you got it from still being around.
I doubt it on whether they cause any slowness. But if you aren't using them, it wouldn't hurt to move them to a separate location anyway.
There's not a limit to the number of subfolders you can have. There is likely a limit on how many levels you can do. And a path length of 256 characters (including the \'s and the C:\
Special characters and spaces slow down the game (CC file names are notorious for this)
The more subfolders deep you go can also affect things as well.