Actually you do need the download manager to install patches, updates, CAW worlds, and new eps and sps. Also if EA makes a new world like they did Riverview, the launcher request the EADM or doesn't want to install it. It should only be active when your game needs updating anyway or when it's called upon to activate - which most of the time the launcher is doing the calling. Uncheck automatic updates on the launcher's update page - then you can decide when you want the EADM to activate.
Also when EA updates the EADM, it will then activate as well usually.