I think I've said this before in other similar posts but there's no harm in re-posting I guess. The multiplayer lobby system works in that way that one player is always the host (has a crown next to his name). This means that you are relying on this persons connection which in most cases makes everyone lag skip around the map and making any form of interaction mostly impossible.
Whenever that person either leaves the game or loses the connection to internet, all players in the lobby will be "disconnected" and taken to the host migration screen where someone else is selected as a new host. Not only, as you're experiencing, will this "pause" the event you're in but there's also a big chance that the new host had the same, got frustrated and decides to leave the game as well. New host migration screen.
Since there's no multiplayer component anyway (more or less impossible to get people to join you) I strongly would suggest just to turn it off completely by going into the menu, select all drive and change to single player instead of public. This will rid the constant disconnecting and you will also not have anyone else possibly interfering with your own progression in terms of single player.
Only time I would suggest a public lobby is if you're just looking for busting other people as a cop, other than that it's completely useless unless you have a few friends online and you get into a private lobby to do some "real" multiplayer racing.