Garden Ops> Playing on same internet/same home> Losing connection to session

My brother and i were trying to play some PvZ GW since i bought it in this most recent sale, we were playing Garden Ops over the same internet connection, in the same home, and he was hosting the game. Every game we played 🏀 zi lagged out probably about 6 times each match, I was able to rejoin from the invite he had sent me before we started playing. We have no idea why I was constantly losing connection to the session happening within my home, It most definitely isnt our internet connection seeing how we have some of the best internet modems and routers, etc. 
Every time i lagged out of the match it would tell me i lost connection to the session i was in, and if i was planting something in a pot or using a revive as i lagged out i would permanently lose it, the lag spikes would happen sporadically and I had no control over them.
I just want to play PvZ GW with my brother, please someone help.