Can't visit neighbors

When I use the party boat to go to a neighbors town it loads to the town but when I chose a cake farm I get kicked off the game back to my home screen