I mean, the load times aren't very long unless you are playing from a PS4 or possibly low end PC. The game does the extra step after fast traveling to the Ferry or the Stadium because it loads a new instance and if u have a party, asks if u want to bring the party with u. Personally, this has never bothered me. To avoid the load times and waiting u could also skate to the Stadium after arriving in San Van via the Ferry to get some skate time in and not have to wait for a loading screen that normally only takes about 5seconds to load anyway.
This doesn't mean the game is without annoying mechanics.
One mechanic that I find annoying is how the game starts u on ur board when doing missions or challenges. I'd much rather start the missions or challenges off my board, but changing this would require reworking the start points of some challenges since they would have to take the extra possible steps into consideration.
Another mechanic is how the marker to start the Own the Spot challenges in the Stadium aren't located where u are standing after it is over. It's annoying having to go back to the marker to start the throwdown again after it's done. Those markers should be where ur skater is standing after the throwdown is over.