I know this sounds dumb, but what is the difference between physical and special damage?

I honestly couldn't tell you; for some characters it might be completely different. For some, it looks like physical with basic, special with special, but it isn't really a pattern anymore, at least not with the more recent characters. For the resistance heroes, I can see why the devs would use special damage; they were designed when the 501st were meta and 5s has a much higher defense than resistance, so special damage would be more effective in in defeating him. Galactic Legend Rey uses physical damage, but she often isn't actually attacking, and when she is, she is using sudden whirlwind, so armor is irrelevant at that point.