Color Damage Question?

What is the difference between white, yellow, red, green and blue damage numbers?

Do the red damage numbers produce negative numbers that count against your overall damage score?