How Many Stars Does It Take to Level Up With That Fish?

I was under the impression, that you would get one level for each visit you paid him. However, when I tried him out, my XP barely increased. It seems like it would take an exorbitant amount of stars ...