Harry Potter is going to be a reference for pretty much anything magic-related for a while. It's a big popular series (franchise), it's everywhere in current pop culture and there are very few people who don't know about it, whether they like it or not.
None of that makes RoM a cash grab or a copy or a rip-off of HP. There's no such thing as an original idea and people's minds will look for the familiar in anything they see. With magic, because of how vastly spread the HP world is, it's unsurprising that it's what people relate it to.
Sure, the town square can be seen as Diagon Alley, the broomstick flying and duels and wands too. But none of those are exclusive to HP, never have been. Personally, I like the connections to HP (intentional or not) as I'm a fan.