Well, at least in my experience, the conferences consist of a core of lectures - which can be character debates, live podcasts, Harry Potter Alliance rallies, etc - as well as a bunch of "informal" fun stuff, like quidditch, wizard rock, fandom movie screenings, a yule ball, "meet-ups", etc. C'est enorme!
Do you live in or close to the UK? Because although it's true that most Harry Potter conferences and events are in the U.S., I've heard there are similar fandom events in England.
And, of course, though it's not exactly ideal financially, there are always the people who fly to the conferences. I met quite a few international attendees at Infinitus.