I have a question about something a fair amount of fantasy writers are afraid of...?

...their book being too similar to another popular book. In my case, it's Harry Potter. I had the idea for my book before I'd even read or seen the Harry Potter movies (I'm a fan now though! :) and it seems pretty similar not in the characters or...story to story plot line per se...but in the general plot of the series. It has a boy going to a place secret from this world where there is magic that humans can learn and yes, there is a school that teaches that because what society could function without a school haha. And I've tried adding details to it to make it different but the overview that I just said stays almost the same! I guess I'm asking... what are the limits at which point it's too recognizably similar to Harry Potter to look over or ignore. I mean if some people are saying Percy Jackson is similar to Harry Potter than my story certainly is...the issue is that magic (or so we'll call it) is essential to the plot and it's just great fun to write about. I have no idea what to do...I can't even begin to write the series (I have it all outlined) until I sort this out... Thanks for any advice and opinions! :)