I'm late to the thread, since I've just recently checked the forums about my problems. And I want to confirm you are not going batty!
I have your exact issue, using my American Deluxe Tele and Ibanez Talman Prestige. I can use stock patches or my own, to a PA or through headphones. In any case I get what sounds like digital clipping/distortion in patches with any reasonable preamp gain. Amps that in the "real world" get creamy as they overdrive produce this nasty noise on the Helix, and it really doesn't remove it when I mess with the the EQ - even if I get extreme and destroy the air and presence of the patch.
The only solution I have found is to run my Helix into the power amp of my Kustom Coupe '36 tube amp - defeating the amp/cab settings on the Helix. I know that kills off getting full range output, and it's the tube power amp and cab that are taming the bad noise.
Bad news - Sorry, I have no sage advise/help for your issue. But I would LOVE for Line6 to chime in with some input!
Good news - the Helix/Kustom combo sounds very good. I know that's like saying my really expensive FX sound great as long as you run them through an already great sounding amp.
So, overall I'm really disappointed with the the Helix. Frankly, all those incredibly cool features aren't worth it if I can't run it into a PA and get good tone. So, I've worked with this beastie for about 3 months and still go to every gig without it.