I know this thread is a little dated, but I think one important aspect of getting good sound out of the Variax got overlooked.
I recently purchased a JTV-89F (upgraded to latest firmware 2.22), I plugged it into a Blackstar practice amp, as well as a Mesa Mini Rectifier / Mesa V30 1x12 cab.
I got the exact same poor results as the OP, the JTF-89F using mag pickups sounded great and the models sounded like crap, I also compared to a Strat and LesPaul, and both those sounded way better on both amps than the JTV models.
1) I got a Helix (still did not sound great out of the box), I was using headphones as output initially
2) I upgraded the Helix firmware to latest (Nov 2017) and stock presets - big improvement in sound
3) Then I got some of the really good custom presets from CustomTone - massive improvement in sound
4) Then I got a Line6 StageSource L3T FRFR speaker, and the result is now my JTV-89F models (Spank/Lester) sound way better than my Strat, LesPaul or the JTV in Mag mode. I still use JTV in Mag mode for some metal songs, but generally use the models most of the time. I was always after the Hendrix/Marshall/Strat sound, and the above setup with the RHCP Frusciante patch from Custom Tone sounds awesome to me
At one point I considered getting a Marshall DSL5C for home just to dial in that tone when I needed, but after getting the Line6 setup right, I don't think I need to do that any more. I am also not playing my strat any more at all, just going to the JTV-89F.
The big takeaways are
1) Update Firmware
2) Get good custom patches for Helix
3) Get a really good FRFR output from the Helix (StageSource, Atomic CLR, etc...) there are other forums on that.
There are many other forums pointing to the issue of using a modelled guitar through a regular guitar amp, and poor results. I think once you go modelled, you gotta go with a good FRFR.
And there are lots of other forums on this, but Atomic gets you an FRFR that has the character of a guitar cab, whereas the Line6 speaker is just a really true and powerful FRFR. The interesting thing is it has a 2x12 guitar cab model mode, and that also sounds like crap, and what sounds good is putting in in Reference Monitor or PA mode.
Hope that helps.