One more thing with this problem is that, for a tone where the amp model is disabled, when switched to this tone from other ones, it turns out different as well.
What I am guessing is that, when the amp model is turned off by the foot switch or through the edit software, there is something left from the amp model in the signal path, or in other words, the amp model is not completely disabled or turned off. It still affects the tone somehow, even when it is "turned off".
I guess that the amp model gets completely turned off when the "fx only" bottom is pressed. That's probably why the the tone doesn't change when the "fx only" bottom is pressed.
Line 6 staffs might get unhappy when they see customers choose not to use the amp model that they have devoted so much effort into... probably this is some kind of message from them saying "get back to our amp models!" haha...