There's not really anything you can do on the M5 itself. Like the vast majority of stompboxes, It has instrument level ins and outs. So if the signal it's getting from your amp's effects send is too hot, you either need to try to reduce the level of that signal somewhat by lowering the preamp level of the amp or investing in something like this.

The Growler effect, along with a few of the other filter models, is very sensitive to the input level, particularly the attack. I've found that to get that even to use that effect to its full capacity in front of an amp, I have to roll back my guitar volume.