I use scipy.  It's a much looser toolchain than you're expecting if you're coming from Matlab, but it's totally sufficient for audio processing.  The graphs come out decent and are zoomable similar to Matlab's, which is important when you're hunting for glitches in your algorithm:

I've only used scipy on Linux, I'm not sure what the Windows support is like but I imagine it's ok.

Once I've got the algorithm working I copy my Python code and convert it to assembly line by line... the poor man's compiler .  This is a 1000 times better than coding up some assembly and wondering why it makes clicks once in a while.