I had similar problems a while ago.
I use Logic X on MBPro, with Helix floor. I also use an RME babyface interface. In Logic my input device was Helix and output Babyface, as I just prefer using the volume control on the Babyface for monitors and headphones (the dim function is really useful).
I was having a similar issue as you, where initially everything would be great, then after an arbitrary amount of time, a huge amount of latency (2-3 seconds) would appear out of nowhere. Unplugging/rebooting either interface fixed the problem, but only temporarily.
The fix, for me anyway, was to create an aggregate device, with Helix in and Babyface out. Since using this, I've not had a single repeat of the problem!