This really isn't all that unusual. Your taps are going to be off a tiny little bit from the metronome, and so with each measure that passes, those tiny little bits add up. After a minute or so, those differences will show up. The tap tempo doesn't actually interact with the looper in any way, but trying to get the looper to be in sync with a metronome is probably a lost cause. As far as tapped effects, most of the time these little difference don't really matter. Your essentially re-syncing the effect with the metronome everytime you play a new note.