How many passes have you made?
If there is a significant slope to the waste board on the first pass, you could be pushing through more material at one end than at the other. This might cause some deflection. The second and third passes might be a bit smoother.
I also wonder if it would be worthwhile running a spiral box path to see what difference that makes.