Offset problem during cut

I’m having this issue where the spindle gets offset during a a job. It seems like it’s happening when the spindle lifts up to move onto the second cut. I’ve attached an image that shows the problem in detail. It’s a pair of sunglasses so they should obviously be symmetrical, but the image demonstrates how it gets offset.

Could this be occurring if the potentiometers aren’t perfectly calibrated?

I’ve calibrated the machine, tightened everything, but can’t seem to figure out why it’s happening. I’ve also tried with other designs and softwares so I know it’s the machine. Any help would be greatly appreciated!