My machine out of the box did this as well. About the same amount of deviation even with the lightest cuts. I tested the machine accuracy it by doing test traces with a fineliner on paper (so essentially a plotter) in place of the spindle, so essentially no drag. Dimensions were perfect with a pen. Here’s a thread on that: Everything i cut is undersized - #13 by TravisBrown1
So I threw out the garbage Z-axis that came with the machine and built a new one. Stiffened up the X and Y. Changed to 9mm belts. Problem went away. So basically, make the X-Carve what it should have been out of the box.
Changing the Z-axis slide will be the biggest step to getting you more accurate. You can side-step other deficiencies with lighter passes.
Also, make sure you have sufficient and even tension in your belts. I go for 5lbs @ 1.5 inches. And calibrate your steps after.