Check that your spindle and/or bit is not loose.
Got flat piece of pine or poplar?
Try a test like this. If the first layer is off, and all others are stepped 1mm down, it’s a “smart” clamp issue (not unheard of). If they’re all too deep, there are other issues. Knowing exactly how far off makes a difference.