Could it be the bit deflecting during the initial plunge cut?
What is your Plunge rate?
The max a stock XC is rated at is 500mm/min (19.6 inch/min)
(Edit its mm/min not sec - oops!)
What kind of bit are you using? Spiral? (you want to make sure your bits are made to do plunge cutting. Most are but it is a good idea to be sure)
You might want to try the Ramp Plunge moves option. Depending on the cut operation you may have different choices.
I tend to use fixed angle or fixed distance and avoid spiral (constant ramping over the whole operation). As ramping slows the whole cut down to the plunge rate. and the greater the ramping the more time it adds.
(The first time I tried it I selected spiral and could not figure out why my 10 min cut was now well over a hour.)
This is a long shot but make sure the bottom of the bit has a sharp edge to it for plunge cutting.
I recommend going to the hardware store and picking up a 1/4" spiral upcut and a 1/4" spiral down cut bit.
The upcut will be your go to bit for most woods and the downcut for plywood and MDF.
(Unrelated tip I also recommend a Dust Deputy for your shop vac, it makes a HUGE difference with the fine dust that otherwise blows right through the vac. Especially if you are working with MDF)