Carve Issue with cribbage holes

I when drilling the 1/8 inch holes they are coming out larger on top and step down to 1/8 half way down. It is suppose to drill straight down but i have a bigger hole on the top.

Probably not rigid enough at initial entry.