You mentioned that you are using the 1/8" collet adapter. I can see the adapter sticking out the bottom of the Dewalt 611 in your image.
When doing so the bottom of the collet adapter should be flush with the bottom of the 1/4" collet for the Dewalt 611. Without them being flush at the bottom with one another the proper clamping forces are not being applied where they should be and the bit could flex the collet adapter horizontally.
I use the standard Dewalt collet with the Inventables 1/8" adapter, tighten the ever-living snot out of it every time, and I have never had a bit snap inside the collet.
There should be no stick-out of the adapter from the bottom; the two should be flush with one another.
My best artwork yet… You get the idea though…
As other have suggested I would use an upcut bit as well.