I'm not sure I understand the problem, so here is what I found while trying to make your design, let me know if this helps.
For certain bit sizes, the inlay app combines the different parts of the design together (I think this is what you mean @TysonSwan). Here is a screenshot of exactly what I'm seeing:
The way the inlay generator works internally is that it makes repeated offsets to the design outwards and then inwards (and vice versa), which has the effect of rounding the edges which would be sharp and uncuttable. However because the parts are so close together, the offset of one section "bleeds" into the other, combining the shapes. I think it indicates that the male part simply wouldn't be able to be cut (the bit couldn't get inside the white spaces). The problem, as you showed, isn't with the female part (i.e. milling the black part) but with the male part (i.e. cutting around the black part).
Using a smaller bit should help. Here is my result using a 1/16" bit:
Another option would be to modify the design slightly. For example, using the "Offset" app to offset the design inwards slightly would be a way to add more whitespace as @Mike suggested.
@TysonSwan does that help?