You have it exactly correct, the shank size of your tool must match the collet size. The actual cutting diameter of the tool may be different from the shank size.
So if you only have a 1/4 inch collet, you will be able to find most cutting diameters you will need with the same 1/4 inch shank. But, they may be expensive. For some reason 1/4 inch shank size tools are much more expensive than the same tool with a 6mm shank.
So if you have a few different size collets you can then search eBay for the diameter tools you need and get them in the least expensive shank size.
So search eBay and get some great deals!