Solution 1 - The trick is to strategically place your smaller-holed key to avoid the 4mm shaft. This however means you need to have a few other keys / USB key / bottle opener to fill up the 5mm shaft space as shown
Solution 2 - You can enlarge the hole size using a file or drill. It requires a bit of extra effort, but with the right tool it takes just under 1 minute!
Solution 3 - As a last resort, you can always utilise the optional attachment for anything that doesn't fit.