"To fit all my taps, I made 3 different sizes. Each one has 2 different sized holes so you pick the one that fits best. These fit everything from my 4-40 up to 1/2 inch taps."

I'd be tempted to drill the six different sized holes into a single threaded rod connector. Weakening of the connector likely inconsequential. One tool instead of three.