I run into this problem quite a bit. Lets say I have an existing hole that is 1" in diameter. I need it to be 1 1/2" in diameter. Using a hole saw is the best way to cut the new diameter but the existing hole is way to big to allow using the typical 1/4" diameter drill to guide the saw. This is what I do to get around that problem.
I use a scrap piece of deck board, 2 x 4 or 2 x 6 to guide the saw.
Cut a hole thru the scrap with the saw you need to use. Clamp the new "drill guide" to your work surface in position over the existing hole. Now cut the new hole with the hole saw. here I am using a Rota-Broach.
No wandering or sloppy cuts. Makes for a nice cut right where you positioned it. I use these when I need a nice cut in thin sheet metal.