I suggest you test the device with 3 dial indicators to measure the tool motion (left-right, up-down and in-out) while in use.
I think I'm seeing quite a bit of undesirable motion, due to a combination of clearances and deflection when engaging the workpiece.
Keep working on it. The problems are solvable.