Did the 1-2-3 block help you square the top of the part? It seems that it might actually get in the way, trying to rotate the part while maintaining contact with the block.

Since the movable vise jaw is not in contact with the 1-2-3 block, and the 1-2-3 block is only (?) to locate the edge of the part, couldn't you clamp the squared part, then slide the 1-2-3 block in to locate the edge, then remove the block before you did the drilling?