Trying to cut steel on Homemade Milling Machine

[Frank S]

Your mill looks like a regular drill press with a 30 taper. A fixed head and a standard spindle pull down as nearly all drill presses have.
Your 2 main problems are not your x y table or motor HP. The XY table can usually be tightened up even though it was made for a drill press.
Your ER collet adapter is way too long placing the end mill very far from the bottom quill bearing. Not sure if you can even install a larger diameter spindle in the quill.
From the video it looked like you were taking a fairly deep cut these machines don't do well with deep cuts because the head has a short distance between the bottom and the top spindle bearings, also the column does not have the mass to prevent flex.
If you will focus on limiting the depth of cut per pass and always feed into the cut, get some tooling to place the end mill closer to the bottom of the spindle you should notice better results. It will never be able to make large deep cuts, but even a 3 HP a J head Bridgeport pattern mill has its limitations as well