Even in a milling machine with all axes, apart from the quill movement, locked I start with a centre drill.
Drill gently to ensure that the centre is perfectly aligned and drill into the material forming a small chamfered edge.
Drill the hole through the workpiece using a drill that is slightly under size. I normally use a drill that is 0.5mm smaller diameter than the final hole size required.
Use cutting fluid if appropriate as this will reduce the temperature of the cut and avoid any local work hardening.
Finally drill the part to the actual size required. Again, use cutting fluid to reduce the heat and take a reasonably gentle cut.