As I suspected, the set screw was broken. Probably a bad unit, failed when assembler over-tightened, and there we are. I've not removed it yet. Need to figure how to fix it to the mill... or just try to drill it by hand, though that option is not my first choice.