Changing the boot order on the server and restricting console access would prevent unauthorized access attempts via USB devices at startup, as the server would not boot from any external media and only authorized users could access the console. Setting a BIOS/UEFI password on the server would also help, but it could be bypassed by resetting the CMOS battery or using a backdoor password. Configuring the host OS to deny login attempts via USB would not prevent booting from a malicious USB device that could compromise the system before the OS loads. Disabling all the USB ports on the server would limit functionality, as some peripherals or devices may need to use them. References:
https://www.pcmag.com/how-to/dont-plug-it-in-how-to-prevent-a-usb-attack
https://www.techopedia.com/definition/10362/boot-order
https://www.techopedia.com/definition/10361/console-access
https://www.techopedia.com/definition/102/bios-password
https://www.techopedia.com/definition/10363/cmos-battery