You need to configure the fraud alert settings.
Fraud alert setting literally has an option to configure it to automatically block.
To enable and configure fraud alerts, complete the following steps:
1. Go to Azure Active Directory > Security > MFA > Fraud alert.
2. Set Allow users to submit fraud alerts to On.
3. Configure the Automatically block users who report fraud or Code to report fraud during initial greeting setting as needed.
4. Select Save.
Reference:
https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa- mfasettings#fraud-alert