In Cisco Secure Endpoint, to create a custom detection file list for detecting and quarantining future files, an advanced custom detection should be created, and the hash of each file to be detected and quarantined should be uploaded. This allows the system to uniquely identify and take action on files based on their hash values, providing a precise method for targeting specific malicious or unwanted files.