The effective permissions of a file created with the permissions 777 can be calculated considering the create mask and force create mode.
create mask = 711 implies that the permission bits are ANDed with 0711, i.e., only the owner can read, write, and execute.
force create mode = 750 implies that certain permission bits are always set, specifically 0750, i.e., read, write, and execute for the owner, and read and execute for the group.
The create mask reduces the permissions to 0711, and then force create mode adds the 0750 mask to the result.
Original permission: 777 AND with create mask (711): 711 OR with force create mode (750): 751 Thus, the effective permission is 751.
Reference:
Samba smb.conf man page - create mask