Explanation
The command grub-install is used to install GRUB on a device or a partition. The option --root-directory specifies the directory where GRUB images are put. The default is /boot/grub. The argument /dev/sda specifies the device where GRUB is installed, usually the Master Boot Record (MBR) of the disk. Therefore, the command grub-install --root-directory=/custom-grub /dev/sda will install GRUB on the MBR of /dev/sda and put the GRUB images under the directory /custom-grub/boot/grub. This means that the new configuration files, such as grub.cfg, will be found in the directory /custom-grub/boot/grub.
References:
* GNU GRUB Manual 2.06: Installing GRUB using grub-install
* Installing GRUB using grub-install - GNU GRUB Manual 0.97