Q: What is the mechanism that decides that setgroups() and initgroups() must be called with the superuser? From APUE

#include <grp.h> /* on Linux */
int setgroups(int ngroups, const gid_t grouplist[]);

The setgroups function can be called by the superuser to set the supplementary group  ID  list  for  the  calling  process: grouplist contains  the  array  of  group  IDs,  and
  ngroups specifies the number of elements in the array. The value of ngroups cannot be larger than NGROUPS_MAX.
#include <grp.h> /* on Linux and Solaris */
int initgroups(const char *username, gid_t basegid);

One must  be  superuser  to  call initgroups(), since  it  calls setgroups().

What is the mechanism that decides that setgroups() and initgroups() must be called with the superuser?
By "mechanism", I mean something similar to or as a counterpart  to the following. in Linux, I  learned that we can determine accessibility of a process to a file, based on access control list (ACL):


*

*the effective user ID, effective group ID and supplementary group IDs of the process and

*the permission bits of the file.
Thanks.

A: You can look at the actual code for setgroups. It starts like this:
SYSCALL_DEFINE2(setgroups, int, gidsetsize, gid_t __user *, grouplist)
{
    struct group_info *group_info;
    int retval;

    if (!may_setgroups())
        return -EPERM;

    // other stuff
}

And here is may_setgroups:
bool may_setgroups(void)
{
    struct user_namespace *user_ns = current_user_ns();

    return ns_capable(user_ns, CAP_SETGID) &&
        userns_may_setgroups(user_ns);
}

And here is userns_may_setgroups, if you aren't using user namespaces:
static inline bool userns_may_setgroups(const struct user_namespace *ns)
{
    return true;
}

So: setgroups will return an EPERM error if you don't have the CAP_SETGID capability.
Corollary: You can call setgroups if you have the CAP_SETGID capability.
Root has all capabilities automatically (so root can call setgroups), but you can also call it if you are not root and you have the CAP_SETGID capability.

A: The wording in APUE is perhaps a bit imprecise; it's a tutorial guide for programmers, not a legal compendium. Any process can call setgroups() or initgroups(). But if the process does not have root privileges, the function will do nothing more than set errno to EPERM and return an error indication.
Obviously, the permissions test cannot be done with user privileges. The library function is just a thin wrapper around a syscall, and the tests are done inside the kernel.
For what it's worth, here is the first hit Google gave me when asked about the Linux syscall mechanism: http://www.linux.it/~rubini/docs/ksys/. But perhaps you have a better resource closer to hand.

A: In Linux, if a process has the CAP_SETGID capability, the kernel will honor setgid(), setegid(), setregid(), setresgid(), and setgroups() calls (unless denied by a Linux Security Module like SELinux).
There are two main mechanisms how an unprivileged user may obtain a capability:


*

*Inheriting it from a privileged process. For normal users, extra capabilities may be granted by PAM modules at login.

*Via filesystem capabilities.
The filesystem capabilities are the more interesting, and useful for services/applications. They only work for binaries, not scripts, because the kernel updates the capabilities as part of the mechanism for executing binaries; scripts are interpreted by shells using a completely different, entirely userspace mechanism.
(The filesystem must support extended attributes. Some filesystems like ext2, ext3, and reiserfs, must be mounted with the user_xattr mount option to enable extended attributes. ext4, xfs, jfs, btrfs, and zfs all should support extended attributes using default mount options. Some Linux distributions like Debian and Ubuntu already rely on extended attributes and filesystem capabilities.)
Note: you can use the getcap and lsattr utilities to examine binaries filesystem capabilities and extended attributes. On my Ubuntu 16.04.4 LTS system, the /usr/bin/systemd-detect-virt binary has both CAP_DAC_OVERRIDE and CAP_SYS_PTRACE capabilities permitted and effective.
Using filesystem capabilities is very easy in practice, but for safe operation, the security implications should be carefully considered first: it is a powerful tool, but you (developer, package manager, and/or system administrator) are responsible for making sure it is used correctly.
For example, if you are installing /usr/bin/yourprog that is trusted and requires the CAP_SETGID capability, all you need to do is set that capability permitted and effective for that binary. To do that, you run setcap cap_setgid=pe /usr/bin/yourprog with root privileges. (In a Debian .deb package install script, that is typically run in the post-install script.)
