The Linux Programming Interface

* Chapter 2. Fundamental Concepts - The Linux kernel executable typically resides at the pathname /boot/vmlinuz, or something similar. The derivation of this filename is historical. On early UNIX implementations, the kernel was called unix. Later UNIX implementa- tions, which implemented virtual memory, renamed the kernel as vmunix . On Linux, the filename mirrors the system name, with the z replacing the final x to signify that the kernel is a compressed executable. ** The Core Operating System: The Kernel *** Tasks performed by the kernel - Process scheduling - Memory management - Provision of a file system - Creation and termination of processes - Access to devices - Networking - Provision of a system call application programming interface(API) *** Kernel mode and user mode - Modern processor architectures typically allow the CPU to operate in at least two different modes: user mode and kernel mode(sometimes also referred to as supervisor mode). Hardware instructions allow switching from one mode to the other. Correspondingly, areas of virtual memory can be marked as being part of user space or kernel space. When running in user mode, the CPU can access only memory that is marked as being in user space; attempts to access memory in kernel space result in a hardware exceptiono. When running in kernel mode, the CPU can access both user and kernel memory space. ** The Shell ** Users and Groups ** Single Directory Hierarchy, Directories, Links, and Files - For the root directory, the dot-dot entry is a link to the root directory itself(thus, /.. equates to /) *** Symbolic links - Like a normal link, a symbolic link provides an alternative name for a file. But whereas a normal link is a filename-plus-pointer entry in a directory list, a symbolic link is a specially marked file containing the name of another file. (In other words, a symbolic link has a filename-plus-pointer entry in a directory, and the file referred to by the pointer contains a string that names another file.) - Often hard link and soft link are used as alternative terms for normal and sym- bolic links. The reasons for having two different types of links are explained in Chapter 18. *** Current working directory - Each process has a current working directory(sometimes just refered to as the process's working directory or current directory). - A process inherits its current working directory from its parent process. A login shell has its initial current working directory set to the location named in the home directory field of the user’s password file entry. ** File I/O Model - The kernel translates the application’s I/O requests into appropriate file system or device-driver operations that perform I/O on the target file or device. - UNIX systems have no end-of-file character; the end of a file is detected by a read that returns no data. ** Programs ** Processes - From a kernel point of view, processes are the entities among which the kernel must share the various resources of the computer. *** Process memory layout - Text - Data - Heap - Stack *** Process creation and program execution - The child inherits copies of the parent’s data, stack, and heap segments, which it may then modify independently of the parent’s copies. (The pro- gram text, which is placed in memory marked as read-only, is shared by the two processes.) - The child process goes on either to execute a different set of functions in the same code as the parent, or, frequently, to use the execve() system call to load and execute an entirely new program. An execve() call destroys the existing text, data, stack, and heap segments, replacing them with new segments based on the code of the new program. *** Process ID and parent process ID *** Process termination and termination status *** Process user and group identifiers(credentials) - Real user ID and real group ID: These identify the user and group to which the process belongs. A new process inherits these IDs from its parent. A login shell gets its real user ID and real group ID from the corresponding fields in the system password file. - Effective user ID and effective group ID: These two IDs (in conjunction with the supplementary group IDs discussed in a moment) are used in determining the permissions that the process has when accessing protected resources such as files and interprocess communication objects. Typically, the process’s effective IDs have the same values as the corresponding real IDs. Changing the effective IDs is a mechanism that allows a process to assume the privileges of another user or group, as described in a moment. - Supplementary group IDs: These IDs identify additional groups to which a pro- cess belongs. A new process inherits its supplementary group IDs from its par- ent. A login shell gets its supplementary group IDs from the system group file. *** Privileged processes - Traditionally, on UNIX systems, a privileged process is one whose effective user ID is 0 (superuser). Such a process bypasses the permission restrictions normally applied by the kernel. By contrast, the term unprivileged (or nonprivileged) is applied to processes run by other users. Such processes have a nonzero effective user ID and must abide by the permission rules enforced by the kernel. - A process may be privileged because it was created by another privileged process—for example, by a login shell started by root (superuser). Another way a process may become privileged is via the set-user-ID mechanism, which allows a process to assume an effective user ID that is the same as the user ID of the program file that it is executing. *** Capabilities - Since kernel 2.2, Linux divides the privileges traditionally accorded to the super- user into a set of distinct units called capabilities. Each privileged operation is asso- ciated with a particular capability, and a process can perform an operation only if it has the corresponding capability. A traditional superuser process (effective user ID of 0) corresponds to a process with all capabilities enabled. *** The init process - The init process can’t be killed (not even by the superuser), and it terminates only when the system is shut down. The main task of init is to create and monitor a range of processes required by a running system. *** Daemon processes - It is long-lived. A daemon process is often started at system boot and remains in existence until the system is shut down. - It runs in the background, and has no controlling terminal from which it can read input or to which it can write output. *** Environment list - When a new process is created via fork(), it inherits a copy of its parent’s environment. Thus, the environment pro- vides a mechanism for a parent process to communicate information to a child pro- cess. When a process replaces the program that it is running using exec(), the new program either inherits the environment used by the old program or receives a new environment specified as part of the exec() call. - Environment variables are created with the export command in most shells (or the setenv command in the C shell) - C programs can access the environment using an external variable (char **environ), and various library functions allow a process to retrieve and modify values in its environment. *** Resource list - Using the setrlimit() system call, a process can establish upper limits on its consumption of various resources. - Each such resource limit has two associated values: a soft limit, which limits the amount of the resource that the process may consume; and a hard limit, which is a ceiling on the value to which the soft limit may be adjusted. - An unprivileged process may change its soft limit for a particular resource to any value in the range from zero up to the corresponding hard limit, but can only lower its hard limit. ** Memory Mappings - The mmap() system call creates a new memory mapping in the calling process’s virtual address space. - Memory mappings serve a variety of purposes, including initialization of a process’s text segment from the corresponding segment of an executable file, allocation of new (zero-filled) memory, file I/O (memory-mapped I/O), and interprocess communication (via a shared mapping). ** Static and Shared Libraries *** Static libraries *** Shared libraries - If a program is linked against a shared library, then, instead of copying object modules from the library into the executable, the linker just writes a record into the executable to indicate that at run time the executable needs to use that shared library. - When the executable is loaded into memory at run time, a program called the dynamic linker ensures that the shared libraries required by the executable are found and loaded into memory, and performs run-time linking to resolve the func- tion calls in the executable to the corresponding definitions in the shared libraries. ** Interprocess Communication and Synchronization - signals, which are used to indicate that an event has occurred. - pipes and FIFOs, which can be used to transfer data between processes. - sockets, which can be used to transfer data from one process to another, either on the same host computer or on different hosts connected by a network. - file locking, which allows a process to lock regions of a file in order to prevent other processes from reading or updating the file contents. - message queses, which are used to exchange messages(packets of data) between processes. - semaphores, which are used to synchronize the actions of processes. - shared memory, which allows two or more processes to share a piece of memory. When one process changes the contents of the shared memory, all of the other processes can immediately see the changes. ** Singals - Signals are often described as "software interrupts." - Signals are sent to a process by the kernel, by another process(with suitable permissions), or by the process itself. - A kernel may send a signal to a process when one of the following occurs: - the user typed the interrupt character (usually Control-C) on the keyboard; - one of the process's children has terminated; - a timer(alarm clock) set by the process has expired; or - the process attempted to access an invalid memory address. - In the interval between the time it is generated and the time it is delivered, a signal is said to be pending for a process. Normally, a pending signal is delivered as soon as the receiving process is next scheduled to run, or immediately if the pro- cess is already running. However, it is also possible to block a signal by adding it to the process’s signal mask. If a signal is generated while it is blocked, it remains pending until it is later unblocked (i.e., removed from the signal mask). ** Threads - One way of envisaging threads is as a set of processes that share the same virtual memory, as well as a range of other attributes. Each thread is executing the same program code and shares the same data area and heap. However, each thread has it own stack containing local variables and function call linkage information. ** Process Groups and Shell Job Controls - All major shells, except the Bourne shell, provide an interactive feature called job control, which allows the user to simultaneously execute and manipulate multiple commands or pipelines. In job-control shells, all of the processes in a pipeline are placed in a new process group or job. (In the simple case of a shell command line containing a single command, a new process group containing just a single process is created.) - Each process in a process group has the same integer process group identifier, which is the same as the process ID of one of the processes in the group, termed the process group leader. - The kernel allows for various actions, notably the delivery of signals, to be per- formed on all members of a process group. Job-control shells use this feature to allow the user to suspend or resume all of the processes in a pipeline, as described in the next section. ** Sessions, Controlling Terminals, and Controlling Processes - A session is a collection of process groups(jobs). All of the processes in a session have the same session identifier. A session leader is the process that created the session, and its process ID becomes the session ID. - Sessions are used mainly by job-control shells. All of the process groups cre- ated by a job-control shell belong to the same session as the shell, which is the session leader. ** Pseudoterminals - A pseudoterminal is a pair of connected virtual devices, known as the master and slave. This device pair provides an IPC channel allowing data to be transferred in both directions between the two devices. - Pseudoterminals are used in a variety of applications, most notably in the implementation of terminal windows provided under an X Window System login and in application providing network login services, such as telnet and ssh. ** Date and Time Two types of time are of interest to a process: - Real time is measured either from some standard point (calendar time) or from some fixed point, typically the start, in the life of a process (elapsed or wall clock time). On UNIX systems, calendar time is measured in seconds since midnight on the morning of January 1, 1970, Universal Coordinated Time (usually abbreviated UTC), and coordinated on the base point for timezones defined by the longitudinal line passing through Greenwich, England. This date, which is close to the birth of the UNIX system, is referred to as the Epoch. - Process time, also called CPU time, is the total amount of CPU time that a process has used since starting. CPU time is further divided into system CPU time, the time spent executing code in kernel mode (i.e., executing system calls and performing other kernel services on behalf of the process), and user CPU time, the time spent executing code in user mode (i.e., executing normal program code). ** Client-Server Architecture ** Realtime - The provision of realtime responsiveness, especially where short response times are demanded, requires support from the underlying operating system. Most operating systems don’t natively provide such support because the requirements of realtime responsiveness can conflict with the requirements of multiuser time- sharing operating systems. Traditional UNIX implementations are not realtime operating systems, although realtime variants have been devised. Realtime variants of Linux have also been created, and recent Linux kernels are moving toward full native support for realtime applications. - POSIX.1b defined a number of extensions to POSIX.1 for the support of real- time applications. These include asynchronous I/O, shared memory, memory- mapped files, memory locking, realtime clocks and timers, alternative scheduling policies, realtime signals, message queues, and semaphores. Even though they don’t strictly qualify as realtime, most UNIX implementations now support some or all of these extensions. ** The /proc File System - The /proc file system is a virtual file system that provides an interface to kernel data structures in a form that looks like files and directories on a file system. - The /proc file system is not specified by any standards, and the details that we describe the Linux-specific.

操作系统含义:

1. central software(管理资源) + accompanying standard software 的package

2. 单纯指central software，即内核

可执行的内核位置: /boot/vmlinuz

内核任务:

1. Process scheduling: linux类型为preemptive(调度由scheduler处理而非进程本身) multitasking

2. Memory management: 虚存管理的进程隔离和部分载入机制

3. Provision of a file system: 支持磁盘(广义)上的文件操作

4. Creation and termination of processes: 进程的资源分配

5. Access to devices: 设备访问接口和仲裁

6. Networking: 收发网络包、路由到指定系统

7. system call API: 提供进程请求内核完成任务的入口，即系统调用

8. virtual private computer: 多用户环境

用户态和内核态的区分、进程视角和内核视角

shell(或login shell): 作为command interpreter的用户进程

用户信息包括login name/UID/GID/home/login shell，放在/etc/passwd

组信息包括group name/GID/user list，放在/etc/group

超级用户root，UID=0

文件类型: regular(plain), devices, pipes, sockets, directories, symbolic links

推荐的文件名字符集[-._a-zA-Z0-9]，称为SUSv3的portable filename character set (但-不起头)

进程的current working directory属性继承自父进程，login shell默认为用户的home

文件和路径的ownership and permissions: 拥有者/组用户/其他的读/写/执行权限

universality of I/O: 同一I/O操作的系统调用(如open())可用于所有类型的I/O

file descriptors: 用来代表打开的文件，由调用open()得到

进程: 正在执行的程序实例; 执行的含义是载入虚存、分配了变量的空间和其他bookkeeping结构(PID等)

进程的segments，存放不同内容: text(程序指令, read-only) 、data(静态变量)、heap(动态分配空间)、stack(局部变量和函数调用)

创建信进程: fork()系统调用; 子进程共享父进程text segment，拷贝父进程data, heap和stack segment

子进程可调用execve()(或位于上层的exec()类函数)加载新程序，同时破坏并替换各segment

每个进程都有进程ID(PID)和父进程ID(PPID)

进程的终止: 进程自己调用_exit()(以及其他exit())， 或被信号kill掉

终止状态由_exit()给出，或由相应类型的信号得出; 一般0表示正常; shell下通过$?查询刚结束进程的终止状态

进程用户ID和组ID分为: 实际用户ID和组ID、有效用户ID和组ID、附加组id

特权进程(privileged processes): 有效用户ID为0的进程; 其他进程可通过set-user-ID变为privileged

capability: 以CAP_开头，每种能力对应一种特权操作

init进程: 所有进程都直接或间接由其创造; 程序为/sbin/init，PID=1，特权进程; 不能被kill，仅系统shutdown时才结束

daemon process: 常驻的后台程序，不与终端关联(无IO)

environtment list: 包含环境变量，继承自父进程，可由exec()替换;

环境变量在shell下用export命令创建，在C语言中通过char** environ获取

进程资源限制继承自父进程，可调用setrlimit()设置; 命令行提供ulimit命令

非特权进程仅能设置软限制(范围0~硬限制)，和降低硬限制

在虚存中创建新内存映射:mmap(); 按对象，分为文件映射和匿名映射; 按方式，分为私有和共享

object libraries:可被应用程序调用的编译好的函数代码，包括static libraries和shared libraries两种

static libraries:链接过程中拷贝所需函数代码到程序中; 浪费磁盘、内存空间，改动要重新链接

shared libraries:程序运行时由dynamic linker将所需函数代码载入并链接，没有静态库缺点

IPC: 包含signal(标志事件的发生)、pipe和FIFO(进程间数据传输)、socket(本地/远程数据传输)、file locking(锁定文件区域禁止其他进程读写)、message queues(交换消息)、semaphores(进程同步)、shared memory

signal: 即软中断; 用于通知事件/异常条件的发生; 为整数类型，表示为SIGxxx

用kill命令或kill()函数发信号，进程可以忽略/终止/挂起(后恢复)

系统调用(system call): 内核提供的API，用于进程向内核请求服务

系统调用流程:

1. 用户程序调用wrapper function

2. wrapper function把参数从栈拷贝到寄存器中(包括system call number，拷贝到%eax寄存器)

3.wrapper function调用int 0x80或sysenter等软中断指令

(1). 进入内核态，调用system_call()处理软中断(/i386/entry.S )

(2). system_call()保存寄存器值到内核栈、检查system call number，并以其为根据在sys_call_table找到相应service routine调用(命名为sys_xyz())

(3). service routine检查参数，执行并返回结果(正值成功，负值失败)给system_call()

(4). system_call()恢复寄存器值，将返回结果入栈，然后返回到wrapper function

5. 回到用户态，若出错，wrapper function将service routine的错误码取反置入全局变量errno; 然后wrapper function返回-1

跟踪系统调用命令: strace

-------------------------------------------------------------------------------------------------------------------------

library functions: C语言标准库提供的函数，有些使用了系统调用(接口更友好)，有些没有

unix平台C标准库: glibc(即GNU C library);

其他C库: uClibc和diet libc等适用于嵌入式应用的库

查看glibc版本: 直接在命令行运行标准库文件: $ /lib/x86_64-linux-gnu/libc.so.6

查看程序myprog依赖的动态库命令: ldd

编译时，glibc的版本号宏: 主版本号__GLIBC__，次版本号__GLIB_MINOR__

运行时，返回版本号字符串的函数: const char *gnu_get_libc_version(void);

-------------------------------------------------------------------------------------------------------------------------

错误处理: 除getpid()、_exit()等总成功的函数外，其他系统调用和库函数必须检查返回状态

读取errno中的错误类型，用头文件<errno.h>

先检查系统调用返回值是否为-1，若是再检查errno; 对于可能返回合法-1的函数，先置errno为0，再将返回值和errno一同作为检查条件

打印错误信息的相关函数:

void perror(const char * msg )，位于<stdio.h>； 打印msg，后接errno提供的错误信息

char *strerror(int errnum )，位于<string.h>； 返回errnum错误号给出的提示信息

错误诊断、数值比较和转换等功能见"tlpi_hdr.h"

-------------------------------------------------------------------------------------------------------------------------

feature test macros用于暴露某个指定标准的定义，两种方式:

1. 源码中在引入头文件前加上相关宏，如: #define _BSD_SOURCE 1

2. 编译时加上相关宏的选项，以-D开头; 如: $ cc -D_BSD_SOURCE prog.c

POSIX.1/SUS相关宏: _POSIX_SOURCE, _POSIX_C_SOURCE, _XOPEN_SOURCE

glibc相关宏: _BSD_SOURCE, _SVID_SOURCE, _GNU_SOURCE

-------------------------------------------------------------------------------------------------------------------------

系统的数据类型定义通常以_t结尾，大多位于 <sys/types.h>

printf()的输出格式兼容性: 格式总使用"%ld"，变量强转为long; off_t类型为"%lld"和long long

-------------------------------------------------------------------------------------------------------------------------

结构体变量必须按成员逐个初始化; 值存入文件时必须注明各成员

平台相关宏使用前用#ifdef检查，如: #ifdef WCOREDUMP

文件描述符为int类型，可表示任何类型的文件: FIFO、socket、终端、设备、常规文件等

标准输入、输出、错误I/O的文件描述符宏，位于<unistd.h> .:

STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO，值分别为0,1,2

-------------------------------------------------------------------------------------------------------------------------

打开文件open(pathname, flags, mode)，需要<sys/stat.h>和<fcntl.h>:

返回打开文件的描述符(可用的最小值)，失败返回-1同时置errno

errno的可能取值: 权限不足EACCES、试图写目录EISDIR、超过进程描述符数目限制RLIMIT_NOFILE错误EMFILE、超过系统描述符数目限制错误ENFILE、文件/路径/符号链接不存在ENOENT、只读系统上请求写操作EROFS、写可执行文件时忙ETXTBSY

flags相关错误:创建检查失败EEXIST、路径不是目录名ENOTDIR、路径是符号链接ELOOP

其中参数flags分为:

1.file access mode flags: 只读O_RDONLY，只写O_WRONLY，读写O_RDWR

2.file creation flags: 原子地检查并创建O_CREAT|O_EXCL，创建并截短O_CREAT|O_TRUNC，以及close-on-exec、buffer cache、路径、大文件 、访问时间、控制终端、符号链接路径名等相关的位

3.open file status flags: 写到末尾O_APPEND和其他signal-driven或synchronous I/O、nonblocking等相关的位错误

1,3可通过fcntl()的F_GETFL操作获取，3可通过F_SETFL设置

进程描述父信息可从/proc/ PID /fdinfo中获取

参数mode为创建文件时赋予的权限，未创建则可省略，类型为mode_t

过时的创建文件函数creat(...):

等价于open(path, O_WRONLY | O_CREAT | O_TRUNC, mode);

-------------------------------------------------------------------------------------------------------------------------

读取文件read(fd, buffer, count): 从fd读count(类型为size_t)个字节数据到事先分配的buffer中

位于<unistd.h>，返回值类型为ssize_t，为实际读取字节数或0(EOF)或-1(出错)

若要打印buffer中的字符，要预先留出一个位置给'\0'

-------------------------------------------------------------------------------------------------------------------------

写入文件write(fd, buffer, count): 向fd写buffer中count个字节数据，返回实际写入字节数

部分写入现象，可因进程的文件大小限制RLIMIT_FSIZE发生

由于存在缓冲机制，函数成功返回后数据未必真正写入磁盘

-------------------------------------------------------------------------------------------------------------------------

关闭文件status = close(fd)，返回状态

-------------------------------------------------------------------------------------------------------------------------

更改文件偏移lseek(fd, offset, whence)

返回值表示当前偏移，参数offset为相对偏移值，类型都为off_t; 偏移操作如下:

whence表示的位置: 文件起始SEEK_SET，当前偏移SEEK_SET，文件结尾后的一字节SEEK_SET

ESPIPE错误: 试图改变不支持该操作的文件的偏移

file hole: 原先文件的EOF和偏移到后面新写入的字节，两者之间的空间;

不占磁盘空间(位于块中间或非兼容系统会占)，使文件名义大小和实际大小不一致; 读出为全0

手动分配空间函数: glibc的posix_fallocate()、系统调用fallocate()

-------------------------------------------------------------------------------------------------------------------------

universal I/O之外的操作 ioctl(fd, request, ...):

由参数request指定操作，第三个参数随操作而有所不同