程序员的自我修养

本篇笔记全文摘自后面参考文献列出的地址，不过原文中有一些错别字，我这里把错别字改了改，然后也是按照自己机器上代码的结果来进行演示。再结合本书前面第6章“6.5 Linux内核装载ELF过程简介”来进行说明。1、开始 The question is simple: how does linux execute my main()? Through this document, I'll use the following simple C program to illustrate how it works. It's called "execute_main.c"

int main()
{
   return 0;
}

2、编译 gcc -o execute_main execute_main.c3、What's in the executable?

i4@i4:~/文档/tempfile$ objdump -f execute_main

execute_main:     file format elf32-i386
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08048300

4、What's at address "0x08048300", that is, starting address?

i4@i4:~/文档/tempfile$ objdump -d execute_main

The output is a little bit long so I'll not paste all the output from objdump. Our intention is see what's at address 0x08048300. Here is the output.

Disassembly of section .text:

08048300 <_start>:
 8048300:	31 ed                	xor    %ebp,%ebp
 8048302:	5e                   	pop    %esi
 8048303:	89 e1                	mov    %esp,%ecx
 8048305:	83 e4 f0             	and    $0xfffffff0,%esp
 8048308:	50                   	push   %eax
 8048309:	54                   	push   %esp
 804830a:	52                   	push   %edx
 804830b:	68 30 84 04 08       	push   $0x8048430
 8048310:	68 c0 83 04 08       	push   $0x80483c0
 8048315:	51                   	push   %ecx
 8048316:	56                   	push   %esi
 8048317:	68 b4 83 04 08       	push   $0x80483b4
 804831c:	e8 cf ff ff ff       	call   80482f0 <__libc_start_main@plt>
 8048321:	f4                   	hlt    

Looks like some kind of starting routine called "_start" is at the starting address. What it does is clear a register, push some values into stack and call a function. According to this instruction, the stack frame should look like this. ---------------- %eax ---------------- %esp ---------------- %edx ---------------- 0x8048430 ---------------- 0x80483c0 ---------------- %ecx ---------------- %esi ----------------Stack Top 0x80483b4 ---------------- Now, as you already wonder,we've got a few questions regarding this stack frame. Q1: What are those hex values about? Q2: What's at address 80482f0, which is called by _start? (PS: 即上面反汇编代码中804831c: e8 cf ff ff ff call 80482f0 <__libc_start_main@plt>) Q3: Looks like the assembly instructions doesn't initialize any register with possibly meaningful values. Then who initializes the registers? (PS: 个人认为这个很重要，后面有较多篇幅来描述) A1: The hexa values. If you look at disassembled output from objdump carefully, you can answer this question easily.再把反汇编代码摘出来，然后给出答案：

08048430 <__libc_csu_fini>: ......
080483c0 <__libc_csu_init>: ......
080483b4 <main>: ......

Right now, let's not care about these stuffs. And basically, all those hexa values are function pointers. A2: What's at address 80482f0?

080482f0 <__libc_start_main@plt>:
 80482f0:	ff 25 04 a0 04 08    	jmp    *0x804a004
......

Here *0x804a004 is a pointer operation. It just jumps to an address stored at address 0x804a004.那么这里地址0x804a004处是什么呢？我们看下面代码(PS: 原文这里又说了说关于动态链接的内容，这些内容在书第7章已经讲了，我这里不再陈述。大家跟着看下代码就明白了。)

i4@i4:~/文档/tempfile$ objdump -R execute_main

execute_main:     file format elf32-i386

DYNAMIC RELOCATION RECORDS
OFFSET   TYPE              VALUE 
08049ff0 R_386_GLOB_DAT    __gmon_start__
0804a000 R_386_JUMP_SLOT   __gmon_start__
0804a004 R_386_JUMP_SLOT   __libc_start_main

看到没，上面0x804a004其实是__libc_start_main函数的地址，是动态链接里面动态重定位里的一种类型。即，其实我们是想调用__libc_start_main函数。5、What's __libc_start_main? Now the ball is on libc's hand. __libc_start_main is a function in libc.so.6. If you look for __libc_start_main in glibc source code, the prototype looks like this.

int __libc_start_main(
                  int (*main) (int, char * *, char * *), 
                  int argc, 
                  char * * ubp_av, 
                  void (*init) (void), 
                  void (*fini) (void), 
                  void (*rtld_fini) (void), 
                  void (* stack_end)
);

总结： 到此我们知道： all the assembly instructions do is set up argument stack and call _libc_start_main. What this function does is setup/initialize some data structures/environments and call our main(). Let's look at the stack frame with this function prototype. ---------------- %eax this is 0 ---------------- %esp stack_end ---------------- %edx _rtlf_fini ---------------- 0x8048430 _libc_csu_fini ---------------- 0x80483c0 _libc_csu_init ---------------- %ecx argv ---------------- %esi argc ----------------Stack Top 0x80483b4 main ---------------- According to this stack frame, esi, ecx, edx, esp, eax registers should be filled with appropriate values before __libc_start_main() is executed. And clearly this registers are not set by the startup assembly instructions shown before. Then, who sets these registers? Now I guess the only thing left. The kernel. Now let's go back to our third question. A3: What does the kernel do? When we execute a program by entering a name on shell, this is what happens on Linux.(PS: 下面的描述太长，其实描述的也就是书里第6章“6.5 Linux内核装载ELF过程简介”的内容。我在这段英文后面给出大概的执行流程) 1: The shell calls the kernel system call "execve" with argc/argv. The kernel system call handler gets control and start handling the system call. In kernel code, the handler is "sys_execve". On x86, the user-mode application passes all required parameters to kernel with the following registers. %ebx : pointer to program name string %ecx : argv array pointer %edx : environment variable array pointer. 2: The generic execve kernel system call handler, which is do_execve, is called. What it does is set up a data structure and copy some data from user space to kernel space and finally calls search_binary_handler(). Linux can support more than one executable file format such as a.out and ELF at the same time. For this functionality, there is a data structure "struct linux_binfmt", which has a function pointer for each binary format loader. And search_binary_handler() just looks up an appropriate handler and calls it. In our case, load_elf_binary() is the handler. To explain each detail of the function would be lengthy/boring work. So I'll not do that. If you are interested in it, read a book about it. As a picture tells a thousand words, a thousand lines of source code tells ten thousand words (sometimes). Here is the bottom line of the function. It first sets up kernel data structures for file operation to read the ELF executable image in. Then it sets up a kernel data structure: code size, data segment start, stack segment start, etc. And it allocates user mode pages for this process and copies the argv and environment variables to those allocated page addresses. Finally, argc, the argv pointer, and the envrioronment variable array pointer are pushed to user mode stack by create_elf_tables(), and start_thread() starts the process execution rolling. 这两段描述的大概流程是：shell ---> execve (同时传递argc, argv) ---> sys_execve (用户模式下的一些内容传递给内核模式里的一些寄存器，即%ebx, %ecx, %edx。这三个的具体内容上面1:里有) ---> search_binary_handler() ---> load_elf_binary()。最后这个load_elf_binary()设置了不少内容，设置完之后就是返回到用户模式下运行e_entry的入口函数。 在入口函数_start()执行之前，栈的内容如下： ---------------- env pointer ---------------- argv pointer ----------------Stack Top argc ---------------- And the assembly instructions gets all information from stack by

pop %esi 		<--- get argc
mov %esp, %ecx		<--- get argv (actually the argv address is the same as the current stack pointer.)

And now we are all set to start executing.6、What about the other registers? For %esp, this is used for stack end in application program. After popping all necessary information, the _start rountine simply adjusts the stack pointer (%esp) by turning off lower 4 bits from esp register. This perfectly makes sense since actually, to our main program, that is the end of stack. For edx, which is used for rtld_fini, a kind of application destructor, the kernel just sets it to 0 with the following macro.#define ELF_PLAT_INIT(_r) do { \ _r->ebx = 0; _r->ecx = 0; _r->edx = 0; \ _r->esi = 0; _r->edi = 0; _r->ebp = 0; \ _r->eax = 0; \} while (0) The 0 means we don't use that functionality on x86 linux.7、Summing up Here is what happens. 1: GCC build your program with crtbegin.o/crtend.o/gcrt1.o And the other default libraries are dynamically linked by default. Starting address of the executable is set to that of _start. 2: Kernel loads the executable and setup text/data/bss/stack, especially, kernel allocate page(s) for arguments and environment variables and pushes all necessary information on stack. 3: Control is pased to _start. _start gets all information from stack setup by kernel, sets up argument stack for __libc_start_main, and calls it. 4: __libc_start_main initializes necessary stuffs, especially C library(such as malloc) and thread environment and calls our main. 5: our main is called with main(argv, argv) Actually, here one interesting point is the signature of main. __libc_start_main thinks main's signature as main(int, char **, char **) If you are curious, try the following prgram.

main(int argc, char** argv, char** env)
{
    int i = 0;
    while(env[i] != 0)
    {
       printf("%s\n", env[i++]);
    }
    return(0);
}

8、Conclusion On Linux, our C main() function is executed by the cooperative work of GCC, libc and Linux's binary loader.9、Reference [1] http://linuxgazette.net/issue84/hawk.html [2] http://www.fsl.cs.sunysb.edu/~siddhi/ELF.pdf

实际上在动态链接器的自举代码中，除了不可以使用全局变量和静态变量之外，甚至不能调用函数，即动态链接器本身的函数也不能调用。这是为什么呢？其实我们在前面分析地址无关代码时已经提到过，实际上使用PIC模式编译的共享对象，对于模块内部的函数调用也是采用外部函数调用一样的方式，即使用GOT/PLT的方式，所以在GOT/PLT没有被重定位之前，自举代码不可以使用任何全局变量，也不可以调用函数。

笔者的疑问主要集中在，为什么动态链接器在自举过程中无法调用其本身定义的函数？ 作者说他们在地址无关代码章节（7.3节，P188-P199），给出了解释。但是我实在是没找到为什么不行。调用模块内定义的函数，这属于P196 表7-1中的类型1，即模块内部的指令跳转及引用，只需要使用相对跳转和调用即可实现PIC。192页上半页有比较详细的解释，在192页作者又提及

这样看起来第一个模块内部调用或跳转很容易解决，但实际上这种方式还有一定的问题，这里存在一个名作共享对象“全局符号介入”问题。

。实际上，全局符号介入在218页才给出了详细的介绍。在218页，作者得出结论

所以对于bar()函数的调用，编译器只能选择第三种，即当作模块的外部符号处理。 ... 即使用static关键字定义bar函数，这种情况下，编译器要确定bar()函数不会其他模块覆盖，就可以使用第一类的方法，即模块内部调用指令。

从上面的引文可以看出，函数内部中定义的非static函数也需要通过GOT/PLT调用，而不是P192页中所说的简单的相对地址调用即可。但是这个地方的确切阐述是在P214页之后的P218页完成的。不得不说，这个地方是相对较为严重的疏漏。另外，P218页 全局符号介入与地址无关代码

由于可能存在的全局符号介入的问题，foo函数对于bar的调用不能够采用第一类模块调用的方法，因为一旦bar函数由于全局符号介入被其他模块中的同名函数覆盖，那么foo如果采用相对地址调用的话，那么这个相对地址就需要重定位，这由于共享对象的地址无关性矛盾。所以对于bar()函数的调用，编译器只能选择第三种，即当作模块外部符号处理。

“如果采用相对地址调用的话，那么这个相对地址就需要重定位” 这一句意思是说，模块中的函数可能被其他模块中的同名函数覆盖，即全局符号介入，并且当前模块中对此函数调用也需要被修改为对被覆盖的同名函数的调用，这就导致这个函数的地址不确定，如果此函数在多个模块存在，并且由于模块加载地址的变动，则此函数可能的地址就可能有多个，也就是说需要对这个函数的地址重定位，所以类型1中介绍的相对地址跳转不能解决问题，依然需要借助GOT实现PIC方式。这样来看，模块内部定义的全局变量和非static函数（也就是导出函数）其实本质上一样的，都需要借助GOT实现。

过度优化的问题自然很多，这里不再描述。只说解决方法：

可以使用volatile关键字来试图阻止过度优化，volatile基本可以做到两件事情： 1、阻止编译器为了提高速度将一个变量缓存到寄存器内而不写回。 2、阻止编译器调整操作volatile变量的指令顺序。

但即使这样，还是有问题。 因为即使volatile能够阻止编译器调整指令顺序，也无法阻止CPU动态调度换序。下面举例说明：Singleton模式

volatile T* pInst = 0;
T *GetInstance()
{
        if (pInst == NULL) {
                lock();
                if (pInst == NULL)
                        pInst = new T;
                unlock();
        }
        return pInst;
}

这段代码的问题是CPU的乱序执行。因为C++里的new操作包含了两个步骤：1、分配内存。2、调用构造函数。所以，pInst = new T包含了三个步骤： (1) 分配内存。 (2) 在内存的位置上调用构造函数。 (3) 将内存的地址赋值给pInst。在这三步中，(2)和(3)的顺序是可以颠倒的。也就是说，完全有可能出现这样的情况：pInst的值已经不是NULL，但对象仍然没有构造完毕。这时候如果出现另外一个对GetInstance的并发调用，此时第一个if内的表达式pInst == NULL 为false，所以这个调用会直接返回尚未构造完全的对象的地址(pInst)以提供给用户使用。那么程序这个时候会不会崩溃就取决于这个类的设计如何了。 解决这一问题，即多线程环境中，CPU的换序问题，可以通过barrier指令来完成。

#define barrier() __asm__ volatile("lwsync")
volatile T* pInst = 0;
T *GetInstance()
{
        if (pInst == NULL) {
                lock();
                if (pInst == NULL) { // 通过临时对象指针和屏障指令，就保证了上面(2)(3)的顺序执行
                        T *temp = new T;
                        barrier();
                        pInst = temp; 
                }
                unlock();
        }
        return pInst;
}

OK，完美解决～～

cat /proc/pid/maps

例如：

i4@i4:~/文档/proger$ ./p162.elf &
[1] 6178
i4@i4:~/文档/proger$ cat /proc/6178/maps
08048000-080ee000 r-xp 00000000 08:07 7875904    /home/i4/文档/proger/p162.elf
080ee000-080f0000 rw-p 000a5000 08:07 7875904    /home/i4/文档/proger/p162.elf
080f0000-080f2000 rw-p 00000000 00:00 0 
093ba000-093dc000 rw-p 00000000 00:00 0          [heap]
b7764000-b7765000 r-xp 00000000 00:00 0          [vdso]
bfde5000-bfe06000 rw-p 00000000 00:00 0          [stack]

其中： 第一列是VMA的地址范围 第二列是VMA的权限，r代表可读，w代表可写，x代表可执行，p代表私有，s代表共享 第三列是偏移，表示VMA对应的Segment在映像文件中的偏移 第四列是映像文件所在设备的主设备号和次设备号 第五列是映像文件的节点号 最后一列是映像文件的路径