Archive for the ‘coredump’ Category

android: arm64: unwind stack to get frames and backtrace of a thread

October 29, 2015

This post is to discuss backtrace, frame, and how frames make up a backtrace in arm64. We demonstrate an example in which we unwind stack to get frames and backtrace of a thread.

testing environment
The infrastructure code base here is LA.BF64.1.1-06510-8×94.0 with Android 5.0.0_r2(LRX21M) and Linux kernel 3.10.49. The device CPU is architecture arm-v8 cortex-53.

get backtrace from tombstone
In android: arm64: analyze the call stack after a process hit native crash, we shows how to get the backtrace of a process at native crash from tombstone.

ABI: 'arm64'
pid: 20948, tid: 20948, name: coredumptest  >>> /data/coredumptest <<< 
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 
.......
backtrace:
    #00 pc 0000000000014434  /system/lib64/libc.so (strlen+16)
    #01 pc 0000000000000efc  /data/coredumptest
    #02 pc 0000000000000f84  /data/coredumptest
    #03 pc 000000000000100c  /data/coredumptest
    #04 pc 0000000000001094  /data/coredumptest
    #05 pc 0000000000000d78  /data/coredumptest (main+40)
    #06 pc 0000000000013474  /system/lib64/libc.so (__libc_init+100)
    #07 pc 0000000000000e8c  /data/coredumptest

get the thread’s stack from tombstone
the sp register points to top of the thread’s stack. It appears that the stack has 8 frames, from frame #00 to frame #07. Each frame except frame #00 has a symbol indicating from which to enter this frame.

ABI: 'arm64'
pid: 20948, tid: 20948, name: coredumptest  >>> /data/coredumptest <<< 
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 
.......
    x28  0000000000000000  x29  0000007fcbe33220  x30  0000005595326f00
    sp   0000007fcbe33220  pc   0000007faa597434  pstate 0000000040000000
......
stack:
         0000007fcbe331a0  0000000000000000  
         0000007fcbe331a8  0000000000000000  
         0000007fcbe331b0  0000000000000000  
         0000007fcbe331b8  0000000000000000  
         0000007fcbe331c0  0000000000000000  
         0000007fcbe331c8  0000000000000000  
         0000007fcbe331d0  0000000000000000  
         0000007fcbe331d8  0000000000000000  
         0000007fcbe331e0  0000000000000000  
         0000007fcbe331e8  0000000000000000  
         0000007fcbe331f0  0000000000000000  
         0000007fcbe331f8  0000007faa63e040  /system/bin/linker64
         0000007fcbe33200  0000000000000006  
         0000007fcbe33208  0000000000000000  
         0000007fcbe33210  0000007fcbe33270  [stack]
         0000007fcbe33218  0000007faa5dcc98  /system/lib64/libc.so (__cxa_atexit+60)
    #00  0000007fcbe33220  0000007fcbe33240  [stack]
         ........  ........
    #01  0000007fcbe33220  0000007fcbe33240  [stack]
         0000007fcbe33228  0000005595326f88  /data/coredumptest
         0000007fcbe33230  ffffffffffffffff  
         0000007fcbe33238  0000007fcbe33348  [stack]
    #02  0000007fcbe33240  0000007fcbe33260  [stack]
         0000007fcbe33248  0000005595327010  /data/coredumptest
         0000007fcbe33250  ffffffffffffffff  
         0000007fcbe33258  0000007faa63e198  /system/bin/linker64
    #03  0000007fcbe33260  0000007fcbe33280  [stack]
         0000007fcbe33268  0000005595327098  /data/coredumptest
         0000007fcbe33270  ffffffffffffffff  
         0000007fcbe33278  0000007faa596468  /system/lib64/libc.so (__libc_init+88)
    #04  0000007fcbe33280  0000007fcbe332a0  [stack]
         0000007fcbe33288  0000005595326d7c  /data/coredumptest (main+44)
         0000007fcbe33290  ffffffffffffffff  
         0000007fcbe33298  0000005595326d50  /data/coredumptest (main)
    #05  0000007fcbe332a0  0000007fcbe332d0  [stack]
         0000007fcbe332a8  0000007faa596478  /system/lib64/libc.so (__libc_init+104)
         0000007fcbe332b0  0000007fcbe33358  [stack]
         0000007fcbe332b8  0000000000000000  
         0000007fcbe332c0  ffffffffffffffff  
         0000007fcbe332c8  ffffffffffffffff  
    #06  0000007fcbe332d0  0000007fcbe33300  [stack]
         0000007fcbe332d8  0000005595326e90  /data/coredumptest
         0000007fcbe332e0  0000000000000000  
         0000007fcbe332e8  0000000000000000  
         0000007fcbe332f0  0000000000000000  
         0000007fcbe332f8  0000000000000000  
    #07  0000007fcbe33300  0000000000000000
         0000007fcbe33308  0000007faa63f610  /system/bin/linker64 (_start+8)
         0000007fcbe33310  0000000000000000
         0000007fcbe33318  0000007fcbe33340  [stack]
         0000007fcbe33320  0000000000000000
         0000007fcbe33328  0000005595338ce0  /data/coredumptest
         0000007fcbe33330  0000005595338cf0  /data/coredumptest
         0000007fcbe33338  0000005595338d00  /data/coredumptest
         0000007fcbe33340  0000000000000001
         0000007fcbe33348  0000007fcbe33a3a  [stack]
         0000007fcbe33350  0000000000000000
         0000007fcbe33358  0000007fcbe33a4d  [stack]
         0000007fcbe33360  0000007fcbe33a62  [stack]
         0000007fcbe33368  0000007fcbe33a8e  [stack]
         0000007fcbe33370  0000007fcbe33aa1  [stack]
         0000007fcbe33378  0000007fcbe33d67  [stack]

what is frame pointer
According to arm64 aarch64 state PCS(Procedure Call Standard), register x29 is frame pointer and x30 is link register. arm64 kernel enable frame pointer which makes it easy to unwind thread’s stack into frames. All frames are in a linked list in which each element points to the next element with its frame pointer.

how a frame is constructed and destructed
While entering a function, a frame might be created such as line 2, 3.

  • stack pointer is decreased by frame size.(push)
  • frame pointer and link register are stored at the top of the frame.
  • frame pointer is updated as stack pointer.
  • While leaving a function, the created frame should be removed such as line 11.

  • frame pointer and link register are loaded from top of the frame.
  • stack pointer is increased by frame size.(pop)
  • 0000000000000ec0 <atexit>:
         ec0:   a9be7bfd    stp x29, x30, [sp,#-32]!
         ec4:   910003fd    mov x29, sp
         ec8:   f9000fa0    str x0, [x29,#24]
         ecc:   90000000    adrp    x0, 0 <writev@plt-0xbc0>
         ed0:   913a6000    add x0, x0, #0xe98
         ed4:   f0000081    adrp    x1, 13000 <__dso_handle>
         ed8:   91000022    add x2, x1, #0x0
         edc:   f9400fa1    ldr x1, [x29,#24]
         ee0:   97ffff74    bl  cb0 <__cxa_atexit@plt>
         ee4:   a8c27bfd    ldp x29, x30, [sp],#32
         ee8:   d65f03c0    ret   
    

    how all frames are linked with frame pointer
    All frames form a linked list. Frame pointer is the head of this linked list.

  • In this example, frame pointer x29 0000007fcbe33220 points to the top address of frame #1.
  • ABI: 'arm64'
    pid: 20948, tid: 20948, name: coredumptest  >>> /data/coredumptest <<< 
    signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 
    .......
        x28  0000000000000000  x29  0000007fcbe33220  x30  0000005595326f00
        sp   0000007fcbe33220  pc   0000007faa597434  pstate 0000000040000000
    
  • frame pointer, 0x0000007fcbe33220, of frame #1 points to the top address of frame #02, 0000007fcbe33240
  • link register, 0x0000007fcbe33228, of frame #1 points to 0000005595326f88 /data/coredumptest
  •     #01  0000007fcbe33220  0000007fcbe33240  [stack]
             0000007fcbe33228  0000005595326f88  /data/coredumptest
             0000007fcbe33230  ffffffffffffffff  
             0000007fcbe33238  0000007fcbe33348  [stack]
    
  • frame pointer, 0000007fcbe33240, of frame #2 points to the top address of frame #03, 0000007fcbe33260
  • link register, 0000007fcbe33248, of frame #2 points to 0000005595327010 /data/coredumptest
  •     #02  0000007fcbe33240  0000007fcbe33260  [stack]
             0000007fcbe33248  0000005595327010  /data/coredumptest
             0000007fcbe33250  ffffffffffffffff  
             0000007fcbe33258  0000007faa63e198  /system/bin/linker64
    
  • frame pointer, 0000007fcbe33260, of frame #3 points to the top address of frame #04, 0000007fcbe33280
  • link register, 0000007fcbe33268, of frame #3 points to 0000005595327098 /data/coredumptest
  •     #03  0000007fcbe33260  0000007fcbe33280  [stack]
             0000007fcbe33268  0000005595327098  /data/coredumptest
             0000007fcbe33270  ffffffffffffffff  
             0000007fcbe33278  0000007faa596468  /system/lib64/libc.so (__libc_init+88)
    
  • frame pointer, 0000007fcbe33280, of frame #4 points to the top address of frame #05, 0000007fcbe332a0
  •     #04  0000007fcbe33280  0000007fcbe332a0  [stack]
             0000007fcbe33288  0000005595326d7c  /data/coredumptest (main+44)
             0000007fcbe33290  ffffffffffffffff  
             0000007fcbe33298  0000005595326d50  /data/coredumptest (main)
    
  • frame pointer, 0000007fcbe332a0, of frame #5 points to the top address of frame #06, 0000007fcbe332d0
  • link register, 0000007fcbe332a8, of frame #5 points to 0000007faa596478 /system/lib64/libc.so (__libc_init+104)
  •     #05  0000007fcbe332a0  0000007fcbe332d0  [stack]
             0000007fcbe332a8  0000007faa596478  /system/lib64/libc.so (__libc_init+104)
             0000007fcbe332b0  0000007fcbe33358  [stack]
             0000007fcbe332b8  0000000000000000  
             0000007fcbe332c0  ffffffffffffffff  
             0000007fcbe332c8  ffffffffffffffff  
    
  • frame pointer, 0000007fcbe332d0, of frame #6 points to the top address of frame #07, 0000007fcbe33300
  • link register, 0000007fcbe332d8, of frame #6 points to 0000005595326e90 /data/coredumptest
  •     #06  0000007fcbe332d0  0000007fcbe33300  [stack]
             0000007fcbe332d8  0000005595326e90  /data/coredumptest
             0000007fcbe332e0  0000000000000000  
             0000007fcbe332e8  0000000000000000  
             0000007fcbe332f0  0000000000000000  
             0000007fcbe332f8  0000000000000000  
    
  • frame pointer, 0000007fcbe33300, of frame #7 points to 0000000000000000. It’s the last frame.
  • link register, 0000007fcbe33308, of frame #7 points to 0000007faa63f610 /system/bin/linker64 (_start+8)
  •     #07  0000007fcbe33300  0000000000000000
             0000007fcbe33308  0000007faa63f610  /system/bin/linker64 (_start+8)
             0000007fcbe33310  0000000000000000
             0000007fcbe33318  0000007fcbe33340  [stack]
             0000007fcbe33320  0000000000000000
             0000007fcbe33328  0000005595338ce0  /data/coredumptest
             0000007fcbe33330  0000005595338cf0  /data/coredumptest
             0000007fcbe33338  0000005595338d00  /data/coredumptest
             0000007fcbe33340  0000000000000001
             0000007fcbe33348  0000007fcbe33a3a  [stack]
             0000007fcbe33350  0000000000000000
             0000007fcbe33358  0000007fcbe33a4d  [stack]
             0000007fcbe33360  0000007fcbe33a62  [stack]
             0000007fcbe33368  0000007fcbe33a8e  [stack]
             0000007fcbe33370  0000007fcbe33aa1  [stack]
             0000007fcbe33378  0000007fcbe33d67  [stack]
    

    how to get backtrace from linked frames
    The linker register of each frame is stored in the second double word of a frame.

  • #00 pc is current pc register 0000007faa597434
  • #01 pc is current link register 0000005595326f00 – 0x4 = 0000005595326efc
  • #02 pc is link register of frame #01 0000005595326f88 – 0x4 = 0000005595326f84
  • #03 pc is link register of frame #02 0000005595327010 – 0x4 = 000000559532700c
  • #04 pc is link register of frame #03 0000005595327098 – 0x4 = 0000005595327094
  • #05 pc is link register of frame #04 0000005595326d7c – 0x4 = 0000005595326d78
  • #06 pc is link register of frame #05 0000007faa596478 – 0x4 = 0000007faa596474
  • #07 pc is link register of frame #06 0000005595326e90 – 0x4 = 0000005595326e8c
  • The backtrace unwinded from stack is same as the backtrace of tombstone and gdb loading core file in android: arm64: analyze the call stack after a process hit native crash.

    ABI: 'arm64'
    pid: 20948, tid: 20948, name: coredumptest  >>> /data/coredumptest <<< 
    signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 
    ......
        x28  0000000000000000  x29  0000007fcbe33220  x30  0000005595326f00
        sp   0000007fcbe33220  pc   0000007faa597434  pstate 0000000040000000
    ......
    backtrace:
        #00 pc 0000000000014434  /system/lib64/libc.so (strlen+16)
        #01 pc 0000000000000efc  /data/coredumptest
        #02 pc 0000000000000f84  /data/coredumptest
        #03 pc 000000000000100c  /data/coredumptest
        #04 pc 0000000000001094  /data/coredumptest
        #05 pc 0000000000000d78  /data/coredumptest (main+40)
        #06 pc 0000000000013474  /system/lib64/libc.so (__libc_init+100)
        #07 pc 0000000000000e8c  /data/coredumptest
    
    (gdb) bt
    #0  strlen () at bionic/libc/arch-arm64/generic/bionic/strlen.S:71
    #1  0x0000005595326f00 in strlen (s=0x0) at bionic/libc/include/string.h:239
    #2  test4 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:11
    #3  0x0000005595326f88 in test3 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:20
    #4  0x0000005595327010 in test2 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:29
    #5  0x0000005595327098 in test1 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:38
    #6  0x0000005595326d7c in main () at frameworks/native/services/coredumptest/CoredumpTest.cpp:56
    

    conclusion
    In this post, we explain what is backtrace and frames. We then show how to use registers such stack pointer and frame pointer of current context to get frames and backtrace of a thread.

    Advertisements

    android: coredump: analyze core file with gdb

    October 27, 2015

    In android: coredump; how to make kernel dump core file after some process crashes, we discuss how to get core file of a process after it native crashes. In this post, we discuss how to analyze the core file with gdb.

    run gdb and setup environment

  • star gdb
  • $ cd ~/android_source/
    $ ./prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin//aarch64-linux-android-gdb
    GNU gdb (GDB) 7.7
    Copyright (C) 2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "--host=x86_64-linux-gnu --target=aarch64-elf-linux".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://source.android.com/source/report-bugs.html>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word".
    
  • setup file
  • (gdb) file out/target/product/${project}/symbols/system/bin/coredumptest
    Reading symbols from symbols/system/bin/coredumptest...done.
    
  • setup dynamic library search path
  • (gdb) set solib-search-path out/target/product/${project}/symbols/system/lib64/
    
  • setup core-file
  • (gdb) core-file 20948.coredumptest.18446744073709551615.core
    [New LWP 20948]
    warning: Could not load shared library symbols for 4 libraries, e.g. /system/bin/linker64.
    Use the "info sharedlibrary" command to see the complete listing.
    Do you need "set solib-search-path" or "set sysroot"?
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  strlen () at bionic/libc/arch-arm64/generic/bionic/strlen.S:71
    warning: Source file is more recent than executable.
    71              ldp     data1, data2, [src], #16
    

    dump stack of the crashing thread

  • Use backtrace(bt) command to get callback
  • (gdb) bt
    #0  strlen () at bionic/libc/arch-arm64/generic/bionic/strlen.S:71
    #1  0x0000005595326f00 in strlen (s=0x0) at bionic/libc/include/string.h:239
    #2  test4 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:11
    #3  0x0000005595326f88 in test3 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:20
    #4  0x0000005595327010 in test2 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:29
    #5  0x0000005595327098 in test1 () at frameworks/native/services/coredumptest/CoredumpTest.cpp:38
    #6  0x0000005595326d7c in main () at frameworks/native/services/coredumptest/CoredumpTest.cpp:56
    
  • The gdb call stack satisfies the code flow of the native crash in coredumptest
  • int test4()
    {
        int ret = strlen(NULL);
        ......
        return ret;
    }
    
    int test3()
    {
        int ret = test4() + 3;
        ......
        return ret;
    }
    
    int test2()
    {
        int ret = test3() + 2;
        ......
        return ret;
    }
    
    int test1()
    {
        int ret = test2() + 1;
        ......
        return ret;
    }
    
    int main()
    {
        ......
        int n = test1();
        ......
        return 0;
    }
    

    dump registers of the crashing thread
    Use info command to get the values of all registers

    (gdb) info registers
    x0             0x0      0
    x1             0x0      0
    x2             0x7fcbe33358     548881511256
    x3             0xa      10
    x4             0x1      1
    x5             0x0      0
    x6             0xb      11
    x7             0x0      0
    x8             0xa4     164
    x9             0x0      0
    x10            0x7fcbe32f88     548881510280
    x11            0x101010101010101        72340172838076673
    x12            0x1      1
    x13            0x1e     30
    x14            0x7faa6560f0     548319617264
    x15            0x7faa656100     548319617280
    x16            0x5595338fb8     367575404472
    x17            0x7faa597424     548318835748
    x18            0x0      0
    x19            0xffffffffffffffff       -1
    x20            0x7fcbe33348     548881511240
    x21            0x1      1
    x22            0x5595326d50     367575330128
    x23            0x0      0
    x24            0x0      0
    x25            0x0      0
    x26            0x0      0
    x27            0x0      0
    x28            0x0      0
    x29            0x7fcbe33220     548881510944
    x30            0x5595326f00     367575330560
    sp             0x7fcbe33220     0x7fcbe33220
    pc             0x7faa597434     0x7faa597434 <strlen+16>
    cpsr           0x40000000       1073741824
    fpsr           0x0      0
    fpcr           0x0      0
    

    dump memory near the stack of the crashing thread

    (gdb) x/100x 0x7fcbe33220
    0x7fcbe33220:   0xcbe33240      0x0000007f      0x95326f88      0x00000055
    0x7fcbe33230:   0xffffffff      0xffffffff      0xcbe33348      0x0000007f
    0x7fcbe33240:   0xcbe33260      0x0000007f      0x95327010      0x00000055
    0x7fcbe33250:   0xffffffff      0xffffffff      0xaa63e198      0x0000007f
    0x7fcbe33260:   0xcbe33280      0x0000007f      0x95327098      0x00000055
    0x7fcbe33270:   0xffffffff      0xffffffff      0xaa596468      0x0000007f
    0x7fcbe33280:   0xcbe332a0      0x0000007f      0x95326d7c      0x00000055
    0x7fcbe33290:   0xffffffff      0xffffffff      0x95326d50      0x00000055
    0x7fcbe332a0:   0xcbe332d0      0x0000007f      0xaa596478      0x0000007f
    0x7fcbe332b0:   0xcbe33358      0x0000007f      0x00000000      0x00000000
    0x7fcbe332c0:   0xffffffff      0xffffffff      0xffffffff      0xffffffff
    0x7fcbe332d0:   0xcbe33300      0x0000007f      0x95326e90      0x00000055
    0x7fcbe332e0:   0x00000000      0x00000000      0x00000000      0x00000000
    0x7fcbe332f0:   0x00000000      0x00000000      0x00000000      0x00000000
    0x7fcbe33300:   0x00000000      0x00000000      0xaa63f610      0x0000007f
    0x7fcbe33310:   0x00000000      0x00000000      0xcbe33340      0x0000007f
    0x7fcbe33320:   0x00000000      0x00000000      0x95338ce0      0x00000055
    0x7fcbe33330:   0x95338cf0      0x00000055      0x95338d00      0x00000055
    0x7fcbe33340:   0x00000001      0x00000000      0xcbe33a3a      0x0000007f
    0x7fcbe33350:   0x00000000      0x00000000      0xcbe33a4d      0x0000007f
    0x7fcbe33360:   0xcbe33a62      0x0000007f      0xcbe33a8e      0x0000007f
    0x7fcbe33370:   0xcbe33aa1      0x0000007f      0xcbe33d67      0x0000007f
    0x7fcbe33380:   0xcbe33da4      0x0000007f      0xcbe33dbd      0x0000007f
    0x7fcbe33390:   0xcbe33ddf      0x0000007f      0xcbe33dfe      0x0000007f
    0x7fcbe333a0:   0xcbe33e13      0x0000007f      0xcbe33e3d      0x0000007f
    

    conclusion
    In this post, we demonstrate how to run and setup gdb to load corefile. We also demonstrate basic gdb commands to show call stacks, registers, and some memory contents.

    android: coredump; how to make kernel dump core file after some process crashes

    October 27, 2015

    This post is to discuss how to make kernel dump core file after some process crashes. The testing code base is android-5.0.2_r1 . In android: coredump: analyze core file with gdb, we further demonstrates how to use gdb to analyze core files.

    enable coredump in kernel
    Enable CONFIG_COREDUMP while building kernel image.

    specify core file absolute path and name

    $ adb shell "echo /data/core/%p.%e.core > /proc/sys/kernel/core_pattern"
    
    1. %p indicates pid of the dumped process
    2. %e indicates the executable name

    set core file size for the process
    If the resource limit of core file size is 0, then kernel wouldn’t dump core file of this process after terminating. Thus, the process needs to set core file size if its resource limit is 0.

    int main()
    {
        struct rlimit core_limit;
        core_limit.rlim_cur = RLIM_INFINITY;
        core_limit.rlim_max = RLIM_INFINITY;
    
        if (setrlimit(RLIMIT_CORE, &core_limit) < 0) {
            ALOGD("Failed to setrlimit: %s", strerror(errno));
            return 1;
        }
    
        int n = test1();
        ALOGD("Ready to enter test");
    
        return 0;
    }
    

    an native process example which has core file after it crashes
    The native example coredumptest set resource limit of core file size as infinity. It then deferences NULL pointer and hit native crash.

    int test4()
    {
        int ret = strlen(NULL);
    
        ALOGD("enter %s: %d", __func__,  ret);
    
        return ret;
    }
    

    The core file is dumped under /data/core. In this case, 20577 is pid and coredumptest is the name of the executable file.

    $ adb shell ls /data/core
    20577.coredumptest.core
    

    conclusion
    This post shows how to make kernel dump core file after some process crashes. We also give an example coredumptest which will crash immediately and have core files if kernel supports coredump.

    reference

    1. Turn on core/crash dumps programatically
    2. Changing location of core dump

    %d bloggers like this: