============================================================== Linux Kernel State Tracer (LKST) -------------------------------- Date: Aug, 9, 2004 Version: 2.1.1 Authors: Yumiko Sugita Kazuyoshi Serizawa Tetsuhito Nakamura Keisuke Hatasaki Takahiro Tsurukubo Yasunori Goto Satoshi Sakamoto Takao Indoh Masami Hiramatsu Katsuhisa Ogasawara ============================================================== COPYRIGHT (C) HITACHI,LTD. 2001-2004 COPYRIGHT (C) FUJITSU,LTD. 2001-2004 ============================================================== INDEX ----- 1. Introduction 1.1 Release Notes 2. Installation 3. How To Use 3.1 How to insert new trace point (and new event) 3.2 How to extract LKST trace data from LKCD dump file 3.3 How to program an event handler 3.4 How to port an event handler to LKST 2.0 3.5 How to build new event handlers 3.6 The Format of lkst_etypes 4. Known Issues 5. Future plans 6. Reporting Problems and Discussing LKST 7. License 8. Change History 1. Introduction --------------- Linux Kernel State Tracer(LKST) records information as trace data about events in the Linux Kernel. It records various events like process context switch, send signal, exception, memory allocation, send packet, and so on. This program is a facility for analyzing trouble of the Linux Kernel. It becomes possible that developers investigate troubles by this facility without stopping important(used for Mission Critical purposes) server. Functions of present Linux could not offer enough information for developers to solve the trouble of the system. - Standard Kernel:When the kernel is panic, it displays only the registers and stack information. If developers want more information, it is necessary to insert printk() in the kernel and recompile it. - LKCD :LKCD store a dump when the system crash. But, if developers use it only, they cannot investigate the sequence of events till kernel panic. (LKCD is a.k.a. Linux Kernel Crash Dumps) Event information of the kernel is always recorded as trace data on the server under running by using LKST. So, these problems can be solved. Moreover, LKST has some features. - It is possible to change dynamically which events are recorded. Developers can obtain information about the events which they concern only. And, it reduces the overhead of components which is not related with trouble. - It is possible to change each handlers related each events. A default handler of all events is just recording the events. But, if it is necessary, default handler can be changed. This function can be used as following, +Notify user processes when the handler detects abnormal status. +Change a maskset,which controls what type of events should be recorded, dynamically by the handler. -> The system can usually run with a few events trace for the cause of good performance. In addition when the handler detects abnormal status, it can change a maskset to get more detail information. 1.1 Release Notes ----------------- - Based on linux-2.6.6 - Add the security logging module. - Display masksets appropriately. - Add the event-type flag. - Fix some bugs. 2. Installation --------------- See file of "Install". 3. How To Use ------------- Please refer the online manual for details of each command. 1) Display the present kernel trace data It is necessary to execute 2 commands. a. Get a log buffer from kernel Enter the command as follows. Then LKST saves the trace data into file. % lkstbuf read -f logfile b. Display the trace data Enter the command as follows. This command displays the trace data. You can specify various options to filter the traced data. % lkstbuf print -f logfile ------------------------------------------------------------------- event_type=context_switch cpu=02, pid=00001008 time=Tue Mar 26 18:48:53.143274134 2002 arg1=0xd68a4000 0x00000000 : pointer to task_struct(prev) arg2=0xf2dde000 0x00000000 : pointer to task_struct(next) arg3=0x00000001 0x0000000a : process state/process count event_type=mem_get_freepage cpu=02, pid=00001008 time=Tue Mar 26 18:48:53.143270345 2002 arg1=0xecc0f000 0x00000000 : pointer to page arg2=0x000001f0 0x00000000 : type of page(gfp_mask) arg3=0x00000000 0x00000000 : the number of page(order) arg4=0xc0168197 0x00000000 : call address event_type=mem_malloc cpu=02, pid=00001008 time=Tue Mar 26 18:48:53.143268419 2002 arg1=0x00000018 0x00000000 : size arg2=0x000001f0 0x00000000 : flags arg3=0xdfe3e5a0 0x00000000 : objp arg4=0xc0168578 0x00000000 : call address ... ... ------------------------------------------------------------------- 2) Change which events are recorded. You can change the traced event types by doing as follows. a.Get a maskset file. At first, please get a maskset file which control what event types should be recorded by LKST. % lkstm read -m id -d | grep 0x > maskset_file "id" is the decimal number which is used as an identifier of the maskset. Id 0, 1, and 2 are the reserved maskset; Id 0 does not record any event types, id 1 records all of the event types, and id 2 records the default event types. b.Edit the maskset file. Next, edit the saved maskset_file. In default, event of spinlock() is not recorded, so id of event handler for event 0x080 is 0x0ff. (Event type number of spin_lock() is 0x080.) : : 0x07f 0x01 0x080 0xff <<< 0x081 0xff : : To record the event, please change the value "0x0ff" to "0x001" at right side of the line which contains its event number. : : 0x07f 0x01 0x080 0x01 <- change 0x081 0xff : : Oppositely, If one event is not wanted to record, please delete the line which is written it's event number. ex) event type of system call entry ( number = 0x030). : : 0x02f 0x01 0x030 0xff <- change 0x031 0x01 : : All the of Event type numbers are written in lkst-events-2.02.pdf that is available from download page. c.Write the new maskset. The new maskset is written when the next command is executed. % lkstm write -m id -f maskset_file "id" can be specified from 4 to 254. At the same time, you can specify the name of the new maskset with "-n" option. Instead of previous, do as below. % lkstm write -m id -n new_maskset_name -f maskset_file "new_maskset_name" must be different from the name of other masksets. Please repeat the operation from a. to c., if you want to make more masksets. If you want to confirm the state of the maskset, maskset_read have to be executed alone. % lkstm read -m id d.Select maskset In order to enable new maskset, execute the lkstm command. % lkstm set -m id e.Confirm which maskset is currently selected. If you want to know which maskset has been selected, use lkst_status command. % lkst status ------------------------------------------------------------------- version of LKST : 2.1.0 number of cpus : 4 number of masksets : 4 number of event-handlers: 3 current_maskset_id : 2 current_buffer_id (cpu: 000): 0 current_buffer_id (cpu: 001): 0 current_buffer_id (cpu: 002): 0 current_buffer_id (cpu: 003): 0 ------------------------------------------------------------------- ID of the selected maskset is shown to the right of "current_maskset_id" in the forth line. In this example, it is 2. And the number of right of "number of masksets" is total number of the masksets. In this example, it is 4. cf. Other information means as follows. "number of event-handlers" :number of registered event handlers "number of cpus" :number of system cpus "current_buffer_id" :ID of currently selected buffers for all CPUs 3) Invoke LKST logging daemon LKST logging daemon continues to output the contents of event buffer to log files. Please refer the online manual of lkstlogd(8) for details. Name of the files are /var/log/lkst/sebufCC.S, CC represents cpu id corresponds to the file and S represents a sequence number (described later). If directory /var/log/lkst does not exist, please make it as follows. % mkdir /var/log/lkst This is an example of command to invoke the daemon, % lkstlogd -a -b 2M -l 10M -n 2 LKST logging daemon will stop and restart to write the file when the daemon receives a signal SIGUSR1. To send the signal, this is convenient, % kill -USR1 `cat /var/run/lkstlogd.pid` The size of these files is limited as specified by -l option, so old records would be overwritten. To avoid overwritten, LKST logging daemon can change files. Said sequence number will be increased on this change and to discriminate new set of files and old files. To change file, % kill -USR2 `cat /var/run/lkstlogd.pid` examples/lkst_watch_daemon is an example of this feature. This script watches unexpected termination of another daemon, then change the files. 4) Dump logs in panic() Please do "3) Install addon tools" in "Install". After that, just do following, % insmod lkst_mod_panicdump.ko maskset_id_base=2 A parameter "maskset_id_base" specifies which maskset to be used as a template of the new maskset. For testing, this module can call panic(). See 6)-c below. 5) How to display stack trace on each event a.Insmod the stack trace module. Please do "3) Install addon tools" in "Install". After that, do following, % insmod lkst_mod_stacktrace.ko maskset_id_base=2 A parameter "maskset_id_base" specifies a template. b.Execute special formatter Execute lkst_mod_stacktrace_read.pl. % lkstbuf read > logfile % perl lkst_mod_stacktrace_read.pl \ -s /boot/System.map-`uname -r` -k /proc/ksyms -d logfile You can omit all of the options. The default value for -s and -k are the same as those values of above command line. If option -d is omitted, this formatter reads from current event buffers. ------------------------------------------------------------------- 0x00057f24: 0000269005494693 [mc], \ type: INT_HARDWARE_ENTRY(0x0010), pid: 0000000000, cpu: 000 \ 0x00000000 0x00000000 0x00000001 0x00000000 \ 0x00000000 0x00000000 0x00000000 0x00000000 \ 0x00057f26: 0000269005511255 [mc], \ type: INT_TASKLETHI_ENTRY(0x0012), pid: 0000000000, cpu: 000 \ 0xc01257d0 0x00000000 0x00000000 0x00000000 \ 0x00000000 0x00000000 0x00000000 0x00000000 \ 0x00057f28: 0000269005513862 [mc], \ type: INT_BH_ENTRY(0x0016), pid: 0000000000, cpu: 000 \ 0x00000000 0x00000000 0xc012a390 0x00000000 \ 0x00000000 0x00000000 0x00000000 0x00000000 \ ------------------------------------------------------------------- NOTE: Contents of stack trace might include some unrelated address. 6) Change event handlers As described on the "Introduction", users can register additional event handler in order to add function more than recording the events. Please refer addons/lkst_mod_panicdump.c as an example. a.Register new event-handler. b.Change maskset. When lkst_mod_panicdump.o is loaded, the initializer in the module will do them automatically. The initializer will register lkst_evhandler_panicdump() as a new eventhandler that ID is 128, and create a new maskset named "maskset_panicdump". c.Control event handler. The control function of the event handler can invoke panic() for testing. % echo panic | lksteh ctrl -e 128 -f - The control function compares the argument and "panic", if correspond, the call panic("LKST PANIC TEST"); 7) Add user buffer If a new user buffer is added, data can be recorded in the added buffer. It becomes possible that a lot of data can be recorded without overflowing the standard buffer. a.First, Create a buffer (or buffers if you run on SMP system). % lkstbuf create -s is the size of the new buffer. b.Next, Select the new buffer to record. % lkstbuf jump -b Please refer examples/lkst_test_run for more information. This is an example of switching the buffer where data is recorded when the specified command is executed. % cd examples % sh lkst_test_run 65536 ps aux The first argument is the size of new buffer, and the latter arguments are command to execute. 8) How to use LKST as an event counter a.Insmod the counter module. Make addons/lkst_mod_event_count.ko, and load it. b.Execute command. Execute addons/lkst_event_count, and then the total of each event counter is displayed. 3.1 How to insert a new trace point (and a new event) ----------------------------------------------------- 1) Define a new event: If you want to use a new event, you should define it before use. LKST supports several ways to define a new event. There is a table to help determining how and where to define it. +----------+------------------------------+-----------------------------+ |The event | Including the kernel binary | Only kernel module(s) | |is caused +---------+----------+---------+---------+---------+---------+ |on ... |One core |Functions |Inline |One |Several |Inline | | |function |capable to|function |module |modules |function | | |(one |build as |or |(one | |or | | |location)|module(s) |several |location)| |several | | | | |locations| | |locations| +==========+=========+==========+=========+=========+=========+=========+ |Definition|(Arch. depended) |The same |The module loaded | |place | include/asm-*/lkst_etypes.h |module |first (the root | | |(Others/General) | |module of module's | | | include/linux/lkst_etypes.h | |dependency tree) | +----------+------------------------------+---------+-------------------+ |Definition| | | |macro | LKST_ETYPE_DEF | LKST_ETYPE_DEF_MODULE (*) | | | | | +----------+---------+----------+---------+---------+---------+---------+ |Hook type | | | | | | | |(**) | NORMAL | MODULE | INLINE | NORMAL | MODULE | INLINE | | | | | | | | | +----------+---------+----------+---------+---------+---------+---------+ |Hook macro| | | | | | |LKST_HOOK| LKST_HOOK_INLINE |LKST_HOOK| LKST_HOOK_INLINE | | | | | | | +----------+---------+--------------------+---------+-------------------+ |Event-type| | | | | | flag(***)| 0x0001 | 0x0003 | 0x0001 | 0x0003 | | | | | | | +----------+---------+--------------------+---------+-------------------+ (*) When using this macro, you should initialize and terminate new event in proper functions. For more detail, see following 1.b) section. (**) If the event that never used by any LKST_HOOK* macro (for example, the event is passed to lkst_evhandlerprim_entry_log() directory), define it as UNDEF hook-type. (***) The details about this flag are in 3.6 For more detail, see following sections. 1.a) If new trace points into kernel binary (vmlinux) with a new event, define new event type in linux/lkst_events.h as follows. LKST_ETYPE_DEF(event-ID, hook-type, event-symbol, "event-name", \ "1st argument description", \ "2nd argument description", \ "3rd argument description", \ "4th argument description") event-ID --- Value of the event type(0x000..0xfff) 0x000-0x0ff: preset 0x100-0x1ff: user 0x200-0xeff: reserved 0xf00-0xfff: system hook-type --- Type of hook header for Kernel Hooks Specify either the following according to the inserting location of the HOOK macro. - NORMAL: If you insert HOOK macro in the kernel ( in other words, in vmlinux), use this type. - MODULE: If you insert HOOK macro in a/some module(s) (or selectable to build as a module), use this type. - INLINE: If you insert HOOK macro in the in-line function of the kernel, use this type. If you insert the same HOOK macro in the two or more places, use this. - UNDEF: If you do not use this event by HOOK macro, use this type. NOTE: If you insert HOOK macro in the in-line function of the module, use MODULE type. ~~~~~~ event-symbol --- The mnemonic of the event type event-name --- The description of the event type 1..4th arguments descriptions : The description for each arguments. (example) LKST_ETYPE_DEF(0x100, NORMAL, NEW_EVENT, "new event", \ "current task", "target task", NULL, NULL) 1.b) If the trace points are ONLY in the modules, the macro for declaration of HOOK header can be put in the prime module (IOW, the root module of dependency tree). And the functions for the initialization and termination of HOOK are inserted respectively in the functions of the module-initialization and module-termination. Declaration macro for HOOK LKST_ETYPE_DEF_MODULE(event-ID, hook-type, event-symbol, \ "event-name", \ "1st argument description", \ "2nd argument description", \ "3rd argument description", \ "4th argument description") !NOTE!: you can specify NORMAL to hook-type if this event is ONLY caused in the same module and in one location. Initialization function lkst_hook_etype_register(&LKST_ETYPE_INFO(event-symbol)); Termination function lkst_hook_etype_unregister(&LKST_ETYPE_INFO(event-symbol)); See sample program "examples/lkst_mod_trc.c". 2) Insert following HOOK macro where you want to add the trace point. Add following sentence to the top of the file in which the trace point is added. #include - LKST_HOOK_INLINE(etype, argument1, argument2, argument3, argument4) If you insert the same HOOK macros in the two or more places, or insert it in the in-line function or in the macro function, use this HOOK. - LKST_HOOK(etype, argument1, argument2, argument3, argument4) The case excepts the above-mentioned. * etype : The value of etype is equal to event-ID. Also, You can use LKST_ET_SYM(event-symbol) or LKST_ETYPE_event-symbol. * argument1..4 : Arguments for HOOK. 3) Use with user-space tools. Since version 2.0.1, lkstbuf and lkstm commands support '-E' option with a CSV formatted file describing events to analyze. Also the LKST module support /proc/lkst_etypes interface that provide information of events currently registered. So, when using above two commands, you can specify /proc/lkst_etypes with -E option as following: % lkstm read -m 3 -E /proc/lkst_etypes % lkstbuf print -E /proc/lkst_etypes -f lkstbuf.log This feature is also useful to analyze off-lined system's log. First, store the system's event description with the system's log as following: % lkstbuf read -f lkstbuf.log % cat /proc/lkst_etypes > etypes.csv Then, you can analyze it with stored event description. % lkstbuf print -E etypes.csv -f lkstbuf.log 3.2 How to extract LKST trace data from LKCD dump file ------------------------------------------------------ !!NOTE!! THIS FEATURE IS EXPERIMENTAL IN THIS VERSION(2.1.1)!! PLEASE USE AT YOUR OWN RISK!! a.Create dump file. See LKCD documents that can be downloaded at http://lkcd.sourceforge.net/. (i) Save system memory to dump device. Enter the command as follows. % lkcd config % echo 1 > /proc/sys/kernel/sysrq Hit Sysrq+C, then start saving system memory to dump device, and rebooting automatically. NOTE: At default, system memory is saved at swap disk. So if swapon runs in booting your system, system memory saved at swap disk is collapsed. Please save system memory at another disk except swap disk, or comment out the line of swapon in the rc.sysinit. (ii) Crate dump file. Enter the command as follows. Dump file is created at /var/log/dump/N (N is an integer of 0 or more.). % lkcd save b.Run lcrash command (i) Enter the command as follows. lcrash command starts and prompt is shown. % cd /var/log/dump/N (N is an integer of 0 or more.) % lcrash -n N map = map.N, dump = dump.N, outfile = stdout, kerntypes = kerntypes .N Please wait... Initializing vmdump access ... Done. Loading system map ................................. Done. Loading type info (Kerntypes) ... Done. Initializing arch specific data ... Done. Loading ksyms from dump ....... Done. lkstdumplib being initialized >> (ii) Run lkstdump sub-command Enter the command as follows. Trace data is written to file . >> lkstdump logfile static buffer write(2048 records) cpu[0].buffer[0] write(1087 records) cpu[1].buffer[0] write(1087 records) 's file format is the same as the file created by "lkstbuf read", so you can see by using "lkstbuf print". 3.3 How to program an event handler ----------------------------------- You can make and add new event-handler as a kernel module. However, it requires high-level of programming ability and knowledge about linux kernel. Please do it at your own risk. 1) Include following four files. #include #include #include #include The kernel module requires first three files. And LKST requires last one. 2) Describe module information. MODULE_AUTHOR(/*write your name*/); MODULE_DESCRIPTION(/*write description of this module*/); MODULE_LICENSE("GPL");/*support GPL only*/ Do *NOT* forget last description. Only GPL'ed modules can use kernel APIs of LKST. 3) Write an event-handler function. You can use lkst_evhandlerprim_* functions in this function. static void lkst_evhandler_example(void *phookrec, int event_type, lkst_arg_t arg1, lkst_arg_t arg2, lkst_arg_t arg3, lkst_arg_t arg4) { preempt_disable(); /*IMPORTANT*/ /*TODO: do something*/ preempt_enable(); /*VERY IMPORTANT*/ } !!!NOTE!!! : DO NOT forget to disable preemption in the handler, or the handler may log an entry to a buffer owned by another processor. 4) Write an event-handler control function.(optional) You can do something to control handler's behavior. static int lkst_evhandler_example_ctrl(void *buf, size_t bufsize) { /*TODO: do something*/ } 5) Write sysfs interface functions.(optional) static ssize_t example_ctrl_store(struct lkst_eh_device *ed, const char *buf, size_t count) { /*TODO: do something*/ return count; /* return 0 means sleep forever.*/ } static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf) { /*TODO: do something*/ return strlen(buf) + 1; /*you must return how many characters you wrote.*/ } 6) Declare module initializer and terminator. static int mod_init(void); static void mod_cleanup(void); module_init(mod_init); /*define as initializer*/ module_exit(mod_cleanup); /*define as terminator*/ 7) Define an lkst_eh_device for registering to lkst class. static LKST_EH_DEV_DEF(/*the name of evhandler*/, lkst_evhandler_example, lkst_evhandler_example_ctrl, example_ctrl_store, example_ctrl_show); !! If the name of evhandler is HOGE, you just write HOGE in 1st argument, Do NOT write "HOGE" as a string!! 8) Program module initializer. In initializer, you have to register new event-handler to LKST. Following is sample code. static int mod_init(void) { int retval; /* register an event handler */ retval = lkst_eh_device_register(&LKST_EH_DEV(/*the name of evhandler*/)); if (retval < 0) goto init_err; return 0; init_err: mod_cleanup(); /*If occurs an error, invokes terminator.*/ return retval; } 9) Program module terminator. In terminator, you have to deregister your event-handler from LKST. Following is sample code. (WARNING: *MUST* deregister it, or kernel might panic.) static void mod_cleanup(void) { if (LKST_EH_DEV(/*the name of evhandler*/).id != LKST_EVHANDLER_ID_VOID) { lkst_eh_device_unregister(&LKST_EH_DEV(/*the name of evhandler*/)); } } 3.4 How to port an event handler to LKST 2.0 -------------------------------------------- * You can use old interface yet. However, it is not recommended. * 1) Define an lkst_eh_device data structure out of functions. static LKST_EH_DEV_DEF(example, example_handler, example_ctrl, NULL, NULL); This code define a structure named 'lkst_eh_dev_example', initialize that name member to "example", evhandler to example_handler, evhandler_ctrl to example_ctrl, and so on. In other words, this macro is expanded as following: static struct lkst_eh_device lkst_eh_dev_example = { .name = "example", .id = LKST_EVHANDLER_ID_VOID, .evhandler = example_handler, .evhandler_ctrl = example_ctrl, .ctrl_store = NULL, .ctrl_show = NULL, .class_dev.class = &lkst_class, .class_dev.dev = NULL, }; There are new members, ctrl_store, ctrl_show, and class_dev. class_dev is a sysfs related data structure, so you should not touch that without thorough understanding. Remaining two new members are discussed in step 4. 2) Register the lkst_eh_device data structure in the module initializer. Instead of using lkst_evhandler_register(), you can use lkst_eh_device_register() to register an event handler. lkst_eh_device_register(&LKST_EH_DEV(example)); LKST_EH_DEV() macro make the symbol (name) from the argument. for example, LKST_EH_DEV(example) is expanded to lkst_eh_dev_example. If succeed, this function returns zero. If fail, it returns an error value less than zero. This behavior different from lkst_evhandler_register(). If you want to know what id number is assigned, please check LKST_EH_DEV(example).id . 3) Deregister the lkst_eh_device data structure in the module terminator. Instead of using lkst_evhandler_unregister(), you can use lkst_eh_device_unregister() to deregister an event handler. lkst_eh_device_unregister(&LKST_EH_DEV(example)); !NOTE!: This function returns nothing (void). 4) Optionally (But recommended), you can define ctrl_store and ctrl_show functions to be able to access from user-mode more easily. These two functions are new features. The ctrl_store function stores control data from user-mode via sysfs interface. And the ctrl_show shows module internal data to user-mode. static ssize_t example_ctrl_store(struct lkst_eh_device *ed, const char *buf, size_t count) { /*TODO: do something*/ return count; /* return 0 means sleep forever.*/ } static ssize_t example_ctrl_show(struct lkst_eh_device *ed, char *buf) { /*TODO: do something*/ return strlen(buf) + 1; /*you must return how many characters you wrote.*/ } Do NOT return zero from the ctrl_store() function. It should return the number of read bytes. Also, the ctrl_show() should return the number of written bytes. After defined these functions, rewrite the definition of the lkst_eh_device data structure as following: static LKST_EH_DEV_DEF(example, example_handler, example_ctrl, example_ctrl_store, example_ctrl_show); 3.5 How to build new event handlers ---------------------------------- Suppose that you have two event handlers(lkst_mod_example1.c, lkst_mod_example2.c) in a directory (~/example_module/), and the package of lkst is extracted into /usr/src. (there is /usr/src/lkst-2.1.0/addons). 1) Copy and rename addons/Makefile.skel to Makefile in the directory. $ cp addons/Makefile.skel example_modules/Makefile 2) Edit the Makefile according to comments written in it. for example: ---- KMODSRC := lkst_mod_example1.c lkst_mod_example2.c KMODOBJ := $(KMODSRC:.c=.o) KMOD_KO := $(KMODOBJ:.o=.ko) PROGSRC := PROGOBJ := $(PROGSRC:.c=.o) PROGBIN := $(PROGOBJ:.o=) ifneq ($(KERNELRELEASE),) obj-m := $(KMODOBJ) else TOPDIR := /usr/src/lkst-2.1.0/addons SUB_DIRS := ADDCMD-clean := ADDCMD-distclean := include $(TOPDIR)/Makefile.base endif ---- 3) run make command. $ cd example_modules $ make 3.6 The Format of lkst_etypes ----------------------------- You can get information of event-types from /proc/lkst_etypes as following. 0x001,PROCESS_CONTEXTSWITCH,0x0001,"context_switch","pointer to task_struct prev ","pointer to task_struct next ","process state","process count" 0x002,PROCESS_WAKEUP,0x0001,"process_wakeup","pointer to wakeup-process p ","process state","synchronus","" 0x003,PROCESS_SIGSEND,0x0001,"process_sigsend","signal number sig ","pointer to process to which signal is sent","pointer to info structure info ","" ... The format of this file is CSV(Comma Separated Value). Each columns mean event-ID, event-mnemonic, event-type flag, event-name, the description of 1st argument, the description of 2nd argument, the description of 3rd argument, and the description of 4th argument respectively. The "event-type flag" is the bitmap of flags. The least significant bit means that the event is "maskable". You can set/unset event handlers to maskable events. 2nd bit means that the event is recorded from multiple place in kernel. This flag corresponds to hook-type of the event as following. UNDEF = 0x0002 : An UNDEF hook-type event may be logged at multiple place. And can't be changed whether it record or not by maskset. NORMAL = 0x0001 : A NORMAL hook-type event is logged at specified place. And can be change the event-handler by maskset. INLINE = 0x0003 : An INLINE hook-type event may be logged at multiple place. And can be change the event-handler by maskset. MODULE = 0x0003 : The MODULE hook-type is same as the INLINE. 4. Known Issues --------------- - The time data in the event records written by IOCTL(LKST_IOC_BUFFER_READ) and lkst_buffer_read do not conform to the draft of POSIX standard 1003.25. 5. Future plans --------------- - Support LKCD again. (make lkstdump.sial working without lkstdumplib.so) - Support raw buffer read mode. - Usability improvement. - Performance improvement. - Add more extensions for analyzing various faults. - Add connectivity to Event Logging (http://evlog.sourceforge.net/). - Cleanup sources and documents. 6. Reporting Problems and Discussing LKST ----------------------------------------- email: lkst-develop@lists.sourceforge.net or lkst-develop@lists.sourceforge.jp visit: https://sourceforge.net/projects/lkst/ or https://sourceforge.jp/projects/lkst/ or http://oss.hitachi.co.jp/sdl/english/lkst.html 7. License ---------- COPYRIGHT (C) HITACHI,LTD. 2001-2004 COPYRIGHT (C) FUJITSU,LTD. 2001-2004 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 8. Change History ----------------- 2.1.1 9th of August, 2004 Release v2.1.1 Update for Linux 2.6.7 Add three workqueue-related events. Drop two obsolate config options. Cleanup codes. Fix some bugs. 2.1 16th of June, 2004 Release v2.1 Update for Linux 2.6.6 Add the security logging module. Display masksets appropriately. Add the event-type flag. Update Documents. Fix some bugs. 2.0.1 14th of January, 2004 Release v2.0.1 Update for Linux 2.6.0 Fix some scripts. Update Documents. Remove unused events. Add -E option to lkstbuf and lkstm. Fix some bugs. 2.0 29th of October, 2003 Release v2.0 Update for Linux 2.6 Support sysfs. Support procfs. Support new kernelhooks. Add new APIs and macros. Cleanup and split sources. Add some scripts. Don't support LKCD (currently). Fix some bugs. 1.5 5th of August, 2003 Release v1.5 Add watchdog tools. Make makefile more useful. Add overwritten detection event. Add -S option to lkstbuf print. Modify default maskset. Fix some bugs. 1.5-rc2 8th of May, 2003 Trial release 2 Fix some bugs. 1.5-rc1 6th of May, 2003 Trial release 1 Merge patches for IPF(IA64). Change bitwidth and dword order of arguments. 1.4 13th of March, 2003 Release v1.4 Fix the display format of time. Change the results of LKST_IOC_BUFFER_LIST to enable to get the number of buffers. 1.4-rc2 18th of February, 2003 Trial release 2 Fix same bugs. Add some configurable kernel options. Add lkst_mod_pmc.c . Add LKST_ETYPE_OOPS_PGFAULT event. Add LKST_ETYPE_OOPS_NMIWDOG event. Simplify format of the list of buffers. Disable logging while kernel dumping. Add lkstbuf print -V option. 1.4-rc1 24th of January, 2003 Trial release Rewrite most of codes associated with event- buffers. Separate resources per processor. Awake reading process in schedule(). Allow to log lock event with lkstlogd. Support inter-buffer jump feature. Update commands. Update addons. Add light-weight handler as an addon module. Modify kernel APIs and ioctl interfaces. Add maskset configuration feature. Add the version of LKST in lkst.h. Fix some bugs. 1.3.1 25th of November, 2002 Release v1.3.1 Update for kernel 2.4.19. Separate Kernel Hooks from LKST patch. Add module that record only specified process. Make sure lkstlogd release own resources. lkstlogd report warning to syslog. Add and modify APIs for event-handler and maskset. Fix some bugs. 1.3 30th of August, 2002 Release v1.3 Performance improvement of Kernel Hooks. Modify default maskset. Support CSV format by lkstbuf command. Add and modify APIs for event-handler. Add config to delete hooks on inline func. Add script to make /dev/lkst. Fix some bugs. 1.2 28th of June, 2002 Release v1.2 Performance improvement of default handler. 1.1 14th of May, 2002 Release v1.1 Replace LKCD patch with the official one. 1.01 26th of April, 2002 Release v1.01 Update lkstlogd command. Add trace data of exception events. Fix some bugs. 1.0 12th of April, 2002 Release v1.0 Add kernel crash dump function by using LKCD. Update some command. Add log header to log file. Fix some bugs. 0.9 31st of March, 2002 Release v0.9 for some enhancements. Add LKST logging daemon, Replace trace point hook to use Kernel Hooks, Add some add-ons and examples. 0.1a 18th of January, 2002 Fix bug of the racing between etrc_evhandler_default() and etrc_buffer_{delete,create}(). Add some minor modification, Clear some comments. 0.1 14th of December, 2001 First public drop See changelog to know more details.