[Crash-utility] Re: [Kgdb-bugreport] compiling kernel with -O0 flag (For optimal debugging with kgdb and/or crash).
Piet Delaney
piet at bluelane.com
Fri Sep 22 22:40:29 UTC 2006
On Thu, 2006-09-21 at 23:54 +0300, emin ak wrote:
> Dear All;
> Firstly thank you very much for your great effort for kgdb that makes
> kernel much understandable.
> I'am using kgdb to debug tcp-ip stack but I have experienced serious
> difficulties while debugging inline functions.
Hi Emin:
Yep, I edit "static inline" to "static_inline" and then define
static_inline as static for KGDB kernels.
In include/linux/compiler-gcc3.h and include/linux/compiler-gcc4.h
I added:
------------------------------------------------------------------
#if defined(CONFIG_KGDB) || defined(CONFIG_KEXEC)
# define static_inline static __attribute__ ((__unused__))
# define static__inline__ static __attribute__ ((__unused__))
# define INLINE __attribute__ ((__unused__))
# define __INLINE__ __attribute__ ((__unused__))
#else
# define static_inline static inline
# define static__inline__ static __inline__
# define INLINE inline
# define __INLINE__ __inline__
#endif
----------------------------------------------------------------------
I'm using it today to understand the device mapping and encryption code.
It's great! Inline's make skipping over code with the gdb 'next'
instruction impossible and you can't see the local variables.
I like having a large stack, compiling -O0 and without inlines
can increase the stack size. I think I notices more stability
by adding this to include/asm-i386/thread_info.h:
----------------------------------------------------------------------
#if defined(CONFIG_DEBUG_PREEMPT_AUDIT) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
#define THREAD_SIZE (8192 * 2)
#else
#ifdef CONFIG_4KSTACKS
#define THREAD_SIZE (4096)
#else
#define THREAD_SIZE (8192)
#endif
#endif
-----------------------------------------------------------------------
Without OPTIMIZATION I found the MMU code needs a tweak
in../linux-4/mm/memory.c:
------------------------------------------------------------------------
#if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
/*
* Allocate page upper directory.
*
* We've already handled the fast-path in-line, and we own the
* page table lock.
*/
pud_t fastcall *__pud_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned
long address)
{
.
.
.
}
#if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
/*
* Allocate page middle directory.
*
* We've already handled the fast-path in-line, and we own the
* page table lock.
*/
pmd_t fastcall *__pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned
long address)
{
.
.
}
---------------------------------------------------------------------------
Maybe I should have used #if !defined(__OPTIMIZE__)
in ../linux-4/mm/memory.c. Another change is I needed
to define a few network byte swapping functions. I currently
define them in ../linux-4/net/core/sock.c but I'm not
resistant to putting it in a better place:
----------------------------------------------------------------------------
/*
* If compiling -O0 we need to define
* these functions somewhere.
*/
#if !defined(__OPTIMIZE__)
#define ___htonl(x) __cpu_to_be32(x)
#define ___htons(x) __cpu_to_be16(x)
#define ___ntohl(x) __be32_to_cpu(x)
#define ___ntohs(x) __be16_to_cpu(x)
__u32 htonl(__be32 x) { return(___htonl(x)); }
__u32 ntohl(__be32 x) { return(___ntohl(x)); }
__be16 htons(__u16 x) { return(___htons(x)); }
__u16 ntohs(__be16 x) { return(___ntohs(x)); }
EXPORT_SYMBOL(htonl);
EXPORT_SYMBOL(ntohl);
EXPORT_SYMBOL(htons);
EXPORT_SYMBOL(ntohs);
#endif
------------------------------------------------------------------------------
Sometimes I only want to compile the tcp/ip code -O0, so I modified the
networking Makefiles and added:
-------------------------------------------------------------------------------
ifdef CONFIG_KGDB
CFLAGS += -gdwarf-2 -O0
else
ifdef CONFIG_KEXEC
CFLAGS += -gdwarf-2 -O0
endif
endif
------------------------------------------------------------------------------
In the top level kernel Makefile I have:
------------------------------------------------------------------------------
ifdef CONFIG_FRAME_POINTER
CFLAGS += -fno-omit-frame-pointer
else
CFLAGS += -fomit-frame-pointer
endif
#
# Compiling the complete kernel without optimization (-O0) for enhanced
debugging
# with kgdb/kdump requires ./mm/memory.c to have:
#
# if !defined(__PAGETABLE_PUD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
# and
# if !defined(__PAGETABLE_PMD_FOLDED) || defined(CONFIG_KGDB) ||
defined(CONFIG_KEXEC)
#
# A less invasive procedure is to use -O1 and only use -O0 for
networking code.
# The networking Makefiles have been setup to support this. So just
change
# -O0 to -O1 below and back out the kgdb change in ./mm/memory.c for a
# less invasive change. Compiling -O0 also required increasing
ROUNDUP_WAIT in
# linux/kernel/kgdb.c; value in 2.6.12 patch was way to low and value in
2.6.16
# is marginal and frequently causes lead CPU to times out prematurely
waiting for
# other CPU's to stop.
#
ifdef CONFIG_DEBUG_INFO
ifdef CONFIG_KGDB
CFLAGS += -gdwarf-2 -O0
else
ifdef CONFIG_KEXEC
CFLAGS += -gdwarf-2 -O1
else
CFLAGS += -g
endif
endif
endif
------------------------------------------------------------------------------------
> I know this is not a
> bug but with -O optimizations and inlines on tcp-ip stack, program
> counter goes everywhere madly even with step or next command and this
> makes debugging incomprehensible.
Yep, I don't understand why everyone else doesn't. It's also
like using debug printf's, I like being able to trace the code
to get the big picture and then a -O0 to look at details with kgdb.
Some believe doing this kind of stuff is blasphemy. The
Bible says I should be killed for working on Sunday; I
happen to disagree.
> At this point I have two questions:
> 1- Is there any way to compile kernel with -O0 flag and if it's
> possible, may it cause any problems?
I offered to post them to Amit back on Sept 06(2:45 PM) but I don't
think I ever heard back. I'd prefer to see the -O0 and KGDB_DEBUG
code for tracing the kgdb stub assimilated. If they would be accepted
I could make a patch to Tom's git repository...
> 2- Why does kernel fail while compiling with O0 flag and why does
> linux kernel depends on inline functions so much?
I think it's an obsession with performance. As long as I/we can map
"static inline" to "static" it's not a big deal.
> Is there anyone
> whoever uses kgdb for debugging linux tcp-ip stack or any effort to
> compile kernel with no optimization?
I'm using it every day; works great. I also recommend by SOCK_DEBUG,
SKB_DEBUG, and TCP_DEBUG macros to trace the TCP code. I also indent
the trace to make it easy to read.
function1() {
function2() {
function3() {
function4();
}
}
}
The brackets make it easy to see the scope of the trace with vi.
I like tracing with 'C' syntax since it what the reader is use to.
If folks are interested I could also add that to the git diff, but
I think that likely belongs else where and isn't likely the current
dogma. See snippet from attached network trace. I gave a talk
at a UNENIX conference back in the 1980 recommending a common UNIX
tracing paradigm and a few liked it. The director of Siemens,
Struck Zimmerman, didn't; you can't please everyone, so I just do what
I think is best and live with the world not being as I'd expect it to
be.
For TCP I'm using the attached sock.h fragment which has a
backward compatible SOCK_DEBUG() macro. I used the same paradigm in
skbuff.h; see attachment. Likewise I'm doing the same in kgdb.h; also
attached.
In printk I added:
--------------------------------------------------------------------
for (tp = tbuf; tp < tbuf + tlen; tp++)
emit_log_char(*tp);
printed_len += tlen - 3;
#ifdef CONFIG_PRINTK_INDENT
if (!in_interrupt()) {
int depth = stack_depth();
int i;
if ((depth > 0) && (depth < 120)) {
for(i = 0; i < depth; i++) {
emit_log_char(' ');
printed_len++;
}
}
}
#endif
-----------------------------------------------------------
and I added stack_depth() function
to ../linux-4/arch/i386/kernel/traps.c
-----------------------------------------------------------
int stack_depth(void)
{
struct thread_info *tinfo;
unsigned long ebp;
int depth = 0;
#ifdef CONFIG_FRAME_POINTER
asm("andl %%esp,%0; ":"=r" (tinfo) : "0" (~(THREAD_SIZE - 1)));
asm ("movl %%ebp, %0" : "=r" (ebp) : );
while (valid_stack_ptr(tinfo, (void *)ebp)) {
ebp = *(unsigned long *)ebp;
if (depth++ > 100) {
break;
}
}
#endif
return(depth);
}
---------------------------------------------------------------------------
Let me know if you you have any questions. Sounds like your on the right
track; IMHO.
-piet
>
> Thanks alot.
> Emin
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Kgdb-bugreport mailing list
> Kgdb-bugreport at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/kgdb-bugreport
--
Piet Delaney Phone: (408) 200-5256
Blue Lane Technologies Fax: (408) 200-5299
10450 Bubb Rd.
Cupertino, Ca. 95014 Email: piet at bluelane.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kgdb.h
Type: text/x-chdr
Size: 11156 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20060922/86de49e5/attachment.bin>
-------------- next part --------------
# ifdef SA_TCP_DEBUG
/*
* A similar socket debug macro exist is net/core/skbuff.h
*/
extern unsigned char sa_debug_global_flag;
#if 0
# define SA_DEBUG(msg...) \
do { \
if (sa_debug_global_flag) { \
printk(KERN_DEBUG "%s: ", __func__); \
printk(msg); \
} \
} while (0)
# endif
# define SOCK_DEBUG(sk, msg...) \
do { \
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# define SOCK_ERR(sk, msg...) \
do { \
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
printk(KERN_ERR "%s: ", __func__); \
printk(msg); \
} \
} \
} while (0)
# define TCP_SOCK_DEBUG(tp, msg...) \
do { \
struct sock *sk = &tp->inet.sk; \
\
if (sa_debug_global_flag && (sk) != NULL) { \
if ((sk)->socket_type || (sk)->vproxy_id || (sock_flag((sk), SOCK_DBG) )) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# define TW_SOCK_DEBUG(tw, msg...) \
do { \
if (sa_debug_global_flag && (tw) != NULL) { \
if ((tw)->socket_type || (tw)->vproxy_id) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Function Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# else /* !SA_TCP_DEBUG */
# define SOCK_DEBUG(sk, msg...) \
do { if ((sk) && sock_flag((sk), SOCK_DBG)) \
printk(KERN_DEBUG msg); \
} while (0)
# define SOCK_ERR(sk, msg...) do { } while (0)
// # define SA_DEBUG(tp, msg...) do { } while (0)
# define TCP_SOCK_DEBUG(tp, msg...) do { } while (0)
# define TW_SOCK_DEBUG(tw, msg...) do { } while (0)
# endif /* SA_TCP_DEBUG */
#else /* !SOCK_DEBUGGING */
// # define SA_DEBUG(sk, msg...) do { } while (0)
# define SOCK_DEBUG(sk, msg...) do { } while (0)
# define SOCK_ERR(sk, msg...) do { } while (0)
# define TCP_SOCK_DEBUG(tp, msg...) do { } while (0)
# define TW_SOCK_DEBUG(tw, msg...) do { } while (0)
#endif /* SOCK_DEBUGGING */
-------------- next part --------------
#ifdef CONFIG_VPROXY
#include <net/vproxy.h>
# if defined(SA_TCP_DEBUG) && !defined(CONFIG_ATM)
extern unsigned char sa_debug_global_flag;
/*
* A similar socket debug macro exist is net/sock.h
* SKB_DEBUG() unfortunatly is also defined by some
* CONFIG_ATM code; might switch to SA_SKB_DEBUG().
*/
# define SKB_DEBUG(skb, msg...) \
do { \
if (sa_debug_global_flag && (skb) != NULL) { \
if ((skb)->vdata.vproxy_flags & PROXY_PACKET ) { \
if (function_entry(msg)) { \
printk(KERN_DEBUG "%s", __func__); /* Entry */ \
} else { \
printk(KERN_DEBUG "%s: ", __func__); \
} \
printk(msg); \
} \
} \
} while (0)
# else /* !SA_TCP_DEBUG */
# else /* !CONFIG_VPROXY */
# define SKB_DEBUG(sk, msg...) do { } while (0)
#endif /* CONFIG_VPROXY */
-------------- next part --------------
Jun 1 19:17:59 localhost kernel: [ 3180.034711] vproxy_ip_local_deliver_finish(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034714] ip_local_deliver_finish(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034718] tcp_v4_rcv(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034721] sa_tcp_v4_rcv(skb:dfbf3c80) {
Jun 1 19:17:59 localhost kernel: [ 3180.034726] sa_tcp_v4_rcv: SA_TCP_INC_STATS(SA_TCP_STATS_PktReceived:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034731] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034739] sa_tcp_v4_rcv: skb:dfbf3c80->{dev->name:'u0', vdata.{outdev->name:'u0', indev->name:'p0'} }
Jun 1 19:17:59 localhost kernel: [ 3180.034741] ... TCP Packet with src:(192.168.136.103:32989) dest:(192.168.135.102:5001)
Jun 1 19:17:59 localhost kernel: [ 3180.034743] ... TCP FLAGS:' S ' Seq:214233949, End_seq:214233950, Ack_seq:0, Window:5840
Jun 1 19:17:59 localhost kernel: [ 3180.034745] ... skb->ip_summed:2, skb->csum:0x0, th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034749] sa_tcp_v4_rcv: 'Client' packet; sk:00000000 = __new_vproxy_tcp_v4_lookup_established();
Jun 1 19:17:59 localhost kernel: [ 3180.034753] sa_tcp_v4_rcv: tmp_checksum_received:2, skb:dfbf3c80->{ip_summed:2, csum:0x0} th->check 0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034756] sa_tcp_v4_rcv: Listening socket 0xde319080 state 10:Listen
Jun 1 19:17:59 localhost kernel: [ 3180.034761] sa_tcp_v4_rcv: Taking lock on client/listen sk:0xde319080
Jun 1 19:17:59 localhost kernel: [ 3180.034764] sa_tcp_v4_rcv: bh_lock_sock(sk:0xde319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034767] sa_tcp_v4_rcv: SA_TCP_INC_STATS(SA_TCP_STATS_PktReceivedForListeningSocket:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034771] sa_tcp_v4_rcv: Client/listen sock pkt: Got the locks
Jun 1 19:17:59 localhost kernel: [ 3180.034775] sa_tcp_v4_do_rcv(sk:0xde319080, skb:0xdfbf3c80) sk->sk_state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034779] sa_client_tcp_v4_hnd_req(sk:de319080, skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034784] vproxy_tcp_v4_search_req(tp:0xde319080, prevp:c0613a40, rport:56704, lport:35091, raddr:6788a8c0, laddr:6687a8c0) {
Jun 1 19:17:59 localhost kernel: [ 3180.034790] vproxy_tcp_v4_search_req: return(req:00000000);
Jun 1 19:17:59 localhost kernel: [ 3180.034792] }
Jun 1 19:17:59 localhost kernel: [ 3180.034794] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034799] __new_vproxy_tcp_v4_lookup_established: Failed to find established socket; return(NULL);
Jun 1 19:17:59 localhost kernel: [ 3180.034802] sa_client_tcp_v4_hnd_req: return(sk:de319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034806] sa_tcp_v4_do_rcv: LISTENING SOCKET; sk_state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034809] sa_tcp_rcv_state_process(sk:0xde319080, skb 0xdfbf3c80, th:0xf7efe074, len:40) sk->state:10:'Listen'
Jun 1 19:17:59 localhost kernel: [ 3180.034816] sa_tcp_v4_conn_request(sk:0xde319080, skb:0xdfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034819] sa_tcp_v4_conn_request: SA_TCP_INC_STATS(SA_TCP_STATS_NewSynReceived:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034827] sa_tcp_v4_conn_request: SA_TCP_INC_STATS(SA_TCP_STATS_OpenreqCreated:0)
Jun 1 19:17:59 localhost kernel: [ 3180.034833] sa_server_create_peer_request(sk:de319080, skb:dfbf3c80, rx_opts:c06137e8, client_req:e6622800, server_req:e6622780)
Jun 1 19:17:59 localhost kernel: [ 3180.034839] sa_server_create_peer_request: server_tcp_req:e6622780->snt_isn:0x0 = client_tcp_req:e6622800->rcv_isn:0xcc4f35d;
Jun 1 19:17:59 localhost kernel: [ 3180.034845] sa_server_tcp_v4_send_syn: Sending out the SYN packet without modifying ANY data.
Jun 1 19:17:59 localhost kernel: [ 3180.034849] sa_server_tcp_v4_send_syn: buff:dfbf3c80->{ip_summed:0, csum:0, h.th->check:0x6ef0}
Jun 1 19:17:59 localhost kernel: [ 3180.034853] sa_server_tcp_v4_send_syn: ntohl(buff:dfbf3c80->h.th->seq:0x5df3c40c):0xcc4f35d [NEW]
Jun 1 19:17:59 localhost kernel: [ 3180.034857] sa_server_tcp_v4_send_syn: tcp_req:e6622800->{rcv_isn:0xcc4f35d, snt_isn:0x0}
Jun 1 19:17:59 localhost kernel: [ 3180.034860] sa_server_tcp_v4_send_syn: peer_tcp_req:e6622780->{rcv_isn:0xcc4f35d, snt_isn:0xcc4f35d}
Jun 1 19:17:59 localhost kernel: [ 3180.034865] sa_tcp_forward_L4_packet(skb:0xdfbf3c80) skb->{ip:summed:0, csum:0, checksum:0x6ef0}
Jun 1 19:17:59 localhost kernel: [ 3180.034869] sa_ip_fast_output(skb:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034876] sa_ip_fast_output: skb:dfbf3c80->{dev->name:'<<No Dev>>', vdata.{outdev->name:'u0', indev->name:'p0'} }
Jun 1 19:17:59 localhost kernel: [ 3180.034878] ... TCP Packet with src:(192.168.136.103:32989) dest:(192.168.135.102:5001)
Jun 1 19:17:59 localhost kernel: [ 3180.034880] ... TCP FLAGS:' S ' Seq:214233949, End_seq:214233950, Ack_seq:0, Window:5840
Jun 1 19:17:59 localhost kernel: [ 3180.034882] ... skb->ip_summed:0, skb->csum:0x0, th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034886] sa_xmit_ip_pkts(skb:dfbf3c80) skb->vdata.{l2type:0, l2addlen:0}, skb->h.th->check:0x6ef0
Jun 1 19:17:59 localhost kernel: [ 3180.034891] __skb_queue_tail(list:c2516314, newsk:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034895] __skb_dequeue: return(result:dfbf3c80)
Jun 1 19:17:59 localhost kernel: [ 3180.034899] sa_ip_fast_output: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034902] tcp_reset_keepalive_timer(sk:de319080, len:2ee)
Jun 1 19:17:59 localhost kernel: [ 3180.034906] sa_tcp_v4_conn_request: return(0);
Jun 1 19:17:59 localhost kernel: [ 3180.034909] sa_tcp_v4_do_rcv: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034912] sa_tcp_v4_rcv: Release client/listen sock lock sk:0xde319080
Jun 1 19:17:59 localhost kernel: [ 3180.034916] sa_tcp_v4_rcv: bh_unlock_sock(sk:0xde319080);
Jun 1 19:17:59 localhost kernel: [ 3180.034919] sa_tcp_v4_rcv: return(ret:0);
Jun 1 19:17:59 localhost kernel: [ 3180.034921] }
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20060922/86de49e5/attachment.sig>
More information about the Crash-utility
mailing list