|
Patch Kit-0003 for Digital UNIX Version 4.0B (a.k.a. BL5) is now available
from dupatchwww:
http://www.zk3.dec.com/dupatchwww/v40pats.html
Included in this kit are a number of important threads fixes.
============================ << START OF PATCH >> ==============================
New PatchID: 96.00
PATCH ID: OSF410-400210 SUBSET(s): OSFBASE410
********************************************************************************
SUPERSEDED PATCHES: OSF410-400131 (24.00), OSF410-400195 (84.00)
PROBLEM: (QAR #49720) (Patch ID: OSF410-400131)
********
Multithreaded applications running on SMP systems, may find their
applications hanging because a thread is never told that a mutex
has been unlocked or that a condition variable has been signaled.
To recognize this problem, use ladebug to display the list of mutexes
and condition variables.
(ladebug) pthread "mutex -faq; condition -faq"
This will display a very long list of mutexes and condition variables,
including those that belong to libc or libpthread. An example of the
output follows:
(ladebug) pthread "mutex -faq; condition -faq"
.
.
.
Mutex 103 (normal) "mutex at 0x140000b10" (0, block 0x140008b90) is
orphaned blocking structure, 2 threads waiting; waiters: 56, 51
.
.
.
Condition variable 203, "cond at 0x14030b510" (0) is orphaned blocking
structure, 2 threads waiting; waiters: 91, 5
.
.
.
The application will hang forever because the threads are waiting on a
blocking structure not attached to the mutex or condition variable.
PROBLEM: (QAR #49520) (Patch ID: OSF410-400131)
********
Multithreaded applications running on SMP systems, may find their
applications hanging because of a corrupted mutex.
To recognize this problem, use ladebug to display the list of mutexes
that are locked:
(ladebug) pthread "mutex -fal"
This will display a list of locked mutexes. Some mutexes will be
recursive or error check type mutexes. Those types of mutexes record
the thread that owns the lock on the mutex. The corruption is exposed
as a recursive or error check mutex without a thread owner:
(ladebug) pthread "mutex -fal"
Mutex 56 (recursive) "exc cr" (0x3ffc0082f80, block 0x140009d10) is
locked (depth 1), 4 threads waiting; ref count is 4
The output should have a "by thread x" after the "is locked" line above.
PROBLEM: (QAR #49520) (Patch ID: OSF410-400131)
********
Multithreaded applications may overflow the stacks of created threads
prematurely. Applications may receive a SEGV and dump core at unexpected
places when this problem occurs. The SEGV should be because a memory
location in or near the thread's stack was referenced.
To recognize this problem, use ladebug to run appliation in question and
wait for the SEGV to occur. Then find the thread that took the SEGV and
check to see what value of the stack pointer is. Then display the full
information on the thread itself and compare the current stack pointer
to the limits displayed. An example debug session:
(ladebug) run <application>
Thread received signal SEGV
stopped at [hstTransferContext: ??? 0x3ff80572ca8]
(ladebug) p $sp
0x140028bb8
(ladebug) pthread "thread -f x"
thread 2 (running) "<anonymous>" (0x140006030), created by pthread
Scheduling: throughput policy at priority 11
Masked signals: none
Pending signals: none
Terminated, result value 0
Object flags: none; self flags: none; sched flags: none; mutex flags:
terminated; atomic flags: none
Thread specific data: 0=0x140006488
Stack: 0x14002fa38; base is 0x140030000, guard area at 0x140027fff
General cancelability disabled, asynch cancelability disabled
Current vp is 0, synch port is 0, vp ID is 0
Join uses mutex 22 and condition variable 7; wait uses mutex 23 and
condition variable 8
The thread's start function and argument are 0x120001bc0
(0x3ffc01834c8)
The thread's latest errno is 0
You should see that there is a considerable amount of free space on the
stack. (0x140028bb8 - 0x140027fff = 0xbb9 or 3001 bytes!)
PROBLEM: (QAR #49804) (Patch ID: OSF410-400131)
********
Multithreaded applications which use fork() may find their applications
hanging because of a deadlock situation.
To recognize this problem, use ladebug to display the list of locked
mutexes and their waiters.
(ladebug) pthread "mutex -alrq"
Mutex 1 (normal) "malloc" is locked by 13848, 4 threads waiting;
waiters: 13846, 3, 567, 8
Mutex 10 (normal) "VM 0 cache" is locked, 1 thread waiting; waiters: 1
Mutex 13 (normal) "VM 3 cache" is locked, 1 thread waiting; waiters: 13848
In the example above, the malloc mutex is held by thread 13848, but it is
waiting on mutex 13. A stack trace inspection reveals that mutex 13 is
held by thread 13846.
PROBLEM: (QAR 50961) (Patch ID: OSF410-400195)
********
The native exception handling facility may corrupt the users stack. This is
most easily seen in an application using threads since the exception handling
facility is always present there. But other applications linked with "-lexc"
may see this problem as well.
A typical stack trace from Ladebug showing this problem may look like this:
(ladebug) t
Stack trace for thread 2
>0 0x3ff80579a94 in /usr/shlib/libpthread.so
#1 0x3ff80110b98 in /usr/shlib/libc.so
#2 0x3ff80110318 in __tis_read_unlock(0x3ff8011031c, 0x3ffc0092500,
0x3ff8017345c, 0x3ffc0092500, 0x3ffc0087fe0, 0x0)
DebugInformationStrippedFromFile738
#3 0x3ff80173458 in /usr/shlib/libc.so
#4 0x3ff807b2004 in exc_lookup_function_entry(0x14005d180, 0x3ff807b1d30,
0x14005d180, 0x23bd5eb8, 0x3ff807b325c, 0x14005c640)
DebugInformationStrippedFromFile0
#5 0x3ff807b3258 in UnknownProcedure16FromFile1(0x0, 0x14005d9a8, 0x100000001,
0x14005c640, 0xabadabad00beed00, 0x3ff80579ac0)
DebugInformationStrippedFromFile1
#6 0x3ff807b3634 in exc_unwind(0x100000001, 0x14005c640, 0xabadabad00beed00,
0x3ff80579ac0, 0x3ff807b3b04, 0x3ff807b3aa0)
DebugInformationStrippedFromFile1
#7 0x3ff807b3b00 in __Ots_CSpecificHandler(0x3ff807b3980, 0x14005d6f8,
0x14005d6d0, 0x14005d180, 0x3ff00000000, 0x14005d408)
DebugInformationStrippedFromFile4
#8 0x3ff807b2ac4 in UnknownProcedure12FromFile1(0x0, 0x3ff8055c8d8, 0x0,
0x63ff0000, 0x14005de78, 0x0) DebugInformationStrippedFromFile1
#9 0x3ff807b2da0 in exc_dispatch_exception(0x14005de78, 0x3ff8055c460,
0x14005d180, 0x23bd5f3c, 0x3ff807b24e0, 0x0)
DebugInformationStrippedFromFile1
#10 0x3ff807b2534 in exc_raise_exception(0x3ffc00802a0, 0x0, 0x3ff80579a60,
0x0, 0x3ffc0087650, 0x120002604) DebugInformationStrippedFromFile1
#11 0x14005d17c
Note that the last frame (frame #11 above) does not have any information
printed. This is symptomatic of the stack being corrupted in such a way
that the debugger cannot interpret it correctly.
PROBLEM: (QAR 50392) (Patch ID: OSF410-400195)
********
Some applications have bugs in them that developers are trying to find. The
nature of these bugs might be to corrupt the internal workings of malloc()
or free() such that calling them causes a SEGV.
When this happens, it is possible for the native exception handling routines
to block while accessing their internal procedure descriptor list, due to
some other thread performing a lookup on that list. If the exception handling
routines actually try to block, it is possible for the blocking operation
itself to try to allocate memory and to call malloc() again. This attempt to
call malloc() will hang.
A typical stack trace from Ladebug showing this error might look like this:
(ladebug) t
>0 0x3ff8057ca24 in /usr/shlib/libpthread.so
#1 0x3ff8057a450 in hstTransferContext(0x1, 0x140048030, 0x3ffc018a010, 0x4,
0x3ffc0188a50, 0x90) DebugInformationStrippedFromFile109:???
#2 0x3ff8056429c in dspDispatch(0x14000c210, 0x140048c68, 0x140048af0, 0x8f4,
0x3ffc0082770, 0x0) DebugInformationStrippedFromFile89:???
#3 0x3ff80567b98 in pthread_mutex_block(0x3ffc018a1c0, 0x3ffc018a1d0,
0x1400e6ef0, 0x8f4, 0x3ffc0082770, 0x0)
DebugInformationStrippedFromFile95:???
#4 0x3ff8057c820 in __pthread_mutex_lock(0x3ffc018a1d0, 0x1400e6ef0, 0x8f4,
0x3ffc0082770, 0x0, 0x3ff80575e88)
DebugInformationStrippedFromFile111:???
#5 0x3ff80575e84 in UnknownProcedure3FromFile103(0x8f4, 0x3ffc0082770, 0x0,
0x3ff80575e88, 0x3ff800d194c, 0x0)
DebugInformationStrippedFromFile103:???
#6 0x3ff800d1948 in malloc(0x3ff800d194c, 0x0, 0x3ff80579170, 0x90, 0x0,
0x3ffc0184e70) DebugInformationStrippedFromFile22:???
#7 0x3ff8057916c in vmAlloc(0x3ff8055d94c, 0x3ffc0082870, 0x0, 0x0,
0x3ffc018a1c0, 0x3ff8055d934) DebugInformationStrippedFromFile108:???
#8 0x3ff8055d948 in cvGetBlock(0x3ffc0082818, 0x3ffc0082800, 0x14000c040,
0x3ffc008f7ec, 0x3ffc008f868, 0x3ff8057c8fc)
DebugInformationStrippedFromFile1:???
#9 0x3ff8055f574 in cvWait(0x3ffc008f868, 0x8f4, 0x3ffc0082870, 0x140048af0,
0x0, 0x3ff80575e88) DebugInformationStrippedFromFile1:???
#10 0x3ff8055d45c in __pthread_cond_wait(0x3ffc0082870, 0x140048af0, 0x0,
0x3ff80575e88, 0x3ff8010fd54, 0x0) DebugInformationStrippedFromFile1:???
#11 0x3ff8010fd50 in /usr/shlib/libc.so
#12 0x3ff8010f570 in __tis_write_lock(0x3ff8010f574, 0x3ff8019241c,
0x3ff80172aec, 0x1, 0x3ff800ccda0, 0x1)
DebugInformationStrippedFromFile735:???
#13 0x3ff80172ae8 in __exc_lookup_function_entry(0x0, 0x0, 0x3ffc018a010, 0x0,
0x3ff800d1540, 0x1401e5020) DebugInformationStrippedFromFile304:???
#14 0x3ff807b2004 in exc_lookup_function_entry(0x3ffc018a010, 0x0,
0x3ff800d1540, 0x1401e5020, 0x3ff807b2bec, 0x1400249e0)
DebugInformationStrippedFromFile0:???
#15 0x3ff807b2be8 in exc_dispatch_exception(0x0, 0x12009407c, 0x140122910,
0x140020000, 0x0, 0x0) DebugInformationStrippedFromFile1:???
#16 0x3ff807b38c4 in exc_raise_signal_exception(0x80, 0x0, 0x3ff8019241c, 0x1,
0x1, 0x140048af0) DebugInformationStrippedFromFile3:???
#17 0x3ff800d3b28 in __sigtramp(0x3ff8019241c, 0x1, 0x1, 0x140048af0,
0x140122ef8, 0x0) DebugInformationStrippedFromFile105:???
#18 0x3ff8019241c in UnknownProcedure22FromFile22(0x3ffc0080c50, 0x1,
0x14017cc00, 0x2f, 0x140024f20, 0x3ff8057c824)
DebugInformationStrippedFromFile22:???
#19 0x3ff80192f40 in UnknownProcedure17FromFile22(0x3ff801943c8, 0x1, 0x10,
0x3ffc0080e18, 0x72, 0x140024f20) DebugInformationStrippedFromFile22:???
#20 0x3ff801943c4 in /usr/shlib/libc.so
#21 0x3ff80194604 in /usr/shlib/libc.so
#22 0x3ff800d1958 in malloc(0x3ff800d195c, 0x0, 0x12008e3bc, 0x140024f20,
0x1401232a0, 0x14017cc00) DebugInformationStrippedFromFile22:???
#23 0x12008e3b8 in lxldalc() /vobs/nlsrtl3/src/lxi/lxldalc.c:54
Note that frame #23 has a call to malloc(). In frame #16 a synchronous signal
(such as a SEGV) is taken, which results in an exception being raised. Notice
in frame #6 that malloc() is called again as part of handling the exception,
which leads to the thread blocking. This is because the previous call to
malloc() caused it to lock its internal mutex for synchronization purposes.
This recursive call to malloc() is the problem. The user might be able to fix
the cause of the original SEGV if its accompanying exception is fully processed.
This patch allows the SEGV to be processed fully and correctly.
PROBLEM: (QAR 49804) (Patch ID: OSF410-400195)
********
Threaded applications may see hangs when trying to use fork(). A prior patch
fixed only a part of the problem.
The user sees the child process with a stack trace from Ladebug like this:
(ladebug) t
>0 0x3ff8057ca24 in /usr/shlib/libpthread.so
#1 0x3ff8057a450 in hstTransferContext(0x1, 0x140ae11f0, 0x3ffc018a010, 0x4,
0x3ffc0188a50, 0x3ffc01852c0) DebugInformationStrippedFromFile109:???
#2 0x3ff8056429c in dspDispatch(0x140051810, 0x1408641a8, 0x140864030, 0x0,
0x3ffc01851d0, 0x0) DebugInformationStrippedFromFile89:???
#3 0x3ff80567b98 in pthread_mutex_block(0x3ffc008ae28, 0x3ffc01876f8,
0x140621c00, 0x0, 0x3ffc01851d0, 0x0)
DebugInformationStrippedFromFile95:???
#4 0x3ff8057c820 in __pthread_mutex_lock(0x3ffc01876f8, 0x140621c00, 0x0,
0x3ffc01851d0, 0x0, 0x3ff8057984c)
DebugInformationStrippedFromFile111:???
#5 0x3ff80579848 in vmReinit(0x0, 0x3ff8057984c, 0x3ff80578e70, 0x0,
0x3ffc0080c50, 0x3ffc0081de0) DebugInformationStrippedFromFile108:???
#6 0x3ff80578e6c in /usr/shlib/libpthread.so
#7 0x3ff800d36e0 in __fork(0x120040830, 0x0, 0x140684bc0, 0x0, 0x1f, 0x0)
DebugInformationStrippedFromFile335:???
#8 0x12004082c in forkTranslator(inoutp_handle=0x140a9b8f8,
inoutp_operation_data=0x140763a00) dp_xlatr.c:3610
#9 0x12004040c in forkedTranslatorInterface(inoutp_handle=0x140a9b8f8,
in_operation_code=eConnect, inoutp_operation_data=0 x140763a00,
getr=0x0, putr=0x0, arg_p=0x0) dp_xlatr.c:3471
#10 0x12003e0c0 in dp_callTranslator(inoutp_handle=0x140a9b8f8,
in_operation_code=eConnect, inoutp_operation_data=0x140763a00,
inp_scb=0x140621c00, getr=0x0, putr=0x0, arg_p=0x0) dp_xlatr.c:1894
#11 0x12003d878 in dp_connectToTranslator(inp_scb=0x140621c00) dp_xlatr.c:1468
#12 0x1200649d0 in dp_pjaConnectToTranslator(inp_Scb=0x140621c00,
in_docPdlOid=1252, inoutp_printerPdlOid=0x140a9b988,
outp_needsTranslator=0x140a9b9a8) pja_doc.c:726
#13 0x12005b4a0 in startDocument_1(getr=0x12004bb0c, putr=0x12005dc40,
inp_fib=0x140a53c00) dp_pja.c:1508
#14 0x12004b234 in fiberRoutineJacket(inp_fib=0x140a53c00) dp_jbstp.c:1276
#15 0x12004b430 in fibThreadJacket(inp_fib=0x140a53c00) dp_jbstp.c:1331
#16 0x3ff80574b94 in thdBase(0x0, 0x3ffc0183430, 0x3ffc018a010, 0x1, 0x45586732,
0x3) DebugInformationStrippedFromFile101:???
Note that frame #7 above shows the call to fork(). The DECthreads library is
then entered with an attempt to lock a DECthreads internal mutex in frame #4.
This attempt causes the thread to block while waiting for another thread to
unlock the mutex. But because this is the only thread left in the child process
after the parent forks, this mutex will never be unlocked. The parent failed
to lock this mutex during the fork operation and so is incorrectly left in the
locked state in the child.
PROBLEM: (HPXQ10P01, HPXQB7HGW) (Patch ID: OSF410-400195)
********
For a threaded application written on a system prior to Digital UNIX V4.0 that
uses C and the DECthreads exception handling macros, or a threaded application
written with C++ on any version of Digital UNIX that also uses the DECthreads
exception handling macros, various synchronous signals might not be caught
by the application's exception handlers.
The DECthreads library is not properly raising synchronous signals as exceptions
that the application can catch.
PROBLEM: (QAR 51033) (Patch ID: OSF410-400195)
********
Threaded applications using pthread_kill() may experience a stack corruption.
This is rather difficult to detect.
PROBLEM: (QAR 51214) (Patch ID: OSF410-400195)
********
A threaded application can experience a DECthreads bug check with a message
that mentions the routine nxm_resched(). This application might also hang
with one of its threads having a stack trace containing a frame for a call
to the routine errBugcheck().
The formatted DECthreads bugcheck message might look like this:
DECthreads bugcheck (version V3.13-435), terminating execution.
vpUpcallThreadUnblocked: (os/kern) invalid argument (4) nxm_resched(20,0)
For a hung application in this scenario, a typical stack trace from Ladebug
might look like this:
(ladebug) t
>0 0x3ff80568b20 in pthread_mutex_unblock(0x3ffc00802a0, 0x0, 0x0,
0x3ffc0181580, 0x3ffc018a968, 0x0)
DebugInformationStrippedFromFile95:???
#1 0x3ff8057c1f8 in __pthread_mutex_unlock(0x0, 0x0, 0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618)
DebugInformationStrippedFromFile111:???
#2 0x3ff80575614 in UnknownProcedure5FromFile103(0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618, 0x3ff800d6904, 0x3ffc0187760)
DebugInformationStrippedFromFile103:???
#3 0x3ff800d6900 in fflush(0x3ff800d6904, 0x3ffc0187760, 0x3ff80565cfc,
0x3ffc018a968, 0x3ffc0187768, 0x0)
DebugInformationStrippedFromFile84:???
#4 0x3ff80565cf8 in errBugcheck(0x3ffc0185da0, 0x3ffc01700d8, 0x4, 0x26e,
0x0, 0x3ffc018aae0) DebugInformationStrippedFromFile90:???
#5 0x3ff8057bcd4 in UnknownProcedure26FromFile109(0x0, 0x14014b438,
0x3ffc018a968, 0x3ffc018a968, 0x1, 0x14014b438)
DebugInformationStrippedFromFile109:???
#6 0x3ff8057b0d8 in UnknownProcedure20FromFile109(0x500, 0x1f4, 0x1,
0xffffffffdfbbee16, 0x1f4, 0x14014b478)
DebugInformationStrippedFromFile109:???
#7 0x3ff80535200 in msg_receive(0x0, 0x3ffc018abf0, 0x0, 0x0, 0x3ffc018a968,
0x0) DebugInformationStrippedFromFile6:???
#8 0x3fefffffffd in ???
the kernel to call into user space to perform an operation). Note that
frame #4 above contains the DECthreads bugcheck call, but also note that it
eventually calls pthread_mutex_unblock() (frame #0 above). This call to
pthread_mutex_unblock() hangs because it is incorrect to call pthread_mutex_
unblock() within the context of an upcall.
PROBLEM: (QAR 51214) (Patch ID: OSF410-400210)
********
A threaded application might hang while attempting to issue a message from the
DECthreads bugcheck. A stack trace of a thread in this state will contain
frames for calls to the routines errBugcheck(), fflush() and pthread_mutex_
unblock().
(ladebug) t
>0 0x3ff80568b20 in pthread_mutex_unblock(0x3ffc00802a0, 0x0, 0x0,
0x3ffc0181580, 0x3ffc018a968, 0x0) DebugInformationStrippedFromFile95:???
#1 0x3ff8057c1f8 in __pthread_mutex_unlock(0x0, 0x0, 0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618) DebugInformationStrippedFromFile111:???
#2 0x3ff80575614 in UnknownProcedure5FromFile103(0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618, 0x3ff800d6904, 0x3ffc0187760)
DebugInformationStrippedFromFile103:???
#3 0x3ff800d6900 in fflush(0x3ff800d6904, 0x3ffc0187760,
0x3ff80565cfc, 0x3ffc018a968, 0x3ffc0187768, 0x0)
DebugInformationStrippedFromFile84:???
#4 0x3ff80565cf8 in errBugcheck(0x3ffc0185da0, 0x3ffc01700d8, 0x4,
0x26e, 0x0, 0x3ffc018aae0) DebugInformationStrippedFromFile90:???
#5 0x3ff8057bcd4 in UnknownProcedure26FromFile109(0x0, 0x14014b438,
0x3ffc018a968, 0x3ffc018a968, 0x1, 0x14014b438)
DebugInformationStrippedFromFile109:???
#6 0x3ff8057b0d8 in UnknownProcedure20FromFile109(0x500, 0x1f4, 0x1,
0xffffffffdfbbee16, 0x1f4, 0x14014b478)
DebugInformationStrippedFromFile109:???
#7 0x3ff80535200 in msg_receive(0x0, 0x3ffc018abf0, 0x0, 0x0,
0x3ffc018a968, 0x0) DebugInformationStrippedFromFile6:???
#8 0x3fefffffffd in ???
Note that frame #8 above is not readable. This is to be expected because the
problem is occuring in an "upcall" frame. (This "upcall" frame is used by
kernel to call into user space to perform an operation). Note that frame #4
above contains the DECthreads bugcheck call, but also note that it eventually
calls pthread_mutex_unblock() (frame #0 above). This call to pthread_mutex_
unblock() hangs because it is incorrect to call pthread_mutex_unblock()
within the context of an upcall.
PROBLEM: (EVMS-GRYPHON-FT QAR 00237) (Patch ID: OSF410-400210)
********
This patch fixes a situation where threads would hang in the pthread_once()
or cma_once() routines.
The DECthreads routines pthread_once() and cma_once() use the DECthreads global
mutex. The first thread to call pthread_once() or cma_once() would lock this
mutex and subsequent threads calling pthread_once() or cma_once() would hang.
This problem was fixed on OpenVMS and the patch has been ported to Digital
UNIX 4.0.
PROBLEM: (IPMT HPAQ11MMTG) (Patch ID: OSF410-400210)
********
For the signals traditionally representing synchronous errors in a program
(such as SIGPIPE, SIGSYS, SIGSEGV, SIGBUS, SIGFPE, SIGEMT, SIGIOT, SIGILL),
DECthreads catches the signal and converts it into an equivalent exception.
Digital UNIX 4.0 attempts to print an abort message for an unhandled exception.
If an unhandled exception occurs in a libc library I/O routine, the thread
causing the exception may hang in abort() because of a deadlock over an I/O
mutex.
FILE(s):
/usr/shlib/libexc.so subset OSFBASE410
CHECKSUM: 55605 48
/usr/shlib/libpthread.so subset OSFBASE410
CHECKSUM: 30836 416
/usr/shlib/libpthreads.so subset OSFBASE410
CHECKSUM: 34736 120
/usr/shlib/libpthreaddebug.so subset OSFBASE410
CHECKSUM: 65353 144
--------------------------------------------------
|
| The "BL5" Patch kits are now available for Digital UNIX V4.0 (Kit-0002),
V4.0A (Kit-0002) and V4.0B (Kit-0003). The kits and release notes can
be obtained from dupatchwww:
http://www.zk3.dec.com/dupatchwww/v40pats.html
Included in these kits are the following important threads patches:
OSF400-210 (V4.0)
OSF405-400210 (V4.0A)
OSF410-400210 (V4.0B)
Below is a summary of the fixes contained in these patches:
============================ << START OF PATCH >> ==============================
New PatchID: 96.00
PATCH ID: OSF410-400210 SUBSET(s): OSFBASE410
********************************************************************************
SUPERSEDED PATCHES: OSF410-400131 (24.00), OSF410-400195 (84.00)
PROBLEM: (QAR #49720) (Patch ID: OSF410-400131)
********
Multithreaded applications running on SMP systems, may find their
applications hanging because a thread is never told that a mutex
has been unlocked or that a condition variable has been signaled.
To recognize this problem, use ladebug to display the list of mutexes
and condition variables.
(ladebug) pthread "mutex -faq; condition -faq"
This will display a very long list of mutexes and condition variables,
including those that belong to libc or libpthread. An example of the
output follows:
(ladebug) pthread "mutex -faq; condition -faq"
.
.
.
Mutex 103 (normal) "mutex at 0x140000b10" (0, block 0x140008b90) is
orphaned blocking structure, 2 threads waiting; waiters: 56, 51
.
.
.
Condition variable 203, "cond at 0x14030b510" (0) is orphaned blocking
structure, 2 threads waiting; waiters: 91, 5
.
.
.
The application will hang forever because the threads are waiting on a
blocking structure not attached to the mutex or condition variable.
PROBLEM: (QAR #49520) (Patch ID: OSF410-400131)
********
Multithreaded applications running on SMP systems, may find their
applications hanging because of a corrupted mutex.
To recognize this problem, use ladebug to display the list of mutexes
that are locked:
(ladebug) pthread "mutex -fal"
This will display a list of locked mutexes. Some mutexes will be
recursive or error check type mutexes. Those types of mutexes record
the thread that owns the lock on the mutex. The corruption is exposed
as a recursive or error check mutex without a thread owner:
(ladebug) pthread "mutex -fal"
Mutex 56 (recursive) "exc cr" (0x3ffc0082f80, block 0x140009d10) is
locked (depth 1), 4 threads waiting; ref count is 4
The output should have a "by thread x" after the "is locked" line above.
PROBLEM: (QAR #49520) (Patch ID: OSF410-400131)
********
Multithreaded applications may overflow the stacks of created threads
prematurely. Applications may receive a SEGV and dump core at unexpected
places when this problem occurs. The SEGV should be because a memory
location in or near the thread's stack was referenced.
To recognize this problem, use ladebug to run appliation in question and
wait for the SEGV to occur. Then find the thread that took the SEGV and
check to see what value of the stack pointer is. Then display the full
information on the thread itself and compare the current stack pointer
to the limits displayed. An example debug session:
(ladebug) run <application>
Thread received signal SEGV
stopped at [hstTransferContext: ??? 0x3ff80572ca8]
(ladebug) p $sp
0x140028bb8
(ladebug) pthread "thread -f x"
thread 2 (running) "<anonymous>" (0x140006030), created by pthread
Scheduling: throughput policy at priority 11
Masked signals: none
Pending signals: none
Terminated, result value 0
Object flags: none; self flags: none; sched flags: none; mutex flags:
terminated; atomic flags: none
Thread specific data: 0=0x140006488
Stack: 0x14002fa38; base is 0x140030000, guard area at 0x140027fff
General cancelability disabled, asynch cancelability disabled
Current vp is 0, synch port is 0, vp ID is 0
Join uses mutex 22 and condition variable 7; wait uses mutex 23 and
condition variable 8
The thread's start function and argument are 0x120001bc0
(0x3ffc01834c8)
The thread's latest errno is 0
You should see that there is a considerable amount of free space on the
stack. (0x140028bb8 - 0x140027fff = 0xbb9 or 3001 bytes!)
PROBLEM: (QAR #49804) (Patch ID: OSF410-400131)
********
Multithreaded applications which use fork() may find their applications
hanging because of a deadlock situation.
To recognize this problem, use ladebug to display the list of locked
mutexes and their waiters.
(ladebug) pthread "mutex -alrq"
Mutex 1 (normal) "malloc" is locked by 13848, 4 threads waiting;
waiters: 13846, 3, 567, 8
Mutex 10 (normal) "VM 0 cache" is locked, 1 thread waiting; waiters: 1
Mutex 13 (normal) "VM 3 cache" is locked, 1 thread waiting; waiters: 13848
In the example above, the malloc mutex is held by thread 13848, but it is
waiting on mutex 13. A stack trace inspection reveals that mutex 13 is
held by thread 13846.
PROBLEM: (QAR 50961) (Patch ID: OSF410-400195)
********
The native exception handling facility may corrupt the users stack. This is
most easily seen in an application using threads since the exception handling
facility is always present there. But other applications linked with "-lexc"
may see this problem as well.
A typical stack trace from Ladebug showing this problem may look like this:
(ladebug) t
Stack trace for thread 2
>0 0x3ff80579a94 in /usr/shlib/libpthread.so
#1 0x3ff80110b98 in /usr/shlib/libc.so
#2 0x3ff80110318 in __tis_read_unlock(0x3ff8011031c, 0x3ffc0092500,
0x3ff8017345c, 0x3ffc0092500, 0x3ffc0087fe0, 0x0)
DebugInformationStrippedFromFile738
#3 0x3ff80173458 in /usr/shlib/libc.so
#4 0x3ff807b2004 in exc_lookup_function_entry(0x14005d180, 0x3ff807b1d30,
0x14005d180, 0x23bd5eb8, 0x3ff807b325c, 0x14005c640)
DebugInformationStrippedFromFile0
#5 0x3ff807b3258 in UnknownProcedure16FromFile1(0x0, 0x14005d9a8, 0x100000001,
0x14005c640, 0xabadabad00beed00, 0x3ff80579ac0)
DebugInformationStrippedFromFile1
#6 0x3ff807b3634 in exc_unwind(0x100000001, 0x14005c640, 0xabadabad00beed00,
0x3ff80579ac0, 0x3ff807b3b04, 0x3ff807b3aa0)
DebugInformationStrippedFromFile1
#7 0x3ff807b3b00 in __Ots_CSpecificHandler(0x3ff807b3980, 0x14005d6f8,
0x14005d6d0, 0x14005d180, 0x3ff00000000, 0x14005d408)
DebugInformationStrippedFromFile4
#8 0x3ff807b2ac4 in UnknownProcedure12FromFile1(0x0, 0x3ff8055c8d8, 0x0,
0x63ff0000, 0x14005de78, 0x0) DebugInformationStrippedFromFile1
#9 0x3ff807b2da0 in exc_dispatch_exception(0x14005de78, 0x3ff8055c460,
0x14005d180, 0x23bd5f3c, 0x3ff807b24e0, 0x0)
DebugInformationStrippedFromFile1
#10 0x3ff807b2534 in exc_raise_exception(0x3ffc00802a0, 0x0, 0x3ff80579a60,
0x0, 0x3ffc0087650, 0x120002604) DebugInformationStrippedFromFile1
#11 0x14005d17c
Note that the last frame (frame #11 above) does not have any information
printed. This is symptomatic of the stack being corrupted in such a way
that the debugger cannot interpret it correctly.
PROBLEM: (QAR 50392) (Patch ID: OSF410-400195)
********
Some applications have bugs in them that developers are trying to find. The
nature of these bugs might be to corrupt the internal workings of malloc()
or free() such that calling them causes a SEGV.
When this happens, it is possible for the native exception handling routines
to block while accessing their internal procedure descriptor list, due to
some other thread performing a lookup on that list. If the exception handling
routines actually try to block, it is possible for the blocking operation
itself to try to allocate memory and to call malloc() again. This attempt to
call malloc() will hang.
A typical stack trace from Ladebug showing this error might look like this:
(ladebug) t
>0 0x3ff8057ca24 in /usr/shlib/libpthread.so
#1 0x3ff8057a450 in hstTransferContext(0x1, 0x140048030, 0x3ffc018a010, 0x4,
0x3ffc0188a50, 0x90) DebugInformationStrippedFromFile109:???
#2 0x3ff8056429c in dspDispatch(0x14000c210, 0x140048c68, 0x140048af0, 0x8f4,
0x3ffc0082770, 0x0) DebugInformationStrippedFromFile89:???
#3 0x3ff80567b98 in pthread_mutex_block(0x3ffc018a1c0, 0x3ffc018a1d0,
0x1400e6ef0, 0x8f4, 0x3ffc0082770, 0x0)
DebugInformationStrippedFromFile95:???
#4 0x3ff8057c820 in __pthread_mutex_lock(0x3ffc018a1d0, 0x1400e6ef0, 0x8f4,
0x3ffc0082770, 0x0, 0x3ff80575e88)
DebugInformationStrippedFromFile111:???
#5 0x3ff80575e84 in UnknownProcedure3FromFile103(0x8f4, 0x3ffc0082770, 0x0,
0x3ff80575e88, 0x3ff800d194c, 0x0)
DebugInformationStrippedFromFile103:???
#6 0x3ff800d1948 in malloc(0x3ff800d194c, 0x0, 0x3ff80579170, 0x90, 0x0,
0x3ffc0184e70) DebugInformationStrippedFromFile22:???
#7 0x3ff8057916c in vmAlloc(0x3ff8055d94c, 0x3ffc0082870, 0x0, 0x0,
0x3ffc018a1c0, 0x3ff8055d934) DebugInformationStrippedFromFile108:???
#8 0x3ff8055d948 in cvGetBlock(0x3ffc0082818, 0x3ffc0082800, 0x14000c040,
0x3ffc008f7ec, 0x3ffc008f868, 0x3ff8057c8fc)
DebugInformationStrippedFromFile1:???
#9 0x3ff8055f574 in cvWait(0x3ffc008f868, 0x8f4, 0x3ffc0082870, 0x140048af0,
0x0, 0x3ff80575e88) DebugInformationStrippedFromFile1:???
#10 0x3ff8055d45c in __pthread_cond_wait(0x3ffc0082870, 0x140048af0, 0x0,
0x3ff80575e88, 0x3ff8010fd54, 0x0) DebugInformationStrippedFromFile1:???
#11 0x3ff8010fd50 in /usr/shlib/libc.so
#12 0x3ff8010f570 in __tis_write_lock(0x3ff8010f574, 0x3ff8019241c,
0x3ff80172aec, 0x1, 0x3ff800ccda0, 0x1)
DebugInformationStrippedFromFile735:???
#13 0x3ff80172ae8 in __exc_lookup_function_entry(0x0, 0x0, 0x3ffc018a010, 0x0,
0x3ff800d1540, 0x1401e5020) DebugInformationStrippedFromFile304:???
#14 0x3ff807b2004 in exc_lookup_function_entry(0x3ffc018a010, 0x0,
0x3ff800d1540, 0x1401e5020, 0x3ff807b2bec, 0x1400249e0)
DebugInformationStrippedFromFile0:???
#15 0x3ff807b2be8 in exc_dispatch_exception(0x0, 0x12009407c, 0x140122910,
0x140020000, 0x0, 0x0) DebugInformationStrippedFromFile1:???
#16 0x3ff807b38c4 in exc_raise_signal_exception(0x80, 0x0, 0x3ff8019241c, 0x1,
0x1, 0x140048af0) DebugInformationStrippedFromFile3:???
#17 0x3ff800d3b28 in __sigtramp(0x3ff8019241c, 0x1, 0x1, 0x140048af0,
0x140122ef8, 0x0) DebugInformationStrippedFromFile105:???
#18 0x3ff8019241c in UnknownProcedure22FromFile22(0x3ffc0080c50, 0x1,
0x14017cc00, 0x2f, 0x140024f20, 0x3ff8057c824)
DebugInformationStrippedFromFile22:???
#19 0x3ff80192f40 in UnknownProcedure17FromFile22(0x3ff801943c8, 0x1, 0x10,
0x3ffc0080e18, 0x72, 0x140024f20) DebugInformationStrippedFromFile22:???
#20 0x3ff801943c4 in /usr/shlib/libc.so
#21 0x3ff80194604 in /usr/shlib/libc.so
#22 0x3ff800d1958 in malloc(0x3ff800d195c, 0x0, 0x12008e3bc, 0x140024f20,
0x1401232a0, 0x14017cc00) DebugInformationStrippedFromFile22:???
#23 0x12008e3b8 in lxldalc() /vobs/nlsrtl3/src/lxi/lxldalc.c:54
Note that frame #23 has a call to malloc(). In frame #16 a synchronous signal
(such as a SEGV) is taken, which results in an exception being raised. Notice
in frame #6 that malloc() is called again as part of handling the exception,
which leads to the thread blocking. This is because the previous call to
malloc() caused it to lock its internal mutex for synchronization purposes.
This recursive call to malloc() is the problem. The user might be able to fix
the cause of the original SEGV if its accompanying exception is fully processed.
This patch allows the SEGV to be processed fully and correctly.
PROBLEM: (QAR 49804) (Patch ID: OSF410-400195)
********
Threaded applications may see hangs when trying to use fork(). A prior patch
fixed only a part of the problem.
The user sees the child process with a stack trace from Ladebug like this:
(ladebug) t
>0 0x3ff8057ca24 in /usr/shlib/libpthread.so
#1 0x3ff8057a450 in hstTransferContext(0x1, 0x140ae11f0, 0x3ffc018a010, 0x4,
0x3ffc0188a50, 0x3ffc01852c0) DebugInformationStrippedFromFile109:???
#2 0x3ff8056429c in dspDispatch(0x140051810, 0x1408641a8, 0x140864030, 0x0,
0x3ffc01851d0, 0x0) DebugInformationStrippedFromFile89:???
#3 0x3ff80567b98 in pthread_mutex_block(0x3ffc008ae28, 0x3ffc01876f8,
0x140621c00, 0x0, 0x3ffc01851d0, 0x0)
DebugInformationStrippedFromFile95:???
#4 0x3ff8057c820 in __pthread_mutex_lock(0x3ffc01876f8, 0x140621c00, 0x0,
0x3ffc01851d0, 0x0, 0x3ff8057984c)
DebugInformationStrippedFromFile111:???
#5 0x3ff80579848 in vmReinit(0x0, 0x3ff8057984c, 0x3ff80578e70, 0x0,
0x3ffc0080c50, 0x3ffc0081de0) DebugInformationStrippedFromFile108:???
#6 0x3ff80578e6c in /usr/shlib/libpthread.so
#7 0x3ff800d36e0 in __fork(0x120040830, 0x0, 0x140684bc0, 0x0, 0x1f, 0x0)
DebugInformationStrippedFromFile335:???
#8 0x12004082c in forkTranslator(inoutp_handle=0x140a9b8f8,
inoutp_operation_data=0x140763a00) dp_xlatr.c:3610
#9 0x12004040c in forkedTranslatorInterface(inoutp_handle=0x140a9b8f8,
in_operation_code=eConnect, inoutp_operation_data=0 x140763a00,
getr=0x0, putr=0x0, arg_p=0x0) dp_xlatr.c:3471
#10 0x12003e0c0 in dp_callTranslator(inoutp_handle=0x140a9b8f8,
in_operation_code=eConnect, inoutp_operation_data=0x140763a00,
inp_scb=0x140621c00, getr=0x0, putr=0x0, arg_p=0x0) dp_xlatr.c:1894
#11 0x12003d878 in dp_connectToTranslator(inp_scb=0x140621c00) dp_xlatr.c:1468
#12 0x1200649d0 in dp_pjaConnectToTranslator(inp_Scb=0x140621c00,
in_docPdlOid=1252, inoutp_printerPdlOid=0x140a9b988,
outp_needsTranslator=0x140a9b9a8) pja_doc.c:726
#13 0x12005b4a0 in startDocument_1(getr=0x12004bb0c, putr=0x12005dc40,
inp_fib=0x140a53c00) dp_pja.c:1508
#14 0x12004b234 in fiberRoutineJacket(inp_fib=0x140a53c00) dp_jbstp.c:1276
#15 0x12004b430 in fibThreadJacket(inp_fib=0x140a53c00) dp_jbstp.c:1331
#16 0x3ff80574b94 in thdBase(0x0, 0x3ffc0183430, 0x3ffc018a010, 0x1, 0x45586732,
0x3) DebugInformationStrippedFromFile101:???
Note that frame #7 above shows the call to fork(). The DECthreads library is
then entered with an attempt to lock a DECthreads internal mutex in frame #4.
This attempt causes the thread to block while waiting for another thread to
unlock the mutex. But because this is the only thread left in the child process
after the parent forks, this mutex will never be unlocked. The parent failed
to lock this mutex during the fork operation and so is incorrectly left in the
locked state in the child.
PROBLEM: (HPXQ10P01, HPXQB7HGW) (Patch ID: OSF410-400195)
********
For a threaded application written on a system prior to Digital UNIX V4.0 that
uses C and the DECthreads exception handling macros, or a threaded application
written with C++ on any version of Digital UNIX that also uses the DECthreads
exception handling macros, various synchronous signals might not be caught
by the application's exception handlers.
The DECthreads library is not properly raising synchronous signals as exceptions
that the application can catch.
PROBLEM: (QAR 51033) (Patch ID: OSF410-400195)
********
Threaded applications using pthread_kill() may experience a stack corruption.
This is rather difficult to detect.
PROBLEM: (QAR 51214) (Patch ID: OSF410-400195)
********
A threaded application can experience a DECthreads bug check with a message
that mentions the routine nxm_resched(). This application might also hang
with one of its threads having a stack trace containing a frame for a call
to the routine errBugcheck().
The formatted DECthreads bugcheck message might look like this:
DECthreads bugcheck (version V3.13-435), terminating execution.
vpUpcallThreadUnblocked: (os/kern) invalid argument (4) nxm_resched(20,0)
For a hung application in this scenario, a typical stack trace from Ladebug
might look like this:
(ladebug) t
>0 0x3ff80568b20 in pthread_mutex_unblock(0x3ffc00802a0, 0x0, 0x0,
0x3ffc0181580, 0x3ffc018a968, 0x0)
DebugInformationStrippedFromFile95:???
#1 0x3ff8057c1f8 in __pthread_mutex_unlock(0x0, 0x0, 0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618)
DebugInformationStrippedFromFile111:???
#2 0x3ff80575614 in UnknownProcedure5FromFile103(0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618, 0x3ff800d6904, 0x3ffc0187760)
DebugInformationStrippedFromFile103:???
#3 0x3ff800d6900 in fflush(0x3ff800d6904, 0x3ffc0187760, 0x3ff80565cfc,
0x3ffc018a968, 0x3ffc0187768, 0x0)
DebugInformationStrippedFromFile84:???
#4 0x3ff80565cf8 in errBugcheck(0x3ffc0185da0, 0x3ffc01700d8, 0x4, 0x26e,
0x0, 0x3ffc018aae0) DebugInformationStrippedFromFile90:???
#5 0x3ff8057bcd4 in UnknownProcedure26FromFile109(0x0, 0x14014b438,
0x3ffc018a968, 0x3ffc018a968, 0x1, 0x14014b438)
DebugInformationStrippedFromFile109:???
#6 0x3ff8057b0d8 in UnknownProcedure20FromFile109(0x500, 0x1f4, 0x1,
0xffffffffdfbbee16, 0x1f4, 0x14014b478)
DebugInformationStrippedFromFile109:???
#7 0x3ff80535200 in msg_receive(0x0, 0x3ffc018abf0, 0x0, 0x0, 0x3ffc018a968,
0x0) DebugInformationStrippedFromFile6:???
#8 0x3fefffffffd in ???
the kernel to call into user space to perform an operation). Note that
frame #4 above contains the DECthreads bugcheck call, but also note that it
eventually calls pthread_mutex_unblock() (frame #0 above). This call to
pthread_mutex_unblock() hangs because it is incorrect to call pthread_mutex_
unblock() within the context of an upcall.
PROBLEM: (QAR 51214) (Patch ID: OSF410-400210)
********
A threaded application might hang while attempting to issue a message from the
DECthreads bugcheck. A stack trace of a thread in this state will contain
frames for calls to the routines errBugcheck(), fflush() and pthread_mutex_
unblock().
(ladebug) t
>0 0x3ff80568b20 in pthread_mutex_unblock(0x3ffc00802a0, 0x0, 0x0,
0x3ffc0181580, 0x3ffc018a968, 0x0) DebugInformationStrippedFromFile95:???
#1 0x3ff8057c1f8 in __pthread_mutex_unlock(0x0, 0x0, 0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618) DebugInformationStrippedFromFile111:???
#2 0x3ff80575614 in UnknownProcedure5FromFile103(0x3ffc0181580,
0x3ffc018a968, 0x0, 0x3ff80575618, 0x3ff800d6904, 0x3ffc0187760)
DebugInformationStrippedFromFile103:???
#3 0x3ff800d6900 in fflush(0x3ff800d6904, 0x3ffc0187760,
0x3ff80565cfc, 0x3ffc018a968, 0x3ffc0187768, 0x0)
DebugInformationStrippedFromFile84:???
#4 0x3ff80565cf8 in errBugcheck(0x3ffc0185da0, 0x3ffc01700d8, 0x4,
0x26e, 0x0, 0x3ffc018aae0) DebugInformationStrippedFromFile90:???
#5 0x3ff8057bcd4 in UnknownProcedure26FromFile109(0x0, 0x14014b438,
0x3ffc018a968, 0x3ffc018a968, 0x1, 0x14014b438)
DebugInformationStrippedFromFile109:???
#6 0x3ff8057b0d8 in UnknownProcedure20FromFile109(0x500, 0x1f4, 0x1,
0xffffffffdfbbee16, 0x1f4, 0x14014b478)
DebugInformationStrippedFromFile109:???
#7 0x3ff80535200 in msg_receive(0x0, 0x3ffc018abf0, 0x0, 0x0,
0x3ffc018a968, 0x0) DebugInformationStrippedFromFile6:???
#8 0x3fefffffffd in ???
Note that frame #8 above is not readable. This is to be expected because the
problem is occuring in an "upcall" frame. (This "upcall" frame is used by
kernel to call into user space to perform an operation). Note that frame #4
above contains the DECthreads bugcheck call, but also note that it eventually
calls pthread_mutex_unblock() (frame #0 above). This call to pthread_mutex_
unblock() hangs because it is incorrect to call pthread_mutex_unblock()
within the context of an upcall.
PROBLEM: (EVMS-GRYPHON-FT QAR 00237) (Patch ID: OSF410-400210)
********
This patch fixes a situation where threads would hang in the pthread_once()
or cma_once() routines.
The DECthreads routines pthread_once() and cma_once() use the DECthreads global
mutex. The first thread to call pthread_once() or cma_once() would lock this
mutex and subsequent threads calling pthread_once() or cma_once() would hang.
This problem was fixed on OpenVMS and the patch has been ported to Digital
UNIX 4.0.
PROBLEM: (IPMT HPAQ11MMTG) (Patch ID: OSF410-400210)
********
For the signals traditionally representing synchronous errors in a program
(such as SIGPIPE, SIGSYS, SIGSEGV, SIGBUS, SIGFPE, SIGEMT, SIGIOT, SIGILL),
DECthreads catches the signal and converts it into an equivalent exception.
Digital UNIX 4.0 attempts to print an abort message for an unhandled exception.
If an unhandled exception occurs in a libc library I/O routine, the thread
causing the exception may hang in abort() because of a deadlock over an I/O
mutex.
FILE(s):
/usr/shlib/libexc.so subset OSFBASE410
CHECKSUM: 55605 48
/usr/shlib/libpthread.so subset OSFBASE410
CHECKSUM: 30836 416
/usr/shlib/libpthreads.so subset OSFBASE410
CHECKSUM: 34736 120
/usr/shlib/libpthreaddebug.so subset OSFBASE410
CHECKSUM: 65353 144
--------------------------------------------------
|