A few people (including myself) encountered a problem running Hudson on Solaris 10. The symptom of the problem is that a build hangs, and when you obtain a thread dump, you see that the thread is hanging in UNIXProcess.forkAndExec
:
"Executor #2 for maddie" Id=81 RUNNABLE (in native) at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.(UNIXProcess.java:53) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) at hudson.Proc$LocalProc.(Proc.java:104) at hudson.Proc$LocalProc.(Proc.java:82) at hudson.Proc$LocalProc.(Proc.java:74) at hudson.Util.createSymlink(Util.java:725) at hudson.model.AbstractBuild$AbstractRunner.createLastSuccessfulLink(AbstractBuild.java:247) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:241) at hudson.model.Run.run(Run.java:778) at hudson.model.Build.run(Build.java:85) at hudson.model.ResourceController.execute(ResourceController.java:70) at hudson.model.Executor.run(Executor.java:85)
When this happens, you can see the forked process from the ps
command, as below:
hudson@rator ~ $ ps -ef | grep java noaccess 832 1 0 Jun 03 ? 285:37 /usr/java/bin/java -server -Xmx128m -XX:+BackgroundCompilation -XX:PermSize=32m mode 24104 1 0 Jul 01 pts/2 0:00 /usr/dist/share/java,v1.5.0_14/5.x-i86pc/bin/java -Djdk.home=/usr/dist/share/ja hudson 20906 536 0 Jul 02 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 3907 536 0 Jul 02 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 536 7 0 Jul 01 ? 393:55 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 7398 536 0 Jul 02 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hg 21255 21086 0 Jun 18 ? 6:48 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=28801 -ja hudson 671 536 0 Jul 01 ? 0:57 /files/hudson/tools/java1.6/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,serv hudson 15109 9416 0 16:39:50 pts/9 0:00 grep java
When you kill one of the forked process (such as 20906), it will bring down the entire Hudson server.
If you run pstack
on one of the forked child process, you see the output like this, which indicates that the hang is around PKCS code.
hudson@rator ~ $ ps -ef | grep java noaccess 832 1 0 Jun 03 ? 300:56 /usr/java/bin/java -server -Xmx128m -XX:+BackgroundCompilation -XX:PermSize=32m hudson 6101 15216 0 Jul 03 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 15216 7 0 Jul 03 ? 557:02 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx mode 24104 1 0 Jul 01 ? 0:00 /usr/dist/share/java,v1.5.0_14/5.x-i86pc/bin/java -Djdk.home=/usr/dist/share/ja hudson 10467 15216 0 Jul 07 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 2565 15216 0 02:16:33 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 27123 15216 0 Jul 03 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hg 21255 21086 0 Jun 18 ? 8:57 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=28801 -ja hudson 15707 15216 0 Jul 03 ? 3:04 /files/hudson/tools/java1.6/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,serv hudson 10936 15216 0 Jul 05 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 16947 15216 0 Jul 05 ? 0:00 java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address=5589 -Xmx hudson 11346 10308 0 16:58:05 pts/15 0:00 grep java hudson@rator ~ $ pstack 16947 16947: java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address= feeaf52b lwp_park (0, 0, 0) feea7ffe mutex_lock_internal (bfbad04, 0, 1) + 1bc feea843d mutex_lock_impl (bfbad04, 0) + 12d feea8492 mutex_lock (bfbad04) + 10 91edd1b9 soft_delete_session (bfbad00, 1, 1) + a9 91edce89 soft_delete_all_sessions (1, 91f0d4bc, 91f24378, 91f24000, 8843160c, feea8492) + 49 91ed79e0 finalize_common (1, 0) + 6c 91ed7b67 softtoken_fini (feffa7d8, 91f60958, 91f0d4bc, 88431674, fefd314b, 8de50538) + 37 91f0d4e1 _fini (8de50538, 92ff0de8, feffa7d8, 8de50550, 8de50538, 92ff0de8) + 25 fefd314b call_fini (feffa2dc, 8de50538) + f3 fefdc08a remove_hdl (91f60cb0, 92ff0de8, 0) + 7c2 fefd7154 dlclose_core (91f60cb0, 92ff0de8, feffa2dc) + a0 fefd71a1 dlclose_intn (91f60cb0, 92ff0de8) + 25 fefd720d dlclose_check (91f60cb0, 92ff0de8) + 2d fefd7273 dlclose (91f60cb0) + 3b 91f838f0 pkcs11_slottable_delete (91f966d8, 91f96000, 0, fef496c0, fede0500, 884317d4) + b4 91f7fa3c finalize_common (0, fede0500, 91f96000, 884317e0, 884317e0, 91f7f78f) + dc 91f7fa89 pkcs11_fini (fede0080, 884317fc, fee47282, 8d137200, fef45000, 88431820) + 41 91f7f78f pkcs11_fork_child (8d137200, fef45000, 88431820, 8d137200, 88431820, 88431820) + 27 fee47282 _postfork_child_handler (b, 93eac24, fef45000, 99a72e80, fef49720, fef49700) + 32 feea167d forkx (0) + 11d feea172e fork (c45d800, f42d369c, f42d36a0, c45d800, 999c2878, 99a72cd8) + 1a fe287edf Java_java_lang_UNIXProcess_forkAndExec (c45d8ec, 88431940, 8843193c, 88431938, 1, 88431930) + 367 fb20ebd2 ???????? (0, fb20b399, 99a732e0, 99a732d0, 99a732c0, 0) fb202de9 ???????? (0, 0, c, 99a72f48, 1, 99a72ee0) fbee9bf0 ???????? () hudson@rator ~ $ pstack 27123 27123: java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address= feeaf52b lwp_park (0, 0, 0) feea7ffe mutex_lock_internal (97e6094, 0, 1) + 1bc feea843d mutex_lock_impl (97e6094, 0) + 12d feea8492 mutex_lock (97e6094) + 10 91edd1b9 soft_delete_session (97e6090, 1, 1) + a9 91edce89 soft_delete_all_sessions (1, 91f0d4bc, 91f24378, 91f24000, 8c19afbc, feea8492) + 49 91ed79e0 finalize_common (1, 0) + 6c 91ed7b67 softtoken_fini (feffa7d8, 91f60958, 91f0d4bc, 8c19b024, fefd314b, 8de50538) + 37 91f0d4e1 _fini (8de50538, 92ff0de8, feffa7d8, 8c19b024, fefd314b, 8de50538) + 25 fefd314b call_fini () + f3 hudson@rator ~ $ pstack 2565 10467 6101 2565: java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address= feeaf52b lwp_park (0, 0, 0) feea7ffe mutex_lock_internal (cb6cb9c, 0, 1) + 1bc feea843d mutex_lock_impl (cb6cb9c, 0) + 12d feea8492 mutex_lock (cb6cb9c) + 10 91edd1b9 soft_delete_session (cb6cb98, 1, 1) + a9 91edce89 soft_delete_all_sessions (1, 91f0d4bc, 91f24378, 91f24000, 902334cc, feea8492) + 49 91ed79e0 finalize_common (1, 0) + 6c 91ed7b67 softtoken_fini (feffa7d8, 91f60958, 91f0d4bc, 90233534, fefd314b, 8de50538) + 37 91f0d4e1 _fini (8de50538, 92ff0de8, feffa7d8, 8de50550, 8de50538, 92ff0de8) + 25 fefd314b call_fini (feffa2dc, 8de50538) + f3 fefdc08a remove_hdl (91f60cb0, 92ff0de8, 0) + 7c2 fefd7154 dlclose_core (91f60cb0, 92ff0de8, feffa2dc) + a0 fefd71a1 dlclose_intn (91f60cb0, 92ff0de8) + 25 fefd720d dlclose_check (91f60cb0, 92ff0de8) + 2d fefd7273 dlclose (91f60cb0) + 3b 91f838f0 pkcs11_slottable_delete (91f966d8, 91f96000, 0, fef496c0, fede0500, 90233694) + b4 91f7fa3c finalize_common (0, fede0500, 91f96000, 902336a0, 902336a0, 91f7f78f) + dc 91f7fa89 pkcs11_fini (fede0080, 902336bc, fee47282, 9134b200, fef45000, 902336e0) + 41 91f7f78f pkcs11_fork_child (9134b200, fef45000, 902336e0, 9134b200, 902336e0, 902336e0) + 27 fee47282 _postfork_child_handler (0, b1cde73, fef45000, f30bcaa3, fef49720, fef49700) + 32 feea167d forkx (0) + 11d feea172e fork (9133c00, f42d3664, f42d3668, 9133c00, a200aa38, 90233730) + 1a fe287edf Java_java_lang_UNIXProcess_forkAndExec (9133cec, 90233800, 902337fc, 902337f8, 2, 902337f0) + 367 fb20ebd2 ???????? (0, fb20b399, 93027718, 93027708, 930276f8, 1) fb202de9 ???????? (1, 93027660, 1, 93027458, 2, 930273f0) fc91de98 ???????? () 10467: java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address= feeaf52b lwp_park (0, 0, 0) feea7ffe mutex_lock_internal (d7ed02c, 0, 1) + 1bc feea843d mutex_lock_impl (d7ed02c, 0) + 12d feea8492 mutex_lock (d7ed02c) + 10 91edd1b9 soft_delete_session (d7ed028, 1, 1) + a9 91edce89 soft_delete_all_sessions (1, 91f0d4bc, 91f24378, 91f24000, 9168b8fc, feea8492) + 49 91ed79e0 finalize_common (1, 0) + 6c 91ed7b67 softtoken_fini (feffa7d8, 91f60958, 91f0d4bc, 9168b964, fefd314b, 8de50538) + 37 91f0d4e1 _fini (8de50538, 92ff0de8, feffa7d8, 8de50550, 8de50538, 92ff0de8) + 25 fefd314b call_fini (feffa2dc, 8de50538) + f3 fefdc08a remove_hdl (91f60cb0, 92ff0de8, 0) + 7c2 fefd7154 dlclose_core (91f60cb0, 92ff0de8, feffa2dc) + a0 fefd71a1 dlclose_intn (91f60cb0, 92ff0de8) + 25 fefd720d dlclose_check (91f60cb0, 92ff0de8) + 2d fefd7273 dlclose (91f60cb0) + 3b 91f838f0 pkcs11_slottable_delete (91f966d8, 91f96000, 0, fef496c0, fede0500, 9168bac4) + b4 91f7fa3c finalize_common (0, fede0500, 91f96000, 9168bad0, 9168bad0, 91f7f78f) + dc 91f7fa89 pkcs11_fini (fede0080, 9168baec, fee47282, fe45ba00, fef45000, 9168bb10) + 41 91f7f78f pkcs11_fork_child (fe45ba00, fef45000, 9168bb10, fe45ba00, 9168bb10, 9168bb10) + 27 fee47282 _postfork_child_handler (1, d6ba709, fef45000, 2, fef49720, fef49700) + 32 feea167d forkx (0) + 11d feea172e fork (8da3c00, f42d3664, f42d3668, 8da3c00, a200c160, 9168bb60) + 1a fe287edf Java_java_lang_UNIXProcess_forkAndExec (8da3cec, 9168bc2c, 9168bc28, 9168bc24, 2, 0) + 367 fb20ebd2 ???????? (0, fb20b399, 9658b978, 9658b968, 9658b958, 0) fb202de9 ???????? (0, 0, 0, 0, 2, 9658b7c0) fb202f0d ???????? (9658b690, 2, 0, 9658b898, c9, 9658b7c0) fb202da7 ???????? (9658aba8, 0, 0, f48c9b20, 9658b4b0, 9658b468) fb202da7 ???????? (0, 0, 9658b410, 9dab8bb0, 9168bd50, f30f4210) fb202da7 ???????? (9658b410, 9dab8bb0, 8da3cec, 0, 9658b410, 9658b428) fd058d70 ???????? (0, 0, 8da38d8, 0, 8c3e048, 0) 00000000 ???????? (fe888b40, fe768ef0, fe7637d4, fe888b34, fe888a8c, fe888b1c) 00000000 ???????? (fffffe18, 3badd9be, f468fe16, e78b7254, fe1dbbcc, 0) 00000000 ???????? () 6101: java -Xdebug -Xrunjdwp:transport=dt_socket,suspend=n,server=y,address= feeaf52b lwp_park (0, 0, 0) feea7ffe mutex_lock_internal (8f809a4, 0, 1) + 1bc feea843d mutex_lock_impl (8f809a4, 0) + 12d feea8492 mutex_lock (8f809a4) + 10 91edd1b9 soft_delete_session (8f809a0, 1, 1) + a9 91edce89 soft_delete_all_sessions (1, 91f0d4bc, 91f24378, 91f24000, 8ce1930c, feea8492) + 49 91ed79e0 finalize_common (1, 0) + 6c 91ed7b67 softtoken_fini (feffa7d8, 91f60958, 91f0d4bc, 8ce19374, fefd314b, 8de50538) + 37 91f0d4e1 _fini (8de50538, 92ff0de8, feffa7d8, 8de50550, 8de50538, 92ff0de8) + 25 fefd314b call_fini (feffa2dc, 8de50538) + f3 fefdc08a remove_hdl (91f60cb0, 92ff0de8, 0) + 7c2 fefd7154 dlclose_core (91f60cb0, 92ff0de8, feffa2dc) + a0 fefd71a1 dlclose_intn (91f60cb0, 92ff0de8) + 25 fefd720d dlclose_check (91f60cb0, 92ff0de8) + 2d fefd7273 dlclose (91f60cb0) + 3b 91f838f0 pkcs11_slottable_delete (91f966d8, 91f96000, 0, fef496c0, fede0500, 8ce194d4) + b4 91f7fa3c finalize_common (0, fede0500, 91f96000, 8ce194e0, 8ce194e0, 91f7f78f) + dc 91f7fa89 pkcs11_fini (fede0080, 8ce194fc, fee47282, 8d111a00, fef45000, 8ce19520) + 41 91f7f78f pkcs11_fork_child (8d111a00, fef45000, 8ce19520, 8d111a00, 8ce19520, 8ce19520) + 27 fee47282 _postfork_child_handler (0, c40195b, fef45000, 93018488, fef49720, fef49700) + 32 feea167d forkx (0) + 11d feea172e fork (a400400, f42d369c, f42d36a0, a400400, 9b880310, 8ce1958c) + 1a fe287edf Java_java_lang_UNIXProcess_forkAndExec (a4004ec, 8ce19640, 8ce1963c, 8ce19638, 2, 8ce19630) + 367 fb20ebd2 ???????? (0, fb20b399, 930191f8, 930191e8, 930191d8, 1) fb202de9 ???????? (1, 93019188, 1, 93018fc8, 2, 93018f58) fbee9bf0 ???????? ()
The source code for these code is available on OpenSolaris site: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/pkcs11/pkcs11_softtoken/common/softGeneral.c#finalize_common
I found a posting on forum that looks very related. More hits in here
Various suggestions
Apply patch 118918? There's an on-going escalation?
A workaround? Apply patch 127111 -> this was obsolted by 127127.
turns out this was reported in Hudson list some time ago and in JENKINS-2296.
Fix
I attempted to apply some of the mentioned patches, but they failed to pkgadd, citing version inconsistencies.
I then applied the suggested workaround in jre/lib/security/java.security file and restart Hudson. Since then I have no longer seen a problem.