Conference smurf::ase

1953.0. "System crashes when booting with ASE enabled" by NETRIX::"mcdonald@decatl.alf.dec.com" (John McDonald) Tue Mar 18 1997 10:35

I have a customer with an interesting problem. He's running an ase
configuration with 3 machines - 1 2000, 1 4100 and 1 2100, running
Digital UNIX 3.2x and ASE 1.3. Up until last week, everything was running
fine - then the 2100 started having problems, so he tried to restart ase
(/sbin/init.d/asemember restart) and the system crashed. No crash files
were created, and the system wasn't able to reboot. On a hunch, he
rebooted with the generic kernel, went into the startup directory
and renamed the aseam and asemember startup scripts and rebooted with
the regualr kernel. The system came up fine without ase starting up.
The following are some excerpts from the log files:

kern.log
--------
Mar 17 08:42:16 cisprod vmunix: Aborted Accept CCB and not disabling the LUN
Mar 17 08:42:16 cisprod last message repeated 5 times
Mar 17 08:42:32 cisprod vmunix: AM contacted from host at bus 2 target 4 lun 7
Mar 17 08:42:33 cisprod vmunix: AM contacted from host at bus 2 target 6 lun 7
Mar 17 08:42:34 cisprod vmunix: AM found a host at bus 2 target 4, lun 7
Mar 17 08:42:34 cisprod vmunix: AM found a host at bus 2 target 6, lun 7
Mar 17 08:42:34 cisprod vmunix: AM found a host at bus 3 target 4, lun 7
Mar 17 08:42:39 cisprod vmunix: AM found a host at bus 3 target 6, lun 7
Mar 17 08:42:40 cisprod vmunix: Aborted Accept CCB and not disabling the LUN
Mar 17 08:42:40 cisprod last message repeated 5 times
Mar 17 08:42:45 cisprod vmunix: AM found a host at bus 2 target 4, lun 7
Mar 17 08:42:45 cisprod vmunix: AM found a host at bus 2 target 6, lun 7
Mar 17 08:42:45 cisprod vmunix: AM found a host at bus 3 target 4, lun 7
Mar 17 08:42:50 cisprod vmunix: AM found a host at bus 3 target 6, lun 7

daemon.log
----------
Mar 17 05:38:09 cisprod xntpd[537]: hourly check: drift -0.05487466 compliance
-0.00007592
Mar 17 05:38:10 cisprod DECsafe: cisprod AseMgr Error: can't send to null
director channel
Mar 17 05:38:50 cisprod last message repeated 4 times
Mar 17 05:40:50 cisprod last message repeated 12 times
Mar 17 05:50:50 cisprod last message repeated 60 times
Mar 17 06:00:50 cisprod last message repeated 60 times
Mar 17 06:10:50 cisprod last message repeated 60 times
Mar 17 06:20:50 cisprod last message repeated 60 times
Mar 17 06:30:50 cisprod last message repeated 60 times
Mar 17 06:38:00 cisprod last message repeated 43 times
Mar 17 06:38:09 cisprod xntpd[537]: hourly check: drift -0.05572668 compliance
-0.00025159
Mar 17 06:38:10 cisprod DECsafe: cisprod AseMgr Error: can't send to null
director channel
Mar 17 06:38:50 cisprod last message repeated 4 times
Mar 17 06:40:50 cisprod last message repeated 12 times
Mar 17 06:50:50 cisprod last message repeated 60 times
Mar 17 07:00:50 cisprod last message repeated 60 times
 
Mar 17 07:10:50 cisprod last message repeated 60 times
Mar 17 07:20:50 cisprod last message repeated 60 times
Mar 17 07:30:50 cisprod last message repeated 60 times
Mar 17 07:38:00 cisprod last message repeated 43 times
Mar 17 07:38:09 cisprod xntpd[537]: hourly check: drift -0.05614174 compliance
-0.00029394
Mar 17 07:38:10 cisprod DECsafe: cisprod AseMgr Error: can't send to null
director channel
Mar 17 07:38:50 cisprod last message repeated 4 times
Mar 17 07:40:50 cisprod last message repeated 12 times
Mar 17 07:50:50 cisprod last message repeated 60 times
Mar 17 08:00:50 cisprod last message repeated 60 times
Mar 17 08:10:50 cisprod last message repeated 60 times
Mar 17 08:20:50 cisprod last message repeated 60 times
Mar 17 08:30:50 cisprod last message repeated 60 times
Mar 17 08:38:00 cisprod last message repeated 43 times
Mar 17 08:38:09 cisprod xntpd[537]: hourly check: drift -0.05997921 compliance
-0.00115474
Mar 17 08:38:10 cisprod DECsafe: cisprod AseMgr Error: can't send to null
director channel
Mar 17 08:38:50 cisprod last message repeated 4 times
Mar 17 08:40:50 cisprod last message repeated 12 times
Mar 17 08:42:00 cisprod last message repeated 7 times
Mar 17 08:42:05 cisprod DECsafe: local HSM Notice: Able to ping lazarus over
the network
Mar 17 08:42:05 cisprod DECsafe: local HSM Notice: Able to ping develop over
the network
Mar 17 08:42:05 cisprod DECsafe: local HSM Notice: Able to ping develop over
the SCSI bus
Mar 17 08:42:06 cisprod DECsafe: local HSM ***ALERT:
HSM_PATH_STATUS:204.69.16.251:UP
Mar 17 08:42:06 cisprod DECsafe: local HSM Notice: member develop is UP
Mar 17 08:42:06 cisprod DECsafe: local HSM Notice: Able to ping lazarus over
the SCSI bus
Mar 17 08:42:06 cisprod DECsafe: local HSM ***ALERT:
HSM_PATH_STATUS:204.69.16.244:UP
Mar 17 08:42:06 cisprod DECsafe: local HSM Notice: member lazarus is UP
Mar 17 08:42:06 cisprod DECsafe: local HSM Notice: Network interface fta0
204.69.16.252 UP
Mar 17 08:42:07 cisprod DECsafe: local HSM ***ALERT:
HSM_NI_STATUS:204.69.16.252:UP
Mar 17 08:42:07 cisprod DECsafe: local Simulator Notice: snd: exiting...
Mar 17 08:42:07 cisprod DECsafe: develop Director Notice: connected to by
shutdown program
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u01: not currently mounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u01 already unmounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u02: not currently mounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u02 already unmounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u03: not currently mounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u03 already unmounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u04: not currently mounted
Mar 17 08:42:11 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u04 already unmounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u05: not currently mounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u05 already unmounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u06: not currently mounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp/u06 already unmounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp: not currently mounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /fisp already unmounted
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rze26c
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rzd26c
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:12 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rzc26c
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rzf25c
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rze25c
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rzd25c
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unregister device
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Error: can't unreserve device
/dev/rzc25c
Mar 17 08:42:13 cisprod DECsafe: cisprod Agent Notice: stopAll: can't
unreserve convert's devices, stopping it anyway
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c1: not currently mounted
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c1 already unmounted
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c2: not currently mounted
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c2 already unmounted
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c3: not currently mounted
Mar 17 08:42:14 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c3 already unmounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c4: not currently mounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c4 already unmounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c5: not currently mounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c5 already unmounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c6: not currently mounted
Mar 17 08:42:15 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c6 already unmounted
Mar 17 08:42:16 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c7: not currently mounted
Mar 17 08:42:16 cisprod DECsafe: cisprod Agent Notice:
/var/ase/sbin/ase_mount_action: /c7 already unmounted
Mar 17 08:42:16 cisprod DECsafe: local AseUtility Notice: msgSvcSend: peer
hung up before we got reply
Mar 17 08:42:16 cisprod DECsafe: local AseUtility Warning: blocking send of
ASE_SHUTDOWN failed or channel hung up
Mar 17 08:42:16 cisprod DECsafe: local AseUtility Notice: msgSvcSend: peer
hung up before we got reply
Mar 17 08:42:16 cisprod DECsafe: local AseUtility Warning: blocking send of
ASE_SHUTDOWN failed or channel hung up
Mar 17 08:42:16 cisprod DECsafe: cisprod Agent Warning: aseagent exiting on
request...
Mar 17 08:42:16 cisprod DECsafe: local AseUtility Notice: ASE shutdown program
exiting...

messages
--------
Mar 17 11:20:48 cisprod vmunix: rz18 at scsi2 bus 2 target 2 lun 0 (DEC    
HSZ40            V25Z)
Mar 17 11:20:48 cisprod vmunix: rz19 at scsi2 bus 2 target 3 lun 0 (DEC    
HSZ40            V25Z)
Mar 17 11:20:48 cisprod vmunix: psiop1 at pci0 slot 7
Mar 17 11:20:48 cisprod vmunix: Loading SIOP: script 1013600, reg 81222100,
data  407539b8
Mar 17 11:20:48 cisprod vmunix: ccfg_simattach(): SIM Attach Failed,
sim_init() reported FAILURE - Retry.
Mar 17 11:20:48 cisprod vmunix: PSIOP: handler_add failed
Mar 17 11:20:48 cisprod vmunix: Loading SIOP: script 1013600, reg 81222100,
data  407539b8
Mar 17 11:20:48 cisprod vmunix: cam_logger: CAM_ERROR packet
Mar 17 11:20:48 cisprod vmunix: cam_logger: bus 4
Mar 17 11:20:48 cisprod vmunix: psiop_pci_script_cntl
Mar 17 11:20:48 cisprod vmunix: SCRIPTS did not start
Mar 17 11:20:48 cisprod vmunix: ccfg_simattach(): SIM Attach Failed,
sim_init() reported FAILURE - Retry.
Mar 17 11:20:49 cisprod vmunix: PSIOP: handler_add failed
Mar 17 11:20:49 cisprod vmunix: Loading SIOP: script 1013600, reg 81222100,
data 407539b8
Mar 17 11:20:49 cisprod vmunix: scsi4 at psiop1 slot 0
Mar 17 11:20:49 cisprod vmunix: tz32 at scsi4 bus 4 target 0 lun 0 (DEC    
TZ87

My initial idea is that it looks like a hardware problem, but I was wondering
if anyone has any other ideas?

John McDonald
Atlanta CSC
[Posted by WWW Notes gateway]

John,

You might try with informational logging turned on.  The only thing that is
obvious is that the daemons can not talk to each other correctly, so you
might want to concentrate on the network side of your configuration.

-- Greg
[Posted by WWW Notes gateway]