| Title: | DECmcc user notes file. Does not replace IPMT. |
| Notice: | Use IPMT for problems. Newsletter location in note 6187 |
| Moderator: | TAEC::BEROUD |
| Created: | Mon Aug 21 1989 |
| Last Modified: | Wed Jun 04 1997 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 6497 |
| Total number of notes: | 27359 |
The exporter is monitoring 15 nodes on a network. After about two
days, it stops exporting. Examination of logical link acitivy finds
all logical links used up by the exporter, all of whioh appear to be
hung. (see below)
NCP>SHOW KNOW LINK STATUS
Known Link Volatile Status as of 3-JUN-1991 17:08:54
Link Node PID Process Remote link State
24625 2.1 (CARROU) 000000A1 BATCH_101 30764 closed
24636 2.1 (CARROU) 000000A1 BATCH_101 18469 closed
16454 2.1 (CARROU) 000000A1 BATCH_101 49 closed
24656 2.1 (CARROU) 000000A1 BATCH_101 22567 closed
24675 2.1 (CARROU) 000000A1 BATCH_101 25644 closed
16496 2.1 (CARROU) 000000A1 BATCH_101 3108 closed
16536 2.1 (CARROU) 000000A1 BATCH_101 17431 closed
167 2.1 (CARROU) 000000A1 BATCH_101 24581 closed
197 2.1 (CARROU) 000000A1 BATCH_101 19475 closed
214 2.1 (CARROU) 000000A1 BATCH_101 27677 closed
16666 2.1 (CARROU) 000000A1 BATCH_101 1037 closed
24870 2.1 (CARROU) 000000A1 BATCH_101 19464 closed
8509 2.1 (CARROU) 000000A1 BATCH_101 16429 closed
24906 2.1 (CARROU) 000000A1 BATCH_101 15408 closed
8534 2.1 (CARROU) 000000A1 BATCH_101 6164 closed
346 2.1 (CARROU) 000000A1 BATCH_101 16410 closed
24923 2.1 (CARROU) 000000A1 BATCH_101 28694 closed
8540 2.1 (CARROU) 000000A1 BATCH_101 29758 closed
24956 2.1 (CARROU) 000000A1 BATCH_101 2062 closed
16772 2.1 (CARROU) 000000A1 BATCH_101 14382 closed
398 2.1 (CARROU) 000000A1 BATCH_101 22568 closed
438 2.1 (CARROU) 000000A1 BATCH_101 13333 closed
16425 31.1 (NYTRDG) 000000E8 _TWA9: 16516 run
8502 31.1 (NYTRDG) 000000DF _TWA8: 16404 run
16665 31.32 (IBIRTR) 000000A1 BATCH_101 55245 run
8334 31.105 (GFX105) 000000A1 BATCH_101 24852 run
157 31.105 (GFX105) 000000A1 BATCH_101 39 run
227 31.105 (GFX105) 000000A1 BATCH_101 51 run
320 31.105 (GFX105) 000000A1 BATCH_101 24789 run
16708 31.105 (GFX105) 000000A1 BATCH_101 344 run
24982 31.105 (GFX105) 000000A1 BATCH_101 25066 run
16390 31.660 (DEC660) 0000011E _TWA11: 16391 run
16391 31.660 (DEC660) 00000057 DNS$TA 16390 run
16585 52.999 (52BRTR) 000000A1 BATCH_101 393 run
BATCH_101, the exporter has stopped writing to its Rdb file.
Attempts to initiate any other DECnet activity, such as querying
DNS, result in DECnet errors.
Showing export status on any of the nodes that I have exporting
running shows the following:
SHOW EXPORTING NODE4 .dna_node.CARROU EXPORT TARGET
DKB700:[YANNIOS]MCC_ROUTER_PERF.RDB
Node4 DEC660_NS:.dna_node.CARROU
AT 3-JUN-1991 17:07:55
Exporting parameters are:
Exporting state = SUSPENDED,
State since = 2-JUN-1991 09:19:39.87,
Export period = 0 00:15:00.00,
Begin time = 31-MAY-1991 18:04:12.52,
End time = 25-MAY-2012 00:00:00.00,
Export target = "DKB700:[YANNIOS]MCC_ROUTER_PERF.RDB",
Request time = 31-MAY-1991 18:04:12.52, Requested by = "SYSTEM",
Time of last successful poll = " 2-JUN-1991 06:49:46.22",
Number of successful polls = 91,
Time of last failed poll = " 2-JUN-1991 09:19:39.87",
Last poll failure reason = "failed to call ETP",
Number of failed polls = 67,
Last export time = " 2-JUN-1991 09:19:39.87",
Time of last export failure = "NONE",
Last export failure reason = "N/A",
Number of export failures = 0,
Sequence name = "CARROU",
Initial sequence number = 0,
Current sequence number = 158
What does "Failed to call ETP" mean?
The exporter has also gradually consumed memeory resources, over a
two day period, it has attained a peak working set size of 33,000
pages (please see note 1069.4 for more detail on this problem)
Memory before killing exporter background
System Memory Resources on 3-JUN-1991 17:22:54.74
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (16.00Mb) 32768 1566 30650 552
Slot Usage (slots): Total Free Resident Swapped
Process Entry Slots 45 13 30 2
Balance Set Slots 40 12 28 0
Fixed-Size Pool Areas (packets): Total Free In Use Size
Small Packet (SRP) List 1170 250 920 112
I/O Request Packet (IRP) List 698 213 485 176
Large Packet (LRP) List 61 37 24 1648
Dynamic Memory Usage (bytes): Total Free In Use Largest
Nonpaged Dynamic Memory 1241600 767872 473728 720384
Paged Dynamic Memory 1000448 749616 250832 746368
Paging File Usage (pages): Free Reservable Total
DISK$SYSDSK:[SYS0.SYSEXE]SWAPFILE.SYS 9160 9160 10000
DISK$SYSDSK1:[SWAPFILES]SWAPFILE.SYS;1 8592 8592 8592
DISK$SYSDSK:[SYS0.SYSEXE]PAGEFILE.SYS 6506 1516 13600
DISK$SYSDSK1:[SWAPFILES]PAGEFILE.SYS;2 54327 6847 99992
memory freed after terminating
System Memory Resources on 3-JUN-1991 17:41:48.20
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (16.00Mb) 32768 9140 23214 414
Slot Usage (slots): Total Free Resident Swapped
Process Entry Slots 45 15 28 2
Balance Set Slots 40 14 26 0
Fixed-Size Pool Areas (packets): Total Free In Use Size
Small Packet (SRP) List 1170 568 602 112
I/O Request Packet (IRP) List 698 409 289 176
Large Packet (LRP) List 61 47 14 1648
Dynamic Memory Usage (bytes): Total Free In Use Largest
Nonpaged Dynamic Memory 1241600 802400 439200 720384
Paged Dynamic Memory 1000448 750288 250160 746368
Paging File Usage (pages): Free Reservable Total
DISK$SYSDSK:[SYS0.SYSEXE]SWAPFILE.SYS 9160 9160 10000
DISK$SYSDSK1:[SWAPFILES]SWAPFILE.SYS;1 8592 8592 8592
DISK$SYSDSK:[SYS0.SYSEXE]PAGEFILE.SYS 10641 5200 13600
DISK$SYSDSK1:[SWAPFILES]PAGEFILE.SYS;2 79323 38178 99992
Of the physical pages in use, 6552 pages are permanently allocated to VMS.
all hung logical links go away expept for the two active CTERM
sessions....
Please advise...thanks!
Regards,
Nick
| T.R | Title | User | Personal Name | Date | Lines |
|---|---|---|---|---|---|
| 1089.1 | Other observations... | SUBWAY::YANNIOS | Tue Jun 04 1991 10:47 | 21 | |
ALSO:
Attempts to DISCONNECT LINK n do not succeed...
Increasing nodes maximum logical links fixes inability to connect to
DNS...
After cancelling and restarting the batch job, the export background
job does not appear to be writing to the database file. Is this job
using Rdb journalling? Are there any specific steps that one must
take to recover from terminating the batch job or is this done
automatically?
If additional entities are added to be monitored after the job is
started, I get "Request queue full...check background process"
What does this mean?
Thanks again...
Nick
| |||||
| 1089.2 | Exporter not handling remote node resource probs? | SUBWAY::YANNIOS | Tue Jun 04 1991 13:20 | 5 | |
Attempts to do a NCP TELL CARROU SHOW EXEC CHAR ... failed with
"network partner exited" status
Nick
| |||||
| 1089.3 | Checking Rdb status results... | SUBWAY::YANNIOS | Tue Jun 04 1991 14:01 | 9 | |
More...
Status of the Rdb database file is viewed with "$ RMU/SHOW SYSTEM"
and "$ RMU /SHOW STATISTICS <db-name>". After I restart the
exporter batch job, it doesn't re-open the database and just "sits"
there....
Nick
| |||||
| 1089.4 | TOOK::SHMUYLOVICH | Tue Jun 04 1991 14:20 | 66 | ||
> What does "Failed to call ETP" mean?
It means that from Show Entity_To_Poll all Identifiers
Exporter does not get a response with desired data. If Identifiers
are not returned Exporter does not call to other partitions and
do not write data in the RDB.
In V1.2 it will be more details of this failure. For now you
can setup recording for the identifier partition (using Historian)
and use recorded information to analyze the returned condition( sorry
for inconvenience).
> After cancelling and restarting the batch job, the export background
> job does not appear to be writing to the database file.
Let's look at show_exporting
Exporting parameters are:
*--------------------> Exporting state = SUSPENDED,
*------------------------> State since = 2-JUN-1991 09:19:39.87,
*---------------------> Export period = 0 00:15:00.00,
Begin time = 31-MAY-1991 18:04:12.52,
End time = 25-MAY-2012 00:00:00.00,
Export target = "DKB700:[YANNIOS]MCC_ROUTER_PERF.RDB",
Request time = 31-MAY-1991 18:04:12.52, Requested by = "SYSTEM",
*--------> Time of last successful poll = " 2-JUN-1991 06:49:46.22",
Number of successful polls = 91,
*-----------> Time of last failed poll = " 2-JUN-1991 09:19:39.87",
Last poll failure reason = "failed to call ETP",
Number of failed polls = 6
Last export time = " 2-JUN-1991 09:19:39.87",
Time of last export failure = "NONE",
Last export failure reason = "N/A",
Number of export failures = 0,
Sequence name = "CARROU",
Initial sequence number = 0,
Current sequence number = 158
Exporting state is "SUSPENDED" so background process does not need to
write in RDB. We can see that this exporting was suspended at "State since"
time which is equal to "Time of last failed poll".
"Time of last successful poll", "Time of last failed poll" and "Export period"
show that there were 10 failed polls running. At the 10-th failed poll the
state is automatically suspended.
> If additional entities are added to be monitored after the job is
> started, I get "Request queue full...check background process"
> What does this mean?
This message means that background process was dead during entering
several "export" and/or "delete exporting" commands.
> The exporter has also gradually consumed memeory resources, over a
> two day period, it has attained a peak working set size of 33,000
> pages (please see note 1069.4 for more detail on this problem)
Please see 1069.5
Sam
| |||||
| 1089.5 | Restarting Exporter does not re-open db file? | SUBWAY::YANNIOS | Tue Jun 04 1991 17:42 | 22 | |
| 1089.6 | Other errors with Exporter noted... | SUBWAY::YANNIOS | Tue Jun 04 1991 17:47 | 61 | |
Examination of the various MCC_EXPORTER_BACKGROUND.LOG
file shows some interesting errors:
.
.
.
.
$ IF (SVRT1 .AND. SVRT2) THEN GOTO OKAY
$ OKAY:
$ WAIT 00:00:30
$!
$! end_of wait procedure
$!
$ BTS == "$SYS$SYSTEM:MCC_EXPORTER_FM_BG.EXE"
$ BTS "DKB700:[YANNIOS]MCC_ROUTER_PERF.RDB"
%SYSTEM-F-ROPRAND, reserved operand fault at PC=003C5899,
PSL=03C00000
although, this one was a while ago and I had since increased account
and system quotas and this cleared up.
In one of my earlier logs, the folowing resulted:
$ BTS "DKB700:[YANNIOS]MCC_PERF.DB"
%SYSTEM-F-ACCVIO, access violation, reason mask=01, virtual
address=18000061, PC
=80000010, PSL=03C00004
Improperly handled condition, image exit forced.
Signal arguments Stack contents
Number = 00000005 80127E40
Name = 0000000C 00000002
00000001 00985204
18000061 009851EC
80000010 00000004
03C00004 00985494
00000001
045CD7FF
00A56A04
05000001
Register dump
R0 = 03C00000 R1 = 18000061 R2 = 0000FFA3 R3 = 0098525C
R4 = 00000000 R5 = 000000CA R6 = 00986E94 R7 = 009868D4
R8 = 00000000 R9 = 00A13860 R10= 00986E9C R11= 009868AA
AP = 009851A0 FP = 00985160 SP = 009851DC PC = 80000010
PSL= 03C00004
SYSTEM job terminated at 31-MAY-1991 05:30:30.78
Accounting information:
Buffered I/O count: 158351 Peak working set size:
6657
Direct I/O count: 10991 Peak page file size:
23265
Page faults: 311261 Mounted volumes:
0
Charged CPU time: 0 00:47:45.00 Elapsed time: 0
10:20:31.22
| |||||
| 1089.7 | check exporting status | TOOK::SHMUYLOVICH | Tue Jun 04 1991 19:07 | 8 | |
Re: .5
Please check status of all your exportings. I think they are
"suspended". If this is true you need to resume them using
Resume Export command.
Sam
| |||||
| 1089.8 | which system quota | TOOK::SHMUYLOVICH | Tue Jun 04 1991 19:11 | 7 | |
re: .6
It would be very usefull if you can tell which system quotas you
increased.
Thanks, Sam
| |||||
| 1089.9 | VIRTUALPAGECNT | SUBWAY::YANNIOS | Tue Jun 04 1991 20:20 | 9 | |
VIRTUALPAGECNT
Was 20,000 Increased to 90,000
PAGEFILEQUOTA for SYSTEM was set at 90,000 in both cases but could
not be fully utilized because VIRTUALPAGECNT was too low.
Nick
| |||||
| 1089.10 | Can you clarify the Suspension rules? | NSSG::R_SPENCE | Nets don't fail me now... | Wed Jun 05 1991 13:44 | 7 |
Samuel, are you saying that if the number of failed polls ever gets to
10, the exporting will SUSPEND? Or, does it have to be 10 in a row for
the same entity? Or what? How can we change this since for some
entities it may be perfectly reasonable for 10 polls to be missed if
the entity was down for a weekend for an upgrade.
s/rob
| |||||
| 1089.11 | suspension rules | TOOK::SHMUYLOVICH | Wed Jun 05 1991 19:40 | 11 | |
Re:.10
If exporting has 10 failed polls in a row
(failed means cvr other that MCC_S_RESPONSE or
MCC_S_TIME_ALREADY_PASSED) the state becomes
"suspended". On my list for V1.2 there is an
item to use a logical for this value.
Sam
| |||||
| 1089.12 | And some day an attribute of Historian right? ;-) | WAKEME::ANIL | Wed Jun 05 1991 20:36 | 3 | |
Can we change that to a management parameter? ;) - Anil Navkal | |||||