[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference azur::mcc

Title:DECmcc user notes file. Does not replace IPMT.
Notice:Use IPMT for problems. Newsletter location in note 6187
Moderator:TAEC::BEROUD
Created:Mon Aug 21 1989
Last Modified:Wed Jun 04 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:6497
Total number of notes:27359

1069.0. "Exporter Performance issue" by ZPOVC::RAMARAJ () Thu May 30 1991 11:35

I need to collect statistics from a node4, line, circuit and node itself, to
do utilization graphs daily, for circuits and reports for line and node.

I've setup exporting for all these, pointing to a single database.
This needs to be running all the time, at least 8-5 everyday.  I find that 
the batch process, and its subprocess, doing this exporting is growing in 
memory size. 

The file size of the xxx.rdb is also growing.

Is there any easier way to do this, without the system running out of memory
and space after some time.

Is there a way to purge away the old data in the export database, like more
than 1 month old, just like the outage in RECORDING utility.

Also, are there any improvement being planned to the exporter, recorder,
and reporting utilities in the next version?

Thanks for any suggestions.

Raj
SWS Singapore
T.RTitleUserPersonal
Name		DateLines
1069.1qar 684 enteredTOOK::CALLANDERJill Callander DTN 226-5316Thu May 30 1991 15:479
I entered the memory growing as a qar in the MCC_INTERNAL database, number
684.

As to how to get around things like this for now, while we investigate potential m
memory leaks, have a batch job that setups and runs the exporter for 5 days
at a time, (5 was an optional choice) and then resubmits itself to restart
it every time is expired. This will cause the exporter to exit and reclaim
any lost space.
1069.2Not easilyNSSG::R_SPENCENets don't fail me now...Thu May 30 1991 16:407
    The exporter NOW runs as a batch job and is never intended to exit. In
    fact, it is usually set up to restart by itself after a system restart.
    
    There is no mechanisim to ask the exporter to exit other than kill the
    batch job or the process. Very messy.
    
    s/rob
1069.3TOOK::CALLANDERJill Callander DTN 226-5316Thu May 30 1991 17:471
I didn't say it wasn't messy...any other ideas?
1069.4Memory Leak...SUBWAY::YANNIOSMon Jun 03 1991 14:4613
    I am also having the same problem as Raj.  I have the exporter
    running collecting stats on 15 routers. The exporter batch job has
    incurred over 3.5 million page faults and has grown to a peak
    virtual size of over 33,000 pages.  I have 113,600 pages total
    between my primary and secondary page files and I the reserve has
    gradually diminished and gone negative on my primary file and is 
    slowly depleting in the secondary file.
    
    Will cancelling the batch job corrupt andy of the data I have
    collected?  Is this safe?  Will there be a patch for this?
    
    Nick
1069.5memory leaks investigationTOOK::SHMUYLOVICHTue Jun 04 1991 14:1920
		


	In the last two days I investigate memory problems in Exporter.
This is what I found:

	1. Node (not node4) gives a significant memory leaks due to problem
	   in PhaseV.

	2. I setup 3 exporting (for node4, line and circuit) with polling
	   interval 5 min. They are running for more that day and I' monitoring
	   background process (show proc/cont). The number of virtual pages
	   increased from 11938 to 12194.
	   I still can not find who (exporter, dna4 or pa) gives this problem.

    
	   
Sam
1069.6delete commandTOOK::CALLANDERJill Callander DTN 226-5316Wed Jun 05 1991 14:504
use the delete exporting command from inside MCC instead of killing the
batch job. I believe if you delete them all the process will go away.
Sam is that right?
1069.7NSSG::R_SPENCENets don't fail me now...Wed Jun 05 1991 17:337
    I don't think that deleting all the exporting for a target file will
    cause the background to go away. As it is, you can start the background
    with no current export directives active. I don't know of any way to
    get the RDB files closed from within DECmcc (a problem as far as I am
    concerned).
    
    s/rob
1069.8Plug up the Memory Leak?SUBWAY::YANNIOSThu Jun 06 1991 19:486
    
    Any update on the memory leak problem and whether there will be a
    patch? 
    
    Nick
1069.9F18::ROBERTMon Oct 07 1991 23:2521
    I am running into the same problem, I have read all the release notes
    and made sure that all my system parameters were ok.
    
    I have a vaxstation model 48 with 32 md of memory and tons of disk
    space. I have a page file that is 160,000 pages.
    
    By the time mcc is up and running and I am looking at one node in
    New Jersey, one in Ohio, the hub in St. Louis and one node in
    Sacramento, I only have one exporting job on each system, I have eaten
    all but 2000 k of memory. I am typing this in from home, if I remember
    correctly I am running mcc 1.1  -
    
    I cannot start any queues for the historian function, and my system
    grinds to a halt. Am I doing something terribly wrong???
    
    I am learning very fast the hard way "OJT" the DEC way.
    
    Can someone shed some enlightenment on me????
    
    Thanks Dave
    DTN 445-2007 or 445-2072
1069.10More info neededTOOK::R_SPENCENets don't fail me now...Tue Oct 08 1991 13:1411
    Please post the UAF entry for the USERNAME you are using.
    
    Also, have you run the AUDIT_MCC.COM proceedure? It can be found
    at NSSG::SYS$PUBLIC:[MCC.S-KIT]AUDIT_MCC.COM .
    
    You should be able to create queues.
    
    What applications other than the DECmcc Director are running on the
    system?
    
    s/rob
1069.11F18::ROBERTTue Oct 08 1991 14:01156
    This is what my authorize file looks like, I will copy this .com file
    that you mention over to my system.
    
    UAF> show system
    
    Username: SYSTEM                           Owner:  SYSTEM MANAGER
    Account:  SYSTEM                           UIC:    [1,4] ([SYSTEM])
    CLI:      DCL                              Tables: DCLTABLES
    Default:  SYS$SYSROOT:[SYSMGR]
    LGICMD:   LOGIN
    Flags:
    Primary days:   Mon Tue Wed Thu Fri
    Secondary days:                     Sat Sun
    No access restrictions
    Expiration:            (none)    Pwdminimum:  8   Login Fails:     0
    Pwdlifetime:         30 00:00    Pwdchange:   4-OCT-1991 09:55
    Last Login:  8-OCT-1991 09:18 (interactive),  8-OCT-1991 09:38
    (non-interactive)
    Maxjobs:         0  Fillm:       100  Bytlm:        40000
    Maxacctjobs:     0  Shrfillm:      0  Pbytlm:           0
    Maxdetach:       0  BIOlm:        36  JTquota:       2048
    Prclm:          36  DIOlm:        18  WSdef:         2048
    Prio:            4  ASTlm:        48  WSquo:         8192
    Queprio:         0  TQElm:       128  WSextent:     10000
    CPU:        (none)  Enqlm:      2000  Pgflquo:      99990
    Authorized Privileges:
      CMKRNL CMEXEC SYSNAM GRPNAM ALLSPOOL DETACH DIAGNOSE LOG_IO
      GROUP ACNT PRMCEB PRMMBX PSWAPM ALTPRI SETPRV TMPMBX WORLD
      OPER EXQUOTA NETMBX VOLPRO PHY_IO BUGCHK PRMGBL SYSGBL MOUNT
      PFNMAP SHMEM SYSPRV BYPASS SYSLCK SHARE GRPPRV READALL
      SECURITY
    Default Privileges:
      CMKRNL CMEXEC SYSNAM GRPNAM ALLSPOOL DETACH DIAGNOSE LOG_IO
      GROUP ACNT PRMCEB PRMMBX PSWAPM ALTPRI SETPRV TMPMBX WORLD
      OPER EXQUOTA NETMBX VOLPRO PHY_IO BUGCHK PRMGBL SYSGBL MOUNT
      PFNMAP SHMEM SYSPRV BYPASS SYSLCK SHARE GRPPRV READALL
      SECURITY
    UAF>
    
    
    $ show mem
                  System Memory Resources on  8-OCT-1991 10:57:17.83
    
    Physical Memory Usage (pages):     Total        Free      In Use   
    Modified
      Main Memory (32.00Mb)            65536        2240       61281       
    2015
    
    Slot Usage (slots):                Total        Free    Resident    
    Swapped
      Process Entry Slots                 90          58          32          
    0
      Balance Set Slots                   81          51          30          
    0
    
    Fixed-Size Pool Areas (packets):   Total        Free      In Use       
    Size
      Small Packet (SRP) List           4100        2600        1500        
    112
      I/O Request Packet (IRP) List      936          56         880        
    176
      Large Packet (LRP) List             30          20          10       
    1648
    
    Dynamic Memory Usage (bytes):      Total        Free      In Use    
    Largest
      Nonpaged Dynamic Memory         999936      557920      442016     
    518752
      Paged Dynamic Memory           1000448      597904      402544     
    593552
    
    Paging File Usage (pages):                      Free  Reservable      
    Total
      DISK$PAGESWAP:[SYSEXE]SWAPFILE.SYS;1          7992        7992       
    7992
      DISK$PAGESWAP:[SYSEXE]PAGEFILE.SYS;1        136328       50465     
    159992
    
    Of the physical pages in use, 8275 pages are permanently allocated to
    VMS.
    
    
    $ @ws
    Working set information:
    
                                     WS    WS    WS     WS   Pages   Page
    Username    Processname   State  Extnt Quota Deflt  Size in WS   Faults
    Image
    
    <system>    SWAPPER        HIB       1     1     1     0     0        0
    SYSTEM      ERRFMT         HIB     512   512   239   239   125       84
    ERRFMT
    SYSTEM      OPCOM          HIB    2048   512   239   389   152      267
    OPCOM
    AUDIT$SERVERAUDIT_SERVER   HIB    1024   478   239   539   483     1352
    AUDIT_SE
    SYSTEM      JOB_CONTROL    HIB    8192   768   239   539   375      167
    JOBCTL
    SYSTEM      IPCACP         HIB    4096  4096  2048  2048   141       80
    IPCACP
    SYSTEM      TP_SERVER      HIB    4096  4096  2048  2048   255      171
    TPSERV
    SYSTEM      BATCH_115      HIB   10000  2048   512  6362  4721     3966
    MCC_EXPO
    SYSTEM      BATCH_116      HIB   10000  2048   512  7562  5667     5553
    MCC_EXPO
    SYSTEM      BATCH_117      HIB   10000  2048   512  6212  4554     3954
    MCC_EXPO
    SYSTEM      BATCH_118      HIB   10000  2048   512  6962  5194     4788
    MCC_EXPO
    DECNET      NETACP         HIB    1500   478   239   539   368      202
    NETACP
    DECNET      EVL            HIB     500   478   239   478    47    64026
    EVL
    SYSTEM      REMACP         HIB     500   478   239   239    56       80
    REMACP
    SYSTEM      RDMS_MONITOR   LEF   16400 16400   239   389    52     2911
    RDMMON
    SYSTEM      RPC$SWL        HIB     500   478   239   389   195      123
    RPC$SWLP
    SYSTEM      DNS$ADVER      HIB    2000   478   239   389   290      210
    DNS$ADVE
    DNS$SERVER  DNS$TA         HIB    2000   600   239   900   600      579
    DNS$TRAN
    DNS$SERVER  DNS$BACK       HIB    2000   600   239   600   600      827
    DNS$BACK
    SYSTEM      SYSTEM_1       HIB   10000  2048   512 10000  2873     4287
    MCC_EXPO
    SYSTEM      SYSTEM_2       HIB   10000  2048   512 10000  2558     3998
    MCC_EXPO
    SYSTEM      SYSTEM_3       HIB   10000  2048   512 10000  4197     3189
    MCC_EXPO
    SYSTEM      SYSTEM_4       HIB   10000  2048   512 10000  4675     3371
    MCC_EXPO
    SYSTEM      DECW$SERVER_0  HIB    4000  1300   500  3350  2497    19619
    DECW$SER
    SYSTEM      SYSTEM         LEF   10000   478   478  2728  1977     5121
    DECW$SES
    SYSTEM      DECW$WM_1      LEF    4000   650   512  2312  1648     1349
    DECW$WIN
    SYSTEM      DECW$TE_1      LEF    4000   650   239  4000  3999     8281
    DECW$TER
    SYSTEM      _TWA30:        HIB   10000  8192  2048 10000  9201    11233
    MCC_MAIN
    SYSTEM      _TWA32:        LEF   10000  8192  2048  2048   498      483
    DTEPAD
    SYSTEM      _TWA31:        CUR   10000  8192  2048  2048   425     3305
    SYSTEM      _TWA33:        LEF   10000  8192  2048  2048   252     5308
    
    Total number of pages counted in WS = 58675
    
    
    I will run the NSSG::SYS$PUBLIC:[MCC.S-KIT]AUDIT_MCC.COM on my system
    and let you know what is going on.
    
    Thanks Rob, Dave
1069.12Only need one ExporterTOOK::R_SPENCENets don't fail me now...Tue Oct 08 1991 14:1211
    You do not need a separate EXPORT Background Process for each entity.
    
    Since you have 4 background processes you have the 4 subprocesses as
    well which means you have 8 copies of the exporter code mapped into
    memory. That's where your memory went.
    
    For exporting data for the number of nodes you have including the child
    entities (line and circuit), one exporter is enough. Also, one RDB file
    is plenty.
    
    s/rob
1069.13F18::ROBERTTue Oct 08 1991 17:4316
    Rob,
    	You are right, I did not realize that I only needed one each background
    process for the historian and the exporter. This freed up a lot of
    memory. Also I ran the .com file that you mentioned in an earlier note,
    this said that I needed to change only 3 sysgen parameters.
    
    I am hacking away. I will let you know how things are going later on.
    I have to add the Graph package and the Vitalink software to complete
    the network. I will let you know how these go.
    
    I have read the info in this notes file on both of the above, is there
    anything else that I should know before I try to install these two
    tools?
    
    Thanks again, Dave
1069.14F18::ROBERTThu Oct 10 1991 22:527
    I started an export job from a system called stlouis from the node
    level, I also did one for line and circuit for stlouis. Is this
    overkill, if i start an export job at the node level in stlouis, do
    I need to do one for line and circuit?
    
    Thanks Dave
1069.15TOOK::R_SPENCENets don't fail me now...Fri Oct 11 1991 15:569
    What do you mean by "Export Job"?
    
    If you want Line info exported, then you must issue an EXPORT directive
    on the LINE information. Same goes for Circuit.
    
    If you only Export the NODE4 info, you will not have Line or Circuit
    info.
    
    s/rob