Conference hydra::axp-developer

To: src@quadstone.com
Cc: pauline@reo.dec.com
Subject: Re: Digital ASAP call #22474 - Qudstone/Simon Chapple 
Date: Fri, 14 Mar 97 16:47:39 +0000
From: "Pauline Snyder" <pauline>
X-Mts: smtp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From:  Pauline Snyder                      Loc: REO/F8    DTN: 830 4096
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simon,

One more question  - Are you using a lot of shared libraries in your
application.  This is known to bump up the vm_mapentries requirement
somewhat but not to the extent you are seeing.  Could you e-mail your
makefile if releavant, please?

Also I did not copy myself on my first reply so could I please ask
you to forward it to me, thank you.

Best regards,

Pauline

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pauline Snyder                    Software Partner Engineering, Reading
Digital Equipment Company Ltd.    
Digital Park,  Worton Grange                      
Imperial Way                                 email: pauline@reo.dec.com
Reading  RG2 0TE                             tel:      +44.118.920.4096
England                                      fax:      +44.118.920.4146
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=======================================================================

To: src@quadstone.com
Cc: pauline@reo.dec.com
Subject: Re: Digital ASAP call #22474 - Qudstone/Simon Chapple  
Date: Mon, 17 Mar 97 09:30:17 +0000
From: "Pauline Snyder" <pauline>
X-Mts: smtp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From:  Pauline Snyder                      Loc: REO/F8    DTN: 830 4096
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simon,

I have some further information about mapentries, attached.

What version of Digital UNIX are you running ?  The cda tool version
depends on the OS version.  I believe cda is an "internal use only"
tool which I can distribute at my discretion - I am checking this.

Would it be feasible for you to bring your application to our lab in 
Reading for a closer look - I do not know where you are located in the
UK ?

Best regards,

Pauline

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pauline Snyder                    Software Partner Engineering, Reading
Digital Equipment Company Ltd.    
Digital Park,  Worton Grange                      
Imperial Way                                 email: pauline@reo.dec.com
Reading  RG2 0TE                             tel:      +44.118.920.4096
England                                      fax:      +44.118.920.4146
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=======================================================================


------- Forwarded Message - extracts

Subject: Re: FWD: Why would you need vm_mapentries = 1 million ?

Pauline,

The map entry structures are used to describe virtual memory regions in
user address space.  They form a linked list in the vm_map structure
describing all the mapped regions in the address space.  For eg. a
typical executable has four sections in it, text, data, bss and stack.
These regions are reflected in the kernel as four separate map
entries.  With shared libraries the number of mapentries used increases
with the number of shared libraries mapped by the application.  Each
shared library will have its own data/bss sections.

Most memory allocations tend to grow an existing map entry.  Examples
are stack and heap (malloc(3)) growth.  In order to grow an existing
region, the new region must be adjacent and the properties of the
region must match the existing one.

mmap(2) allows a user to map files or anonymous memory into the address
space.  mmap()-ing anonymous memory is similar to malloc() and
generally grow existing region.

When mmap()-ing a file, the newly allocated region cannot grow an
adjacent region.  This is because the existing region could be
anonymous memory or mapping a different file.  Thus a map entry is
created to describe the new mapping.  [The exception is when the mmap()
is mapping the same file and the offset into the file make the new
mapping contiguous to the previous region.]

Now to your questions:

>What resources are allocated in memory for each increment in
>vm_mapentries ?  Is it just a mapent struct or is vm_mapentries also
>used to calculate other allocations ?

None.  It is just a limit to prevent (ab)user from consuming too much
system resources.  This is similar to the other Unix limits, such that
Sysadmins have some control over what users can do.  We are revisiting
some of these limits (mapentries, vpagemax etc.).

>This >application is using threads and memory mapped files.  How does
this >impact mapent usage.

Threads allocate stacks which also allocate guard pages (a region of
virtual memory with no access) to protect stack overflow from
corrupting other data.  Inserting a guard page causes an existing map
entry to be split into 2-3 pieces, depending on where the guard pages
are.  You can expect two map entries to be used per thread.

>When are mapent resources freed ?

When the mapped region is deallocated by the application or when the
process exits.  With DECthreads, a pthread_terminate() may or may not
deallocate a stack.  The thread may not be terminated but cached in the
library, or the thread is terminated but the stack (and the guard
pages) is cached.

>Most big installations I have seen vm_mapentries goes to 20,000 or
>potentially even 40,000 but nothing even close to 1 million.  What
>could eat up the resources this way?  Can you have a mapent 'leak'?

The application(s) either have larger number of thread or have many
files mapped or there is a VM bug.  Note that the limit is a per-process
limit.  If an application is not creating 20,000 threads or do not map
20,000 files and still require the limit to be that high, there is some
kernel bug.

You may want to use the "cda" tool to examine a running application's
address map.  

The command to use to list memory regions:

	cda> vm_map -p process-id
or
	cda> vm_map -c command-name

The number of regions indicate the number of map entries used.

Eg. (my comments follows the # sign)

cda> vmm -c cda
21821: cda.V4.0-386
  #              start                end
  0        0x11ffe0000        0x120000000   # this is the stack
  1        0x120000000        0x120800000   # text
  2        0x140000000        0x140016000   # data
  3        0x140016000        0x140034000   # bss
  4      0x20000000000      0x20007f16000   # /dev/mem is mmap()-ed here
cda>

You can see that cda uses 5 map entries.

--s

------- End of Forwarded Message

To: Pauline Snyder <pauline@reo.dec.com>
Subject: Re: Digital ASAP call #22474 - Qudstone/Simon Chapple
Cc: src@quadstone.co.uk
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii

On Mar 17,  9:30am, Pauline Snyder wrote:
> Subject: Re: Digital ASAP call #22474 - Qudstone/Simon Chapple
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> From:  Pauline Snyder                      Loc: REO/F8    DTN: 830 4096
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Simon,
>
> I have some further information about mapentries, attached.
>
> What version of Digital UNIX are you running ?  The cda tool version
> depends on the OS version.  I believe cda is an "internal use only"
> tool which I can distribute at my discretion - I am checking this.
>
> Would it be feasible for you to bring your application to our lab in
> Reading for a closer look - I do not know where you are located in the
> UK ?
>
> Best regards,
>
> Pauline

Thanks for the information Pauline, I think it confirms my suspicions.

The nature of our application is such that sections of tens of files
are typically mapped and unmapped repeatedly as data is traversed from
the files. The sections that are unmapped need not correspond exactly
with those that were mapped. Our application does keep a record of
the number of sections and size of each that has been mapped. We could
use this to compare with the output of the cda tool - our application
never goes beyond a few hundred separate mapped regions at any one time.
Perhaps the unmapping is not releasing a vm_mapentry resource?

We could make a trip to Reading at some point - but this could not be
scheduled for at least six weeks, unless we were in the vicinity anyway.

Thanks for your help,
Cheers for now,
Simon
-------------
Simon Chapple
Quadstone Ltd
Edinburgh :-)

-- 

To: src@quadstone.com
Cc: pauline@reo.dec.com
Subject: I: cda tool + 
Date: Tue, 18 Mar 97 12:21:33 +0000
From: "Pauline Snyder" <pauline>
X-Mts: smtp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From:  Pauline Snyder                      Loc: REO/F8    DTN: 830 4096
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simon,

I am sending a 4mm DAT tape to you at 

Quadstone Ltd
16 Chester Street
Edinburgh
EH3 7RA
 
The tape contains cda for Digital UNIX 3.2c-g and Digital UNIX 4.0any
plus the documentation (such as it is) in html and text (extracted 
from the web so it may be strangely formatted).

Also I have included another tool called dcpi which can be used to
profile the whole system.  It works via a kernel driver which
collects system data, a modified loader and a daemon.  It captures 
the data with low system overhead into files which can then be 
analysed with a series of perl tools. This has proved
useful to other partners with complex applications, the only catches
being that you need to build with symbol information and there is a 
learning curve for its usage.


Best regards,

Pauline

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pauline Snyder                    Software Partner Engineering, Reading
Digital Equipment Company Ltd.    
Digital Park,  Worton Grange                      
Imperial Way                                 email: pauline@reo.dec.com
Reading  RG2 0TE                             tel:      +44.118.920.4096
England                                      fax:      +44.118.920.4146
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=======================================================================

To: src@quadstone.com, esctech@ilo.mts.dec.com
Cc: pauline@reo.dec.com, dgh@quadstone.com, asap@rdgeng.enet.dec.com
Subject: I: Re: Digital ASAP call #22474 - Quadstone - CLOSED 
Date: Tue, 08 Apr 97 15:40:25 +0100
From: "Pauline Snyder" <pauline>
X-Mts: smtp

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From:  Pauline Snyder                      Loc: REO/F8    DTN: 830 4096
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simon,

I understand from Douglas Hanley that you were too pressed to run cda on
the loan server before it was returned and also you are now on holiday
for two weeks.

Consequently I am closing the call for now.  As the problem is still
unresolved I suggest that when you are in a position to pursue
investigation you re-open the call via asap_euro@ilo.mts.dec.com
quoting the point number above.

The offer of time in our lab is still open should you wish to take it up.

Best regards and hope you had a good holiday,

Pauline

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pauline Snyder                    Software Partner Engineering, Reading
Digital Equipment Company Ltd.    
Digital Park,  Worton Grange                      
Imperial Way                                 email: pauline@reo.dec.com
Reading  RG2 0TE                             tel:      +44.118.920.4096
England                                      fax:      +44.118.920.4146
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=======================================================================