[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference azur::mcc

Title:DECmcc user notes file. Does not replace IPMT.
Notice:Use IPMT for problems. Newsletter location in note 6187
Moderator:TAEC::BEROUD
Created:Mon Aug 21 1989
Last Modified:Wed Jun 04 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:6497
Total number of notes:27359

1566.0. "Reason why ILV is restricted to 32,500-byte buffers ?" by BYBLOS::TAMER () Mon Sep 30 1991 18:57

I would like to know why ILV (actually ASN.1) was restricted to 32,500 byte 
buffers when some MCC datatypes can contain up to 65,535 bytes ! 

Per the SRM protocol, I am requesting large out_p buffers from the PM clients, 
and they are obliging without apparently cheking for the buffer sizes. 
However, when the buffers are larger than 32,500 bytes, the message of the
MCC_S_ILVBUFTOOBIG exception is returned when they go to ILV decode the out_p 
buffer.

A question:

a. Can this restriction be lifted, to say allow up to  65,535 <(2**16) - 1> 
   or more bytes ?

b If not, is there a way to segment data to the MCC clients. i.e., to return
  a 65,000 byte buffer (for one attribute), the AM returns them in two or more 
  replies but they get displayed as one buffer to the user.

Thanks,
Phil
T.RTitleUserPersonal
Name
DateLines
1566.1TOOK::SWISTJim Swist LKG2-2/T2 DTN 226-7102Tue Oct 01 1991 11:127
    The "L" in TLV is 2 bytes long, which would give you a max of 32767
    unless it treats the field as unsigned at which point it could
    conceivably go up to 65536, but ASN uses the high order bit of its tags
    for various purposes so I wouldn't be surprised if it reserved the
    sign bit of the length field also.   Maybe as ASN.1 heavy could
    elaborate...
    
1566.2code and history, not concept TOOK::KOHLSRuth KohlsThu Oct 03 1991 13:0825
        The 32500 byte restriction is due to the way ILV/ASN was originally
coded, and due to the restrictions on VMS descriptors which has been 
inherited by mcc descriptors.  A VMS descriptor can point to 65535 of an
item, in the ILV/ASN case, bytes.  Due to size expectations and code
efficiency, the ILV/ASN code reserves 2 length bytes after each tag.  
The ASN.1 BER specifies that if there is more than one byte of length, 
the 7th bit is reserved for continuation. So, we loose another bit.  
The remainder of the loss is a safety factor & rounding.

        We could gain the extra bit by reserving an additional byte for the 
length and adding checking to be sure the actual length fits our descriptors. 
Note that this would require one byte per tag-- one byte for every
ilv_put_xxx call in Native Mode.
        Since the majority of the encodings don't require even 2 length bytes, 
and a better way to do this would be to re-write the mcc_asn routines to "save
bytes" by not pre-reserving the length space, the factor of two in available 
encoding buffer bytes has been sacrificed.

So, the answer to a) is no, not now.

I'm afraid I don't know about b).

Ruth Kohls

1566.3Is this going to be fixed in the future ?BYBLOS::TAMERThu Oct 03 1991 14:5910
re .2,

Thanks Ruth for the explanation. 

Are there firm, future plans to fix this SERIOUS PROBLEM, say by re-writing 
the mcc_asn-* routines ?

regarding b) in the base note, I am still hoping to receive an answer.

Phil 
1566.4TOOK::SWISTJim Swist LKG2-2/T2 DTN 226-7102Thu Oct 03 1991 15:434
    Why is this a SERIOUS PROBLEM  (your caps, not mine)?
    
    What are you ILV encoding?
    
1566.5Using the formal process, Put a requirement on usTOOK::KOHLSRuth KohlsThu Oct 03 1991 16:0028
>Are there firm, future plans to fix this SERIOUS PROBLEM, say by re-writing 
>the mcc_asn-* routines ?

Now is the time to submit v2.0 requirements.  Do you know of another
group that is hitting this limit, or the nesting depth of 12 limit?
Can you tell me that 65 K is your absolute maximum size requirement, or
what is?  Also, nesting depth requirements.

Put your size requirements for buffers or a requirement for segmentation
and reassembly of data buffers containing single encodings, or both, into 
the mcc_futures notesfile, and also send it to requirements gatherers, such
as the DECmcc product and engineering managements.  

>regarding b) in the base note, I am still hoping to receive an answer.

is there any way to split this 65 K attribute into smaller pieces?  Is it
a single Latin1string?  The only way I can think of to do what you want
*generically* is to use the handle protocol, but you will still need smaller 
buffers for attribute segments, and you would not be using the handle protocol 
"according to the book" -- this would be "why more? = data buffer segmentation".

Given a description of the attribute (size & size range, datatypes-- its
msl description) we might be able to come up with a *particular* solution.

Keith, Colin, Jim, other gurus--do you have ideas?

Ruth
1566.6The Call Protocol doesn't support Out-P data segmentationNANOVX::ROBERTSKeith Roberts - DECmcc Toolkit TeamThu Oct 03 1991 16:5627
The DECmcc Call Protocol (MCC_CALL_FUNCTION/ACCESS) only returns one
Entities Partition worth of data at a time.  Like all the Characteristic
Attributes for a Phase 4 Node (show node foo all char).

The protocol doesn't allow for returning segmented Out-P buffers - at least
not today.

Now, for a moment, lets imagine that there is no ILV.  Your Out-P buffer
is an in-memory structured list of MCC Descriptors...of (virtually) 
unlimited size!  Sounds good.  But this would only work if the two Management
Modules were in memory together - as in the DECmcc/VMS model we use today.

For Ultrix, which uses an RPC method, the in-memory Out-P structure would
need some kind of encoding. The data would be shipped over to the remote
Mangement Module (MM) and decoded again.  This would be transparent to your MM.

The mechanism which the IM/Dispatcher (who would route the Data Request) uses
to encode and decode the data would require the ability to manage (virtually)
unlimited size buffers - ILV could be extended here - or whatever encoding
technologies is appropriate.  If the encoding technique actually buffered
up the data, piped it to the remote side which decoded it on the fly, then
there wouldn't need to be this single chuck of intermediate memory (ie, 
the ILV buffer).

Food for thought.

/keith
1566.7I will have on the requirement list.BYBLOS::TAMERThu Oct 03 1991 20:3431
The attributes in question are Expression datatypes, which are basically 
Latin1strings enclosed in parentheses. These are application info and some of
them might be quite large. I was hoping to place a restriction of the
architected size of 65,535. 

I will have it on my requirement list.

re .4,

I consider it a problem because several datatypes are claimed to support up
to 65,535 bytes, but currently there is no apparent way to encode half that 
amount ! 

For now, I guess I have to live with it or have to find a way around it.  



On a related subject,

What is the limit on the PM input buffer? I have an FCL script that issues
a CREATE directive with an argument of type <SET OF RECORD>. When I have
a few elements (each field on a separate line), the FCL is not accepting the 
input. It looks like it is trying to truncate or don't accept more than 
something like 1024 bytes.
Is this true ? If so, why is it this size ? What about the IMPM, what size
does it impose on input ?



Thanks for all replies,
Phil  
1566.8FCL 1024 SMG buf limitTOOK::CALLANDERMCC = My Constant CompanionThu Oct 17 1991 19:344
or so that was my understanding as to why the limit (also that seemed
awful huge).

jill
1566.9why 2-byte length fields were usedNAC::ENGLANDMon Nov 04 1991 16:3318
First of all, MCC already should be able to parse length fields of 1-4 bytes
and indefinite-length.  The only reason that it encodes using 2-byte
lengths is that it makes it MUCH easier to predict how many bytes of
buffer space are required for a constructed value, such as a SET OF.
There is no need for the strict 2-byte encoding for primitive values,
since you already know the length of a primitive value at the time of the
PUT call.  There are other options, such as indefinite-length encoding.
Indefinite-length encoding has the added advantage that you don't have
to know the length of the entire PDU before you send any of it!
A MCC_ASN rewrite is not required to fix this problem.

Keith's idea is good, the DECnet-ULTRIX CMIP protocol engine works more or
less that way.  I didn't do MCC_ASN that way because there was no 
concept of an input or output device associated with an ASN.1 message
or ILV buffer.  This was to all be handled by the AM, dispatcher, etc.
My only concern was to make sure the data could traverse multiple address
spaces with no change (for RPC support).