[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference vmszoo::rms_openvms

Title:RMS asks, 'R U Journaled?'
Moderator:STAR::TSPEERUVEL
Created:Tue Mar 11 1986
Last Modified:Wed Jun 04 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:3031
Total number of notes:12302

3022.0. "Need help with huge file on OpenVMS" by PACKED::MASLANKA () Wed Apr 16 1997 19:33

Hi,

We received a sequential file which contains 27861760 
bytes. It is formatted into 80 byte records. So, by 
calculation there are 348,272 records. The file contains 
no newline characters or other obvious delimiting markers, 
and so it appears as one humongous record to editors and 
other programs. Also, it exceeds the physical limits
which are specified for COBOL programs by quite a lot.
A C program using fopen was unable to open the file.
A dir/full for the file follows. Hopefully, someone else
has had experience dealing with a monster of this size.

Thank you.

John Maslanka

POPFILE.FLP;1                 File ID:  (1548,28,0)
Size:        54418/54420      Owner:    [COBOL,C79QA]
Created:  10-APR-1997 12:08:53.83
Revised:  14-APR-1997 13:59:43.77 (3)
Expires:   <None specified>
Backup:   15-APR-1997 17:35:47.11
File organization:  Sequential
File attributes:    Allocation: 54420, Extend: 0, Global buffer count: 0, No version limit
Record format:      Stream
Record attributes:  Carriage return carriage control
RMS attributes:     None
Journaling enabled: None
File protection:    System:RWED, Owner:RWED, Group:RE, World:
Access Cntrl List:  None

Total of 1 file, 54418/54420 blocks.

T.RTitleUserPersonal
Name
DateLines
3022.1this is NOT a huge fileSTAR::EWOODSWed Apr 16 1997 19:4518
  Well, the file itself is not a monster.  In fact, in terms of real-world
  files, it is trivial in size.  The problem is simply that it describes
  itself as stream with no delimiters.

  Since you know that it really contains fixed 80-byte records
  (which divides evenly into the total # bytes), you can just
  reset the file attributes on OpenVMS to match the reality of its 
  contents as follows:

  $ SET FILE/ATTR=(RFM=fix,MRS=80,LRL=80) filespec
  
<< Also, it exceeds the physical limits which are specified for COBOL 
<< programs by quite a lot.
  
  I sure hope you mean the record size here and not the file size.
   
  -- Elinor
3022.2Thank you, ElinorPACKED::MASLANKAThu Apr 17 1997 17:015
Re .1

>  I sure hope you mean the record size here and not the file size.
   
    The record size.
3022.3Consider a 28 MB section file, too...XDELTA::HOFFMANSteve, OpenVMS EngineeringThu Apr 17 1997 18:43140
   As an alternative to processing a disk file...

   If you know there's nothing in this file but data, and you know that
   each record occupies 80 bytes, then set your PGFLQUOTA and your local
   VIRTUALPAGECNT (assuming this is not V7.0 or later on OpenVMS Alpha),
   sys$crmpsc-map this file as the backing storage file for a private
   section, use the memory management to page-fault the file data into
   memory, and then access the contents of the section directly.  (Using
   this technique, you will see reasonable performance levels, too. :-)

   (And as mentioned, 28 megabytes isn't particularly large by local
   standards -- one of the local files used quite regularly here in
   OpenVMS is well over 1,100,000 blocks; over 550 megabytes.  One
   of the folks tells me that a major file at the CSC is presently
   over 9,600,000 blocks; almost 5 gigabytes.  And files that must
   span multiple disk volumes are definitely around, via volume sets
   or via striping...)

   The attached program uses sys$create and "CIF" -- it will create the
   backing file if not found -- and you may want to use sys$open without
   this option.  And as the attached program is an example, and it maps
   the file into memory twice as a demonstration -- and you likely won't
   want to do this...

	--

#include <descrip.h>
#include <lib$routines.h>
#include <psldef.h>
#include <rms.h>
#include <secdef.h>
#include <ssdef.h>
#include <starlet.h>
#include <stdio.h>
#include <string.h>
#include <stsdef.h>
#include <unixlib.h>

#define MAXACCLEN 16

struct ItemList3
    {
    short int ItemLength;
    short int ItemCode;
    void *ItemBuffer;
    void *ItemRetLen;
    };
struct RmsFileContext
    {
    struct FAB fab;
    struct NAM nam;
    char rss[NAM$C_MAXRSS];
    short max_rec_siz;
    char *data_buffer;
    };
#define P0SPACE ((void*)0x0200)
#define BOGUSMAX 10

RmsFileOpen( struct FAB *fab, char *FileName, char *DefFileName )
    {
    int RetStat;

    *fab = cc$rms_fab;

    fab->fab$l_alq = 10;
    fab->fab$b_fac = 0;
    fab->fab$l_fop = FAB$M_UFO | FAB$M_CIF;
    fab->fab$b_shr = FAB$M_UPI | FAB$M_SHRPUT | FAB$M_SHRGET | FAB$M_SHRUPD;

    fab->fab$l_fna = FileName;
    fab->fab$b_fns = strlen( FileName );
    fab->fab$l_dna = DefFileName;
    fab->fab$b_dns = strlen( DefFileName );

    /*
    // Attempt to open the file...
    */
    RetStat = sys$create( fab, 0, 0 );
    if ( !$VMS_STATUS_SUCCESS( RetStat ) )
	return RetStat;

    return RetStat;
    }

main()
    {
    int RetStat;
    struct ItemList3 ItmLst[10];
    $DESCRIPTOR( SecDsc, "FACNAM_GLOBAL_SECTION_NAME" );
    int i;
    void *InAdr1[2] = {P0SPACE,P0SPACE};
    void *RetAdr1[2] = {NULL,NULL};
    void *InAdr2[2] = {P0SPACE,P0SPACE};
    void *RetAdr2[2] = {NULL,NULL};
    struct FAB Fab1, Fab2;
    struct insec
	{
	int Bogus[BOGUSMAX];
	} *Sec1, *Sec2;

    /*
    //	Create and open, and map the global section...
    */
    RetStat = RmsFileOpen( &Fab1, "BOGUS", "SYS$SCRATCH:.TMP" );
    if (!$VMS_STATUS_SUCCESS( RetStat ))
	lib$signal( RetStat );
    RetStat = sys$crmpsc( InAdr1, RetAdr1, PSL$C_USER, 
	SEC$M_EXPREG | SEC$M_WRT | SEC$M_DZRO | SEC$M_GBL,
	&SecDsc,
	0, 0, Fab1.fab$l_stv, 1, 0, 0, 0 );
    if (!$VMS_STATUS_SUCCESS( RetStat ))
	lib$signal( RetStat );

    /*
    //	Create and open, and map the global section again...
    */
    RetStat = RmsFileOpen( &Fab2, "BOGUS", "SYS$SCRATCH:.TMP" );
    if (!$VMS_STATUS_SUCCESS( RetStat ))
	lib$signal( RetStat );
    RetStat = sys$crmpsc( InAdr2, RetAdr2, PSL$C_USER, 
	SEC$M_EXPREG | SEC$M_WRT | SEC$M_GBL,
	&SecDsc,
	0, 0, Fab2.fab$l_stv, 1, 0, 0, 0 );
    if (!$VMS_STATUS_SUCCESS( RetStat ))
	lib$signal( RetStat );

    /*
    //	Write the information to one "window"...
    //	... read the data back from the other.
    */
    Sec1 = RetAdr1[0];
    Sec2 = RetAdr2[0];
    for ( i = 0; i < BOGUSMAX; i++)
	Sec1->Bogus[i] = i;
    for ( i = 0; i < BOGUSMAX; i++)
	printf( "Bogus[%d] = %d\n", i, Sec2->Bogus[i] );

    return SS$_NORMAL;
    }