MVSFORUMS.com Forum Index MVSFORUMS.com
A Community of and for MVS Professionals
 
 FAQFAQ   SearchSearch   Quick Manuals   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

SORT EXCP on QSAM/BSAM
Goto page 1, 2  Next
 
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities
View previous topic :: View next topic  
Author Message
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Tue Sep 26, 2017 6:12 pm    Post subject: SORT EXCP on QSAM/BSAM Reply with quote

Sort EXCP is always less, when i am reading a Physical sequential file compared to Cobol.

How does sort allocates Buffer space and buffer number for the Physical sequential dataset ?

And what else parameters are considered for reading and writing a dataset.

The reason why i am asking is, Cobol is taking/consuming lot of excp, when i provide BUFNO=30, it reduces and when I increase the BUFNO to 50 EXCP increases little.

There should be some rule to identify the optimized bufno.

Also Please advise if we can have any other parameters for reducing the EXCP for a physical sequential file.



Thanks
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Tue Oct 03, 2017 10:36 am    Post subject: Reply with quote

Magesh_J,

DFSORT writes its own channel program for EXCP access method. It will look up the available storage and allocate the buffers dynamically. If your input dataset has properties (PDSE, EXTENDED/COMPRESSED datasets...) that require BSAM access method, then DFSORT calls BSAM under the covers.

If your COBOL program is doing a lot of I/O then check the file declaration and make sure you have "Block contains 0 records"
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Tue Oct 03, 2017 2:52 pm    Post subject: Reply with quote

Hi Kolusu,

Thanks for the kind response.

I checked all the programs, It has "Block contains 0 records"

I also noticed, DSNTYPE=LARGE specified for the data sets having more EXCP.

do we need to change/handle anything in COBOL for accessing DSNTYPE=LARGE data sets ?

Regards,
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Tue Oct 03, 2017 7:45 pm    Post subject: Reply with quote

Magesh_J wrote:


I also noticed, DSNTYPE=LARGE specified for the data sets having more EXCP.

do we need to change/handle anything in COBOL for accessing DSNTYPE=LARGE data sets ?

Regards,
Magesh


Magesh_J,

If you have DSNTYPE=LARGE, then you need BSAM access method.
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Tue Oct 03, 2017 9:18 pm    Post subject: Reply with quote

Hi Kolusu,

Thanks for the kind info.

Please advise how can i make the program to use BSAM access method.

Do we have any parameters in JCL or Cobol to use BSAM access method ?

Or when ever I use the dataset, i should specify DSNTYPE=LARGE, so that Cobol will use BSAM access method ?

Regards,
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Sun Oct 08, 2017 5:10 am    Post subject: Reply with quote

Magesh_J wrote:
Hi Kolusu,

Thanks for the kind info.

Please advise how can i make the program to use BSAM access method.

Do we have any parameters in JCL or Cobol to use BSAM access method ?

Or when ever I use the dataset, i should specify DSNTYPE=LARGE, so that Cobol will use BSAM access method ?

Regards,
Magesh


If you have DSNTYPE=LARGE , you are getting BSAM access method. I am not sure as to what you will achieve by switching to BSAM. Unless you have zHPF or high performing channel program, I don't see the advantage of switching to BSAM. With BSAM you can actually override the number of buffers allocated.


Try specifying NCP value of 128 in JCL for the DDname in question. Read upon NCP parm in JCL reference manual

https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieab600/dcbsub.htm
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Sun Oct 08, 2017 8:01 pm    Post subject: Reply with quote

Hi Kolusu,

Thanks for the kind inputs.

Kolusu wrote:

Try specifying NCP value of 128 in JCL for the DDname in question. Read upon NCP parm in JCL reference manual


It appears the maximum limit for NCP is 99 per below link

Multiple buffering

Also, it appears we can specify BUFNO or NCP, the system will decide to use NCP or BUFNO based on the method of access.

I tried with NCP=99, Job ran in 23 minutes with EXCP TOTAL 3,081,485

I tried with NCP=40, Job ran in 16 minutes with EXCP TOTAL 2,803,225

I tried with BUFNO=50, Job ran in 12 minutes with EXCP TOTAL 2,901,623


I understand elapsed time always varies depends on the CPU load, but the % of cpu utilized when i specify BUFNO=50 is greater than NCP=40

sdsf;da CPU% have 40 to 50 when BUFNO=50 and 25 to 30 when NCP=40 and 18 to 25 when NCP=99.

I ran twice each job to confirm if it is because of CPU load, since its a weekend CPU is available in plenty.

It always follow the same pattern.

I believe there should be some calculation to find the optimized buffer NCP/BUFNO for a dataset based on LRECL and blocksize.

Please advise


Thanks
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Mon Oct 09, 2017 8:57 am    Post subject: Reply with quote

Magesh_J wrote:
Hi Kolusu,

Thanks for the kind inputs.

Kolusu wrote:

Try specifying NCP value of 128 in JCL for the DDname in question. Read upon NCP parm in JCL reference manual


It appears the maximum limit for NCP is 99 per below link

Multiple buffering


You are looking at c++ manual. You need to look up the jcl manual. The limit is 255

Magesh_j wrote:

Also, it appears we can specify BUFNO or NCP, the system will decide to use NCP or BUFNO based on the method of access.

I tried with NCP=99, Job ran in 23 minutes with EXCP TOTAL 3,081,485

I tried with NCP=40, Job ran in 16 minutes with EXCP TOTAL 2,803,225

I tried with BUFNO=50, Job ran in 12 minutes with EXCP TOTAL 2,901,623


I understand elapsed time always varies depends on the CPU load, but the % of cpu utilized when i specify BUFNO=50 is greater than NCP=40

sdsf;da CPU% have 40 to 50 when BUFNO=50 and 25 to 30 when NCP=40 and 18 to 25 when NCP=99.

I ran twice each job to confirm if it is because of CPU load, since its a weekend CPU is available in plenty.

It always follow the same pattern.

I believe there should be some calculation to find the optimized buffer NCP/BUFNO for a dataset based on LRECL and blocksize.

Please advise


Thanks
Magesh



You need to ask the right question as to how to find the optimal bufno or ncp values but you started off with the access method question. So are you concerned abt the input file excps or the output files? Also what is the lrecl and recfm of the files involved?
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Mon Oct 09, 2017 10:07 am    Post subject: Reply with quote

Kolusu wrote:

You are looking at c++ manual. You need to look up the jcl manual. The limit is 255

You need to ask the right question as to how to find the optimal bufno or ncp values but you started off with the access method question


My bad, Sorry. Sad

kolusu wrote:

So are you concerned abt the input file excps or the output files? Also what is the lrecl and recfm of the files involved?


Code:

//IEFBR14  EXEC PGM=IEFBR14
//FILEED DD DSN=MAIN.FILEED,                         
//            DISP=(,CATLG),UNIT=SYSDA,                           
//            SPACE=&SPACEL,DSNTYPE=LARGE,                           
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//*                                       
//FILEEL DD DSN=MAIN.FILEEL,                         
//            DISP=(,CATLG),UNIT=SYSDA,                           
//            SPACE=&SPACEL,DSNTYPE=LARGE,                           
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//*                                                                 
//FILEEM DD DSN=MAIN.FILEEM,                         
//            DISP=(,CATLG),UNIT=SYSDA,DSNTYPE=LARGE,               
//            SPACE=&SPACEM,                                         
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//*                                                                 
//FILEER DD DSN=MAIN.FILEER,                         
//            DISP=(,CATLG),UNIT=SYSDA,DSNTYPE=LARGE,               
//            SPACE=&SPACER,                                         
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//*                                                                 
//FILEES DD DSN=MAIN.FILEES,                         
//            DISP=(,CATLG),UNIT=SYSDA,DSNTYPE=LARGE,               
//            SPACE=&SPACES,                                         
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//FILEET DD DSN=MAIN.FILEET,                         
//            DISP=(,CATLG),UNIT=SYSDA,DSNTYPE=LARGE,               
//            SPACE=&SPACET,                                         
//            DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08) 
//*
//*All files are opened in Output mode.
//PSTEP01  EXEC PGM=PGMNAME
//FILEED DD DSN=MAIN.FILEED,           
//            DISP=SHR,DCB=BUFNO=50 
//FILEEL DD DSN=MAIN.FILEEL,           
//            DISP=SHR,DCB=BUFNO=50                   
//*                                                   
//FILEEM DD DSN=MAIN.FILEEM,           
//            DISP=SHR,DCB=BUFNO=50                   
//*                                                   
//FILEER DD DSN=MAIN.FILEER,           
//            DISP=SHR,DCB=BUFNO=50                   
//*                                                   
//FILEES DD DSN=MAIN.FILEES,           
//            DISP=SHR,DCB=BUFNO=50                   
//*                                                   
//FILEET DD DSN=MAIN.FILEET,           
//            DISP=SHR,DCB=BUFNO=50                   
//*                                                   


Also same files are used as input in many jobs, so we also need to handle it using input mode, but that is a different job.

Thanks
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Tue Oct 10, 2017 1:24 am    Post subject: Reply with quote

Magesh_J wrote:

DCB=(DSORG=PS,RECFM=FB,LRECL=5448,BLKSIZE=0,BUFNO=08)
//*
//*All files are opened in Output mode.
Also same files are used as input in many jobs, so we also need to handle it using input mode, but that is a different job.

Thanks
Magesh


Your output file LRECL is 5448 and hence the ideal BLKSIZE would be 27240 means you can write 5 records per block. so were the datasets allocated with that BLKSIZE or a different block size?


So specify a BUFNO/NCP which is a multiple of 5 for better results. Just specifying a higher BUFNO will not help as , you also need to provide the below the line storage. Code a REGION=0M parm on the jobcard or step(Hope your shop does NOT have an IEFUSI exit to limit the storage). Also remember that your number of excps are dependent on the number of records written.

Specifying a BUFNO on IEFBR14 step will do NOTHING. The Bufno value comes into picture only when you are writing or reading.
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Tue Oct 10, 2017 3:04 pm    Post subject: Reply with quote

Hi Kolusu,

Thanks for the kind advise.

kolusu wrote:
Your output file LRECL is 5448 and hence the ideal BLKSIZE would be 27240 means you can write 5 records per block. so were the datasets allocated with that BLKSIZE or a different block size?


Code:

 Device type . . . . : 3390       
Data class . . . . . : DCDFAULT   
 Organization  . . . : PS         
 Record format . . . : FB         
 Record length . . . : 5448       
 Block size  . . . . : 27240       
 1st extent cylinders: 600         
 Secondary cylinders : 400         
 Data set name type  : LARGE       
 SMS Compressible. . : NO         


kolusu wrote:

Just specifying a higher BUFNO will not help as , you also need to provide the below the line storage.
Code a REGION=0M parm on the jobcard or step

Region=0m is already in place for the step.

Kolusu wrote:

so specify a BUFNO/NCP which is a multiple of 5 for better results.


So the formula is BLKSIZE/LRECL ?

For example if i have a data set of LRECL=120, and the blocksize would be 27960

27960/120 = 233.

Should i specify BUFNO=233

And for LRECL=80

27920/80 = 349

I think, i misunderstood, please advise.

Thanks
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Wed Oct 11, 2017 9:40 am    Post subject: Reply with quote

Magesh_J wrote:


So the formula is BLKSIZE/LRECL ?

For example if i have a data set of LRECL=120, and the blocksize would be 27960

27960/120 = 233.

Should i specify BUFNO=233

And for LRECL=80

27920/80 = 349

I think, i misunderstood, please advise.

Thanks
Magesh


Not really. For an LRECL of 120 , the ideal blksize is 27960 which means you can read/write 27960/120 = 233 records at a time. So each block can contain 233 records. Your BUFNO is the number of such blocks. If you specify BUFNO=5 you will have 5 blocks each that can contain 233 records.

Also don't specify a high number of buffers which will take up your below the line storage.
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Wed Oct 11, 2017 4:55 pm    Post subject: Reply with quote

Thanks Kolusu,

Kolusu wrote:

Also don't specify a high number of buffers which will take up your below the line storage


below - the - line - storage
Is it 16 MB ?

16x1024x1024 = 16777216 Bytes.

Which mean, My entire buffer should not cross 16 MB ?

And is it applicable for one file or for all files ?

In the example, i have 6 Large files, I have to make sure my buffer size should not cross 16777216 bytes for each file or for all files ?

What will happen if it crosses 16 MB ?

Thanks
Magesh
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12357
Topics: 75
Location: San Jose

PostPosted: Thu Oct 12, 2017 7:51 am    Post subject: Reply with quote

Magesh_J wrote:
Thanks Kolusu,

Kolusu wrote:

Also don't specify a high number of buffers which will take up your below the line storage


below - the - line - storage
Is it 16 MB ?

16x1024x1024 = 16777216 Bytes.

Which mean, My entire buffer should not cross 16 MB ?

And is it applicable for one file or for all files ?

In the example, i have 6 Large files, I have to make sure my buffer size should not cross 16777216 bytes for each file or for all files ?

What will happen if it crosses 16 MB ?

Thanks
Magesh

Magesh,

Even though it is 16 mb storage most shop restrict that to 10 mb. And that values is applicable for all the ddnames. Check the IEF032I or IEF033I which will display the storage.

If you exceed that limit with high bufno value you would end up with s878 abends
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Magesh_J
Intermediate


Joined: 21 Jun 2014
Posts: 259
Topics: 54

PostPosted: Thu Oct 12, 2017 2:17 pm    Post subject: Reply with quote

Hi Kolusu,

Please find the below message for the step.
Code:

IEF032I STEP/XXXX   /STOP  2017284.1944                                     
        CPU:     0 HR  05 MIN  06.06 SEC    SRB:     0 HR  00 MIN  12.96 SEC 
        VIRT:   500K  SYS:   392K  EXT:    74248K  SYS:    14068K             


It has lot of numbers, which one i should consider.

This is the log after adding bufno=50, so am i safe here, or i should reduce the bufno or i can increase it ?

Please advise.

Thanks
Magesh
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> Utilities All times are GMT - 5 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


MVSFORUMS
Powered by phpBB © 2001, 2005 phpBB Group