******************************** Top of Data **************************************** Top of Data *********************
SARPAGE 5 DS=SYSOUT .STEP01 .PSTEP01 /0000103
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 1 - CONTROL STATEMENTS FOR XXXX-A01, Z/OS DFSORT V1R10 - 21:00 ON TUE JUN 10, 2014 -
SORT FIELDS=(1,26,CH,A) 10000
INCLUDE COND=(1,1,CH,EQ,C'R') 20000
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE751I 0 C5-K90025 C6-K90025 C7-K54603 C8-K62201 EE-K51707 E4-K51707 C9-BASE E5-K62201 E7-K62201
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE088I 1 ZZZZZH1 .STEP01 .PSTEP01 , INPUT LRECL = 612, BLKSIZE = 22528, TYPE= F
ICE093I 0 MAIN STORAGE = (MAX,67108864,67108864)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (67051504,67051504)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=67108864,MAXLIM=2097152,MINLIM=450560,EQUALS=Y,LIST=Y,ERET=RC16 ,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=8192,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW Y,DSA=64
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE084I 0 EXCP ACCESS METHOD USED FOR SORTOUT
ICE084I 1 VSAM ACCESS METHOD USED FOR SORTIN
ICE750I 0 DC 60857245116 TC 0 CS DSVVV KSZ 30 VSZ 30
ICE752I 0 FSZ=99439943 RC IGN=0 E AVG=616 0 WSP=79559723 C DYN=0 0
ICE751I 1 DE-K61785 D5-K62201 D3-BASE ED-BASE E8-K61438
ICE090I 0 OUTPUT LRECL = 612, BLKSIZE = 27540, TYPE = FB
ICE055I 0 INSERT 0, DELETE 97575885
ICE054I 0 RECORDS - IN: 99439943, OUT: 1864058
ICE134I 0 NUMBER OF BYTES SORTED: 1140803496
ICE253I 0 RECORDS SORTED - PROCESSED: 1864058, EXPECTED: 99439943
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 9000 , TRACKS USED: 0
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 1132440K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT
I could see AMP parameter, having more index buffer, may I use
AMP=('ACCBIAS=SO') for DFSORT ?
If I use option DYNALLOC=(,16),FILSZ=E120000000, will there be a improvement
Will DFSORT uses AMP parameter passed externally, or it will take its own Buffers ?
I need to use FILSZ for the number of input records or for the number of records processed ?
in our case processed record is 1864058
and input records are 99439943
I tried with FILSZ=E1864058, which is working fine, Please advice the best practice.
May I mask all the account/application information and send it to dfsort@us.ibm.com or do we need to send exact information ?
Do I need to send it from official email or from personal email ?
Because if I need to send exact info or from official email, I need to get prior approval.
Joined: 26 Nov 2002 Posts: 12378 Topics: 75 Location: San Jose
Posted: Mon Jun 23, 2014 5:41 pm Post subject:
Magesh_J wrote:
Thanks Kolusu for the kind response.
I have some questions.
Will DFSORT uses AMP parameter passed externally, or it will take its own Buffers ?
DFSORT reads the VSAM file sequentially, so BUFNI really does nothing to help in your case.
Magesh_J wrote:
I need to use FILSZ for the number of input records or for the number of records processed ?
in our case processed record is 1864058
and input records are 99439943
I tried with FILSZ=E1864058, which is working fine, Please advice the best practice.
Passing an ESTIMATED file size ONLY helps when DFSORT is unable to determine the number of records (ex: program invoked sorts..) but in this case we are able to determine the file size so we you really don't have to pass the estimated file size.
Magesh_J wrote:
May I mask all the account/application information and send it to dfsort@us.ibm.com or do we need to send exact information ?
Do I need to send it from official email or from personal email ?
Because if I need to send exact info or from official email, I need to get prior approval.
You can mask the account information if you want to. All we need is the DFSORT diagnostic messages. It doesn't matter from which email id you send. As Long as you can receive the information you are good.
Before you send the sysout to DFSORT hotline, how about you run another test with the following. Based on ICE088I message, I am guessing that your CI size is 22528 which means that we can get 33 CIs per CA. So increase the BUFND to 34 which would allow DFSORT to read a full cylinder at a time.
Joined: 26 Nov 2002 Posts: 12378 Topics: 75 Location: San Jose
Posted: Mon Jun 23, 2014 5:48 pm Post subject:
Also I noticed that your INCLUDE is only selecting records with "R" in position 1. So you can skip sorting that field as it is a constant for all records. It cuts the sorting key by a byte.
Code:
//SYSIN DD *
INCLUDE COND=(1,1,CH,EQ,C'R')
SORT FIELDS=(2,25,CH,A)
//*
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 9000 , TRACKS USED: 0
Now looking at the LISTCAT, you are "sorting" on the VSAM key. This is unnecessary, as a VSAM KSDS is always in key order when read sequentially.
Make it a COPY instead of an actual SORT.
You have specified an average record-length of 80, and a maximum of 612 and are treating the records as F. That implies a lot of padding of the short records (if the average is correct).
You have Shareoptions 2,3 and the number of records has changed between the two runs you have shown. Is the file open at the time the SORT step is run?
Hi William
Its a online file.. yes the file was opened.
Key length 1,27 but sort required for 1,26 or as per kolusu advice 2,25
I tried with copy, output record sequence mismatched.
I will run with a closed file and let u know the results.
You key is indeed 27 bytes in length, which means that it is "sorted" already on any length of key starting from position one (although duplicates are certain/potential depending on the length from position one).
You have a flat file, where can it contain about sequence mismatched? Is it going into another KSDS later? Are you sure you don't have duplicate keys when considering only the first 26 of they key?
I wonder why you think "I have a dfsort step taking 1hr 30 minutes" as the CPU time is less than 6 mins. Is it the elapsed time that you are talking about?
Magesh_J wrote:
Hi William
Its a online file.. yes the file was opened.
Key length 1,27 but sort required for 1,26 or as per kolusu advice 2,25
If your first byte is "R" it doesn't need to be sorted. And you did NOT show us the results from the run with SORT FIELDS=(2,25,CH,A) . But if your key is 27 bytes why are you only sorting on the first 26 bytes?
William Collins wrote:
This was interesting from your first post:
Code:
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 9000 , TRACKS USED: 0
Check the message
Code:
ICE180I 0 HIPERSPACE STORAGE USED = 1135092K BYTES
Well DFSORT was able to get enough HIPERSPACE so it never used the sortwk datasets _________________ Kolusu
www.linkedin.com/in/kolusu
Yes there is a good improvement in compared with the existing code, thank you very much for the kind advice.
Quote:
I wonder why you think "I have a dfsort step taking 1hr 30 minutes" as the CPU time is less than 6 mins. Is it the elapsed time that you are talking about?
Yes elapsed time, I think space manager is restricting the allocation of work dataset and CPU availability.
Quote:
If your first byte is "R" it doesn't need to be sorted. And you did NOT show us the results from the run with SORT FIELDS=(2,25,CH,A) . But if your key is 27 bytes why are you only sorting on the first 26 bytes?
To be honest I don't know why developer have this sort criteria, I couldn't see any impact technically down the line.
But when I change the code to
option copy or sort field = copy, and compared against sorted output
the record mismatch in 386th position.
When I have option copy or sort fields=copy the output flat which looks incorrect
All the records matches, i.e all the other columns matches except the 386th position.
Looks really strange, don't know why this '.....YY....' is being copied on other records.
Joined: 26 Nov 2002 Posts: 12378 Topics: 75 Location: San Jose
Posted: Tue Jun 24, 2014 12:22 pm Post subject:
Magesh_J wrote:
But when I change the code to
option copy or sort field = copy, and compared against sorted output
the record mismatch in 386th position.
When I have option copy or sort fields=copy the output flat which looks incorrect
The residual data is because of short records. When your VSAM file is read as
DFSORT Application Programming Guide wrote:
Fixed-length (F). The records are processed without an RDW, so the data starts in position 1. Control statement positions should be specified accordingly.
An RRDS can always be processed as fixed-length. A KSDS, ESDS or VRRDS used for input should only be processed as fixed-length if all of its records have a length equal to the maximum record size defined for the cluster. Otherwise, input records which are shorter than the maximum record size are padded with bytes that may or may not be zeros (that is, "garbage" bytes).
Changed the code as per your advice, It works like a charm, it consume little more CPU because of this conversion, but never mind, because existing code has a bug and our code doesn't use workspace so it will not wait for workspace allocation, hence elapsed time will get reduced during the batch hours.
Thank you very much for all the advice.
Code:
************************************************************************************************************************************
* *
* JOB STEP TERMINATION REPORTING *
* HH.MM.SS.TH MMM.THT *
* STEPNAME PRIV AREA SIZE 6144K PAGES SWAP IN 0 STEP VECTOR TIME 00.00.00.00 *
* PROCSTEP RSLTSORT PRIV USED TOP 300K PAGES SWAP OUT 0 INIT VECTOR TIME 00.00.00.00 *
* PGM NAME SORT PRIV USED BOT 2132K REREADS 0 CPU TCB TIME 00.06.18.27 6.304 *
* COMP CODE 0000 STEP SERVICE 7,780,571 TAPE MOUNTS 0 CPU SRB TIME 00.00.00.96 0.016 *
* MEMLIMIT SIZE 200MB CPU TIME 348.710 *
* *
* UNIT-T-DDNAME------EXCP CNT---MB TRANSFER UNIT-T-DDNAME------EXCP CNT---MB TRANSFER UNIT-T-DDNAME------EXCP CNT---MB TRANSFER *
* 4844 SORTIN 18,955 293.347 4E2B SORTIN 11,148 172.526 3870 SORTIN 6,795 105.159 *
* 387E SORTIN 5,324 82.394 3E87 SORTIN 12,128 187.692 3A86 SORTIN 7,768 120.217 *
* 4847 SORTIN 11,116 172.031 3841 SORTIN 10,612 164.231 3877 SORTIN 3,923 60.712 *
* 4848 SORTOUT 10,381 285.892 *
* *
* EXCP TOTAL 98,150 TOTAL DASD MB TRANSFERS 1,644.201 TOTAL TAPE MB TRANSFERS 0.000 *
* *
************************************************************************************************************************************
IEF373I STEP/RSLTSORT/START 2014175.1048
IEF374I STEP/RSLTSORT/STOP 2014175.1057 CPU 6MIN 18.27SEC SRB 0MIN 00.96SEC VIRT 2132K SYS 300K EXT 6880K SYS 12052K
************************************************************************************************************************************
* *
* JOB END TERMINATION REPORTING *
* HH.MM.SS.TH MMM.THT *
* JOBNAME ZZZZZZZZ DASD MB XFERS 1,644.201 CPU SERVICE 7,711,860 JOB VECTOR TIME 00.00.00.00 *
* SYSID XXX TAPE MB XFERS 0.000 SRB SERVICE 19,568 INIT VECTOR TIME 00.00.00.00 *
* SRVCLAS TSTBATMD TAPE MOUNTS 0 MSO SERVICE 0 CPU TCB TIME 00.06.18.27 6.304 *
* CLASS S I/O SERVICE 49,143 CPU SRB TIME 00.00.00.96 0.016 *
* TOT SERVICE 7,780,571 CPU TIME 348.710 *
* *
************************************************************************************************************************************
IEF375I JOB/ZZZZZZZZ/START 2014175.1048
IEF376I JOB/ZZZZZZZZ/STOP 2014175.1057 CPU 6MIN 18.27SEC SRB 0MIN 00.96SEC
Code:
1ICE805I 1 JOBNAME: ZZZZZTP , STEPNAME: RSLTSORT
ICE802I 0 BLOCKSET TECHNIQUE IN CONTROL
ICE201I H RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE858I 0 LA=409060, DA=6144, AA=402916, BA=216, CP=1, TA=265
ICE859I 0 LB=6052, DB=2064, AB=3988, BB=0, CP=0, RS=0, TB=20
ICE860I 0 F=YN, P=2, M=N, B=256
ICE751I 0 C5-K90025 C6-K90025 C7-K54603 C8-K62201 EE-K51707 E9-K60823 C9-BASE E5-K62201 E7-K62201
ICE143I 0 BLOCKSET COPY TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 1 - CONTROL STATEMENTS FOR XXXX-A01, Z/OS DFSORT V1R10 - 10:48 ON TUE JUN 24, 2014 -
0 OPTION COPY
INCLUDE COND=(5,1,CH,EQ,C'R')
RECORD TYPE=V
OUTFIL VTOF,BUILD=(5,612)
ICE201I H RECORD TYPE IS V - DATA STARTS IN POSITION 5
ICE858I 0 LA=409060, DA=6144, AA=402916, BA=216, CP=1, TA=265
ICE859I 0 LB=6052, DB=2064, AB=3988, BB=0, CP=0, RS=0, TB=20
ICE860I 0 F=YN, P=2, M=N, B=256
ICE751I 0 C5-K90025 C6-K90025 C7-K54603 C8-K62201 EE-K51707 E9-K60823 C9-BASE E5-K62201 E7-K62201
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE088I 0 ZZZZZTP.RSLTSORT. , INPUT LRECL = 616, BLKSIZE = 22528, TYPE = V
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=2097152,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=8192,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=MAX ,ODMAXBF=262144,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=MAX
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE084I 1 VSAM ACCESS METHOD USED FOR SORTIN
ICE889I 0 CT=MAX , SB=3, L=0, D=0000, CCW=1MAM
ICE902I 0 O I NP10
ICE231I 0 STORAGE USED FOR OUTFIL : BELOW 16M = 20480, ABOVE 16M = 271360
ICE855I 0 SORTOUT : TX=N, R=J, L=J, B=D, BL=0, BR=0, DCT=4, VS=N, RU=X, SB=8
ICE210I 0 SORTOUT : EXCP USED, LRECL = 612, BLKSIZE = 27540, TYPE = FB
ICE751I 1 EF-K49535 CB-K64631 F0-K66716 E8-K61438
ICE900I 0 CON=1,MUV=1,VOL=1,ENU=0,SBK=0,SRC=0,VEM=0
ICE055I 0 INSERT 0, DELETE 97839942
ICE054I 0 RECORDS - IN: 99708456, OUT: 1868514
ICE227I 0 SORTOUT : DELETED = 0, REPORT = 0, DATA = 1868514
ICE228I 0 SORTOUT : TOTAL IN = 1868514, TOTAL OUT = 1868514
ICE804I 5 OUTFIL EXCP COUNT: 10381
ICE891I 1 6283792 WMAIN, 7664 CMAIN, MAX CALLOC, N SCN, B BA, 0 AZ, 0 BZ, NY QC, 0 CZ, 0 DZ, 1 PLE
ICE892I 1 616 RIN 620 BLI 620 BLO 613 RUN 0 BUN 2818 CPU 00 CVC
ICE052I 0 END OF DFSORT
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum