View previous topic :: View next topic |
Author |
Message |
rintu Beginner
Joined: 28 Feb 2006 Posts: 7 Topics: 3
|
Posted: Tue Feb 28, 2006 9:39 pm Post subject: Syncsort - sortin/sortout |
|
|
Hi,
We have a syncsort which has the same dataset (sequential) as sortin and sortout (both with disp=shr) and control parms as OPTION COPY,STOPAFT=12345678
recently while handling over 10 million records, the result seems to be inaccurate. Any idea whether there is a buffer limit for this type of operation |
|
Back to top |
|
|
rintu Beginner
Joined: 28 Feb 2006 Posts: 7 Topics: 3
|
Posted: Wed Mar 01, 2006 10:20 pm Post subject: |
|
|
no one seems to know the answer - I tried other forums also !!! |
|
Back to top |
|
|
kolusu Site Admin
Joined: 26 Nov 2002 Posts: 12369 Topics: 75 Location: San Jose
|
Posted: Thu Mar 02, 2006 11:20 am Post subject: |
|
|
Quote: |
We have a syncsort which has the same dataset (sequential) as sortin and sortout (both with disp=shr) and control parms as OPTION COPY,STOPAFT=12345678
|
You are using the same input as output. If for somereason your job fails you will loose the input dataset.
Quote: |
recently while handling over 10 million records,the result seems to be inaccurate. Any idea whether there is a buffer limit for this type of operation
|
What kind of inaccurate results are you getting? If your file has only 10 million records and your control cards say to stop after copying 12 million 345 thousand and 678 records, which is greater than the no: of input records , your file will have all the 10 million records.
Hope this helps...
Cheers
Kolusu _________________ Kolusu
www.linkedin.com/in/kolusu |
|
Back to top |
|
|
rintu Beginner
Joined: 28 Feb 2006 Posts: 7 Topics: 3
|
Posted: Thu Mar 02, 2006 7:57 pm Post subject: |
|
|
This is part of a commit/restart logic. The STOPAFT will have the count of committed rows. The SORTIN file will need to be cleaned up to get rid of the rows after the committed number of rows.
Inaccurate results, in the sense for eg:
SORTIN = 94543210 records (FB 596 bytes)
STOPAFT= 91543210
sortout = 5010900 (while we expect 91543210 records) |
|
Back to top |
|
|
Phantom Data Mgmt Moderator
Joined: 07 Jan 2003 Posts: 1056 Topics: 91 Location: The Blue Planet
|
Posted: Fri Mar 03, 2006 12:39 am Post subject: |
|
|
Rintu,
If this was your Sort card
Code: |
//SYSIN DD *
OPTION COPY,STOPAFT=91543210
/*
|
Then I don't think there will any problem. But if you have any other statements in your Control card, then those may result in confusion. Could you please post your complete Sort Step, so that we can see what might have gone wrong.
And one more point. I don't seem to understand the Commit/Restart logic that you were mentioning.
Code: |
This is part of a commit/restart logic. The STOPAFT will have the count of committed rows. The SORTIN file will need to be cleaned up to get rid of the rows after the committed number of rows.
|
Normally, we discard all committed records from the input file and process only those which are not committed. In that case, you need to skip Committed records - meaning using OUTFIL STARTREC=xxxxxx instead of OPTION STOPAFT. can you pls explain your process a bit.
Thanks,
Phantom |
|
Back to top |
|
|
kolusu Site Admin
Joined: 26 Nov 2002 Posts: 12369 Topics: 75 Location: San Jose
|
Posted: Fri Mar 03, 2006 9:22 am Post subject: |
|
|
Quote: |
Inaccurate results, in the sense for eg:
SORTIN = 94543210 records (FB 596 bytes)
STOPAFT= 91543210
sortout = 5010900 (while we expect 91543210 records)
|
rintu,
Going with the no: of records I can say it is a multi volume dataset and that might be the reason for inaccurate results.
Anyway this approach for checkpoint-restart logic is very bad.
Kolusu _________________ Kolusu
www.linkedin.com/in/kolusu |
|
Back to top |
|
|
rintu Beginner
Joined: 28 Feb 2006 Posts: 7 Topics: 3
|
Posted: Thu Mar 09, 2006 8:11 pm Post subject: |
|
|
Hi,
The problem was related to space and it happens only when there is large volume of data ( in millions). I know, this logic is bad. But this is the standard used throughout the shop.
The STOPAFT is used to cleanup the output files (not input files).
After much discussion, we decided to use different files for sortin and sortout. We could get hold of the author of the process and according to him, years ago when this was implemented, syncsort had okay'd this approach. But now the syncsort manual says this is a big NO-NO !!! |
|
Back to top |
|
|
|
|