Joined: 15 Jun 2006 Posts: 5 Topics: 2 Location: San Francisco
Posted: Mon Aug 28, 2017 2:56 pm Post subject: DFSORT FILSZ Ennn Required Accuracy
Does anyone know how wrong a DFSORT FILSZ=Ennn value can be before the sort abends? Is it a number, a percentage, something else? Are the upper and lower limits the same?
The manuals recommend a "reasonably accurate" estimate. IBM and I do not always agree on what is reasonable. I see that I can code a Unnn value which is only used for sizing, not for record counts. I may go that way.
For context, I am asking SAS to pass the results of a view to DFSORT, so the size of the input is not available to DFSORT.
Joined: 26 Nov 2002 Posts: 12359 Topics: 75 Location: San Jose
Posted: Mon Aug 28, 2017 4:31 pm Post subject: Re: DFSORT FILSZ Ennn Required Accuracy
twb wrote:
Does anyone know how wrong a DFSORT FILSZ=Ennn value can be before the sort abends? Is it a number, a percentage, something else? Are the upper and lower limits the same?
The question is why do you need to specify the FILSZ? DFSORT automatically calculates the file size and determines the right amount of storage required for the job complete. However when customers are inserting records via an E15 exit or Invoking DFSORT via a program, DFSORT will have no way of knowing an accurate record count for the file size without it being specified. Is that your case? The FILSZ parameter specifies either the exact number of records to be sorted or merged, or an estimate of the number of records to be sorted.
The abend depends on the available resources(Memory objects/hiperspaces/Sortwork disk space..) on your system when the job is running. There is no upper or lower limit
twb wrote:
The manuals recommend a "reasonably accurate" estimate. IBM and I do not always agree on what is reasonable.
Usually DFSORT requires resources about 1.1 - 1.2x of the filesize to be sorted. As for reasonable accuracy, we recommend that you give 20% for future growth of data. For if you are currently sorting 10 million records we recommend coding an estimated Filesz of 12 million for future growth.
twb wrote:
I see that I can code a Unnn value which is only used for sizing, not for record counts. I may go that way.
For context, I am asking SAS to pass the results of a view to DFSORT, so the size of the input is not available to DFSORT.
As I mentioned earlier if SAS is invoking DFSORT then it is better that it passes the FILSZ so that DFSORT can determine the right amount of storage required for the job complete. _________________ Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
Joined: 15 Jun 2006 Posts: 5 Topics: 2 Location: San Francisco
Posted: Mon Aug 28, 2017 4:56 pm Post subject:
If I coded FILSZ=Unnn, then I would expect an abend if available storage was insufficient. If I code FILSZ=Ennn, I still expect to fail if I exceed available resources, but I also expect to fail simply if my estimate is bad, for some definition of bad. For example, if I code FILSZ=E123456789 and the actual input only has 17 records, I expect the job to abend for a bad estimate (an excessive estimate in this example).
To clarify the context, a SAS view allows me to start reading a file, decide whether or not to keep a particular record, and pass any records I keep to DFSORT. I expect there are a couple of buffers involved, but SAS does not read any significant number of records before it starts passing data to DFSORT. If I were willing to pay the I/O to write all selected records to a SAS library before calling DFSORT, SAS would pass the exact record count to DFSORT.
Joined: 26 Nov 2002 Posts: 12359 Topics: 75 Location: San Jose
Posted: Mon Aug 28, 2017 5:04 pm Post subject:
twb wrote:
If I coded FILSZ=Unnn, then I would expect an abend if available storage was insufficient. If I code FILSZ=Ennn, I still expect to fail if I exceed available resources, but I also expect to fail simply if my estimate is bad, for some definition of bad. For example, if I code FILSZ=E123456789 and the actual input only has 17 records, I expect the job to abend for a bad estimate (an excessive estimate in this example).
twb,
DFSORT ignores the bad filesize estimate when it can determine the size of the file to be sorted. Here is an example of it
Just because you provided an inaccurate value for an optional parm, the job does not need to abend. DFSORT tries it use it ONLY when it canNOT determine the file size.
twb wrote:
To clarify the context, a SAS view allows me to start reading a file, decide whether or not to keep a particular record, and pass any records I keep to DFSORT. I expect there are a couple of buffers involved, but SAS does not read any significant number of records before it starts passing data to DFSORT. If I were willing to pay the I/O to write all selected records to a SAS library before calling DFSORT, SAS would pass the exact record count to DFSORT.
In case you are not sure about the number of records to be passed then make sure you have enough sort disk space. Increase the dynamic allocation from the default of 4 datasets to a higher number(may be 32) _________________ Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum