MVSFORUMS.com

jathampy

Due to large number of files (150000+ files) in sftp location, my SFTP job failed failed with return code 255 while executing DIR * command. I can see "xreallocarray: out of memory" message in STDERR log. The debug1 exit status is showing as 0 in STDERR. However the STDOUT is not displaying any files.

Is there any way to fix the failure so that all files can be listed ?, I tried specified the directory listing by specifying wildcard characters to limit the file listing but no luck. The region size used is 0M

Appreciate if anyone could help on this issue

kolusu · Posted: Fri Apr 26, 2024 12:38 am Post subject:

jathampy,

You can list the files in USS directory with BPXBATCH. You can also generate the SFTP commands from that list. Let me know if you are interested in that approach.
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu

jathampy · Posted: Fri Apr 26, 2024 11:24 am Post subject:

Thanks Kolusu for the reply

Currently I am using EXEC PGM=BPXBATCH,PARM='sh sftp -vvv -b <unix file> <User@IPaddress>' to list the files

The sftp commands like CD and DIR * are stored in that unix file.
The SFTP job worked fine for 30000 files. But failed when there are large number of files (150000+ files)

Are you saying that there is another way of getting directory listing using BPXBATCH.? Yes I am interested to know

kolusu · Posted: Fri Apr 26, 2024 7:20 pm Post subject:

jathampy,

Here is an example of getting the directory listing and then you parse the output. I am just showing how the directory listing will be. You can add the parsing logic to generate the SFTP statements for each individual file.

jathampy · Posted: Sat Apr 27, 2024 1:54 pm Post subject:

Thanks Kolusu. Just couple of questions

1. The "sh ls " command you mentioned seems listing directory of mainframe unix directory. The issue I am facing is getting the directory listing of SFTP remote server and not the mainframe Unix directory. So I think I need to execute "sh sftp" first to connect to sftp server and then issue directory listing using ls command. The "cd"' and "ls" command can be stored in /u/jathampy/mydir/sftpcmd mainframe unix directory.

kolusu · Posted: Sun Apr 28, 2024 8:34 am Post subject:

jathampy,

1. Yes. You can SFTP to your remote site and then issue the CD to the desired directory and then LS command to list the files.

2. check this link which explains all the options of LS command

https://www.ibm.com/docs/en/zos/3.1.0?topic=descriptions-ls-list-file-directory-names-attributes

Pay attention to LONG FORMAT explanation.

Is your ls a Single directory or there any sub directories within it?

If Multi-directory you may want to pay attention to

jathampy · Posted: Sun Apr 28, 2024 1:24 pm Post subject:

Many Thanks kolusu. It's a Single directory where all files are placed.