MVSFORUMS.com Forum Index MVSFORUMS.com
A Community of and for MVS Professionals
 
 FAQFAQ   SearchSearch   Quick Manuals   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

SFTP to list large number of files

 
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> TSO and ISPF
View previous topic :: View next topic  
Author Message
jathampy
Beginner


Joined: 21 Dec 2007
Posts: 15
Topics: 6
Location: UK

PostPosted: Wed Apr 24, 2024 1:46 pm    Post subject: SFTP to list large number of files Reply with quote

Due to large number of files (150000+ files) in sftp location, my SFTP job failed failed with return code 255 while executing DIR * command. I can see "xreallocarray: out of memory" message in STDERR log. The debug1 exit status is showing as 0 in STDERR. However the STDOUT is not displaying any files.

Is there any way to fix the failure so that all files can be listed ?, I tried specified the directory listing by specifying wildcard characters to limit the file listing but no luck. The region size used is 0M

Appreciate if anyone could help on this issue
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12369
Topics: 75
Location: San Jose

PostPosted: Fri Apr 26, 2024 12:38 am    Post subject: Reply with quote

jathampy,

You can list the files in USS directory with BPXBATCH. You can also generate the SFTP commands from that list. Let me know if you are interested in that approach.
_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jathampy
Beginner


Joined: 21 Dec 2007
Posts: 15
Topics: 6
Location: UK

PostPosted: Fri Apr 26, 2024 11:24 am    Post subject: Reply with quote

Thanks Kolusu for the reply

Currently I am using EXEC PGM=BPXBATCH,PARM='sh sftp -vvv -b <unix file> <User@IPaddress>' to list the files

The sftp commands like CD and DIR * are stored in that unix file.
The SFTP job worked fine for 30000 files. But failed when there are large number of files (150000+ files)

Are you saying that there is another way of getting directory listing using BPXBATCH.? Yes I am interested to know
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12369
Topics: 75
Location: San Jose

PostPosted: Fri Apr 26, 2024 7:20 pm    Post subject: Reply with quote

jathampy,

Here is an example of getting the directory listing and then you parse the output. I am just showing how the directory listing will be. You can add the parsing logic to generate the SFTP statements for each individual file.

Code:

//***************************************************************
//* Get the directory listing of the uss directory              *
//***************************************************************
//LISTDIR  EXEC PGM=BPXBATCH                   
//STDERR   DD SYSOUT=*                                           
//STDOUT   DD DSN=&&UDL,DISP=(,PASS),SPACE=(CYL,(5,5),RLSE),     
//            DCB=(LRECL=256,RECFM=FB,BLKSIZE=0,DSORG=PS)       
//STDPARM  DD *                                 
sh ls -lap '/u/jathampy/mydir'
/*                                                               
//***************************************************************
//* Parse the STDOUT from the above step to get the file names  *
//***************************************************************
//GETLSI   EXEC PGM=SORT                                         
//SYSOUT   DD SYSOUT=*                                           
//SORTIN   DD DISP=(OLD,DELETE),DSN=&&UDL                         
//SORTOUT  DD SYSOUT=*                                           
//SYSIN    DD *                                               
  OPTION COPY                                                     
/*                                                               


You can add the following to filter just file names
Code:

INCLUDE COND=(01,01,CH,EQ,C'-')

_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jathampy
Beginner


Joined: 21 Dec 2007
Posts: 15
Topics: 6
Location: UK

PostPosted: Sat Apr 27, 2024 1:54 pm    Post subject: Reply with quote

Thanks Kolusu. Just couple of questions

1. The "sh ls " command you mentioned seems listing directory of mainframe unix directory. The issue I am facing is getting the directory listing of SFTP remote server and not the mainframe Unix directory. So I think I need to execute "sh sftp" first to connect to sftp server and then issue directory listing using ls command. The "cd"' and "ls" command can be stored in /u/jathampy/mydir/sftpcmd mainframe unix directory.

Code:
//STDPARM  DD *                                 
sh sftp -vvv -b /u/jathampy/mydir/sftpcmd sftpuserid@ipaddress
//*


Please clarify if this is something you are looking for

2. i would like to know what "ls -lap" means. "ls - la" indicates long listing format that include hidden files. not sure what "p" means .
Back to top
View user's profile Send private message
kolusu
Site Admin
Site Admin


Joined: 26 Nov 2002
Posts: 12369
Topics: 75
Location: San Jose

PostPosted: Sun Apr 28, 2024 8:34 am    Post subject: Reply with quote

jathampy,

1. Yes. You can SFTP to your remote site and then issue the CD to the desired directory and then LS command to list the files.

2. check this link which explains all the options of LS command

https://www.ibm.com/docs/en/zos/3.1.0?topic=descriptions-ls-list-file-directory-names-attributes

Pay attention to LONG FORMAT explanation.

Is your ls a Single directory or there any sub directories within it?

If Multi-directory you may want to pay attention to

Quote:

When issuing the ls command against a large directory structure, the following message might be returned:


Code:
FSUM6786  too many directory entries in "dir"


To alleviate this problem, set the
Code:
_CEE_RUNOPTS="HEAP(,,,FREE)"
environment variable before invoking the ls command. Language Environment will free all unused storage to avoid exhausting the user heap. For more information about heap tuning, see Tuning heap storage in z/OS Language Environment Programming Guide.

_________________
Kolusu - DFSORT Development Team (IBM)
DFSORT is on the Web at:
www.ibm.com/storage/dfsort

www.linkedin.com/in/kolusu
Back to top
View user's profile Send private message Send e-mail Visit poster's website
jathampy
Beginner


Joined: 21 Dec 2007
Posts: 15
Topics: 6
Location: UK

PostPosted: Sun Apr 28, 2024 1:24 pm    Post subject: Reply with quote

Many Thanks kolusu. It's a Single directory where all files are placed.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic   printer-friendly view    MVSFORUMS.com Forum Index -> TSO and ISPF All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


MVSFORUMS
Powered by phpBB © 2001, 2005 phpBB Group