The Ultimate Solution to Simplify Your Data Center
SFTP
Overview
SFTP (Secure FTP) is a File Transfer Protocol (FTP) that allows data to be transferred securely. On z/OS, SFTP is a utility that runs under the Secure Shell (OpenSSH) of Unix System Services (USS). OpenSSH is a secure connection between a client and a server. All SFTP files are transferred to and from a UNIX directory. An additional step is required to copy the file between the UNIX directory and a z/OS data set.
There are 4 methods to transfer data using SFTP:
- A push (PUT) from NWRDC to a remote server.
- A pull (GET) to NWRDC from a remote server.
- A push (PUT) from a remote server to NWRDC.
- A pull (GET) to a remote server from NWRDC.
Setup
Before SFTP can be performed the customer’s ACF2 id must be updated to allow access to USS. This will be performed by NWRDC.
SFTP can be performed in batch mode, interactive mode or a combination of both. For batch mode OpenSSH needs to be set up to exchange keys between servers for authentication without a password prompt (Public Key Authentication). This is accomplished by creating a SSH public/private key pair without a pass phrase on the host server and copying the public key to the remote server. If required, the public key must be converted to the remote server’s encoding scheme(EBCDIC/ASCII). Specific file names, directories and authorities are required depending on the version of SSH on the remote server. The following explains how to set up Public Key Authentication on a remoter server that uses OpenSSH. Other version of SSH may vary.
-
On the host user’s home directory create a folder named .ssh and give it permissions of 700. Be sure the owner is the user’s id.
mkdir .ssh
chmod 700 .ssh
chown userid .ssh -
Generate a SSH public/private key pair by issuing the ssh-keygen command and place it in the .ssh directory. Press enter when asked for the pass phrase.
ssh-keygen –t rsa –f ~/.ssh/id_rsa -
Convert the public key from EBCDIC to ASCII if required.
iconv -t IBM-1047 -f ISO8859-1 id_rsa.pub > id_rsa.pub.ebcdic ; -
On the remote server create an .ssh folder, set the permissions to 700 and copy the public key generated in #2 (or #3) to a file named authorized_keys in the newly created .ssh folder.
cat ~/.ssh/id_rsa.pub | ssh userid@remote_server ‘mkdir ~/.ssh;chmod 700
~/.ssh;cat >> ~/.ssh/authorized_keys;chmod 644 ~/.ssh/authorized_keys’Another option is to sftp the public key to the sftp server.
cd .ssh
sftp userid@remote_server
mkdir .ssh
chmod 700 .ssh
cd .ssh
ascii
put –p id_rsa.pub authorized_keys
- Entering the password is required for this first SSH session on the remote server.
- Userid is optional but is required if the user id is different between the host and remote servers.
-
The –p option on the put statement retains file permissions
- On the host server the user’s home directory must have a known_hosts file. This file must contain the public key from the sftp server. This file is created (if it doesn’t exist) and populated when you ssh to a server for the first time. You will be prompted the first time to add the public key to the known_hosts file.
- Make sure the permissions are 700 on the .ssh folder on both the host and remote server and that the owner is the userid being used.
- Make sure the permissions are 755 on the customer’s home directory on both the host and remote server and that the owner is the user id.
- Make sure the permissions are 644 on id_rsa.pub on the host server and on authorized_keys on the remote server.
Steps
Below are the steps for each of the 4 methods of transferring data via SFTP.
-
Push from NWRDC.
a. Copy the z/OS file to a temporary file in a UNIX folder.
b. SFTP the UNIX file to the remote server with the ASCII option.
c. Delete the temporary UNIX file. -
Pull to NWRDC
a. SFTP the remote file to a temporary file in a UNIX folder with the ASCII option.
b. Copy the temporary file to a z/OS dataset.
c. Delete the temporary UNIX file. -
Push from a remote server.
a. SFTP the file from the remote server to a UNIX folder at NWRDC.
b. On NWRDC copy the UNIX file to a z/OS dataset.
c. Delete the UNIX file. -
Pull from a remote server.
a. Copy the z/OS file to a temporary file in a UNIX folder.
b. SFTP the temporary UNIX file to the remote server.
c. Delete the temporary file.
Examples
Below are examples of SFTP initiated from NWRDC.
- Push.
Interactive example:
This example is using PuTTY's ssh client connected to NWRDC
$ sftp remote_id@remote.server
Connecting to remote.server...
sftp> !cp "//'z/OS-dataset'" temp-Unix-file
sftp> ascii
Sets the file transfer type to ASCII.
sftp> put temp-Unix-file remote-server-file
Uploading temp-\Unix-file to /home/remote_id/remote-server-file
temp-Unix-file 100% 34KB 33.8KB/s 00:00
sftp> !rm temp-Unix-file
sftp>
Batch example:
Valid Jobcard
/*JOBPARM L=99999
//**********************************************************************
//* Place SFTP commands into a file named SFTPCMDS in the home directory
//* The “!” is an escape character for executing Unix commands
//**********************************************************************
//*
//XFER EXEC PGM=BPXCOPY,
// PARM='ELEMENT(SFTPCMDS)'
//SYSUT1 DD *
!cp "//'z/OS-dataset'" temp-Unix-file
ascii
put temp-Unix-file remote-server-file
!rm temp-Unix-file
quit
//SYSUT2 DD PATH='your-nwrdc-home-directory'
//SYSTSPRT DD SYSOUT=*
//**********************************************************************
//* Run SFTP in Batch mode using the commands placed in
//* your-nwrdc-home-directory/SFTPCMDS
//**********************************************************************
//*
//UNPAXDIR EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDPARM DD *
sh
sftp -b your-nwrdc-home-directory/SFTPCMDS remote_id@remote.server
/*
//
- Pull to NWRDC
Interactive example:
This example is using PuTTY's ssh client connected to NWRDC
$ sftp remote_id@remote.server
Connecting to remote.server...
sftp> ascii
Sets the file transfer type to ASCII.
sftp> get remote-server-file temp-Unix-file
Fetching /remote-server-file to temp-Unix-file
/remote-server-file 100% 34KB 33.8KB/s 00:00
sftp> !cp temp-Unix-file "//'z/OS-dataset'"
sftp> !rm temp-Unix-file
sftp>
Batch example:
Valid Jobcard
/*JOBPARM L=99999
//**********************************************************************
//* Place SFTP commands into a file named TEST in the home directory
//* The “!” is an escape character for executing Unix commands
//**********************************************************************
//*
//XFER EXEC PGM=BPXCOPY,
// PARM='ELEMENT(SFTPCMDS)'
//SYSUT1 DD *
ascii
get remote-server-file temp-Unix-file
!cp temp-Unix-file "//'z/OS-dataset'"
!rm temp-Unix-file
!cp "//'NWR.AAANOTES'" temp.txt
quit
//SYSUT2 DD PATH='your-nwrdc-home-directory'
//SYSTSPRT DD SYSOUT=*
//**********************************************************************
//* Run SFTP in Batch mode using the commands placed in
//* your-nwrdc-home-directory/SFTPCMDS
//**********************************************************************
//*
//UNPAXDIR EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//STDPARM DD *
sh
sftp -b your-nwrdc-home-directory/SFTPCMDS remote_id@remote.server
/*
//
Below are examples of SFTP initiated by a remote server.
- Push from a remote server
SFTP the file from the remote server to the z/OS Unix directory. The following is an example
using PuTTY’s SFTP. Bold print is what is entered, italic print is the response from the system.
psftp> open userid@nwrdc.fsu.edu
Using username "userid".
userid@nwrdc.fsu.edu's password: password (password not displayed)
Remote working directory is /u/userid
psftp> put remote\server\path\sftp.file /unix/path/sftped.file
local:remote\server\path\sftp.file => remote:/unix/path/sftp.file
psftp>
Next, the Unix file is copied to a MVS dataset using a batch Job.
//valid job card
//**********************************************************************
//* This job copies a file that is SFTPed from a remote server to a
//* MVS dataset.
//*
//**********************************************************************
//* Instructions
//* 1. Add a valid job card.
//* 2. Change mvs.file to the New Mainframe file name. Leave the
//* quotes as is.
//* 3. Change sftp.file to the name of the file that was SFTPed. The
//* file name is case sensitive.
//* 4. Change /unix/path to the UNIX directory where the SFTPed file
//* is located. The path is case sensitive.
//*
//**********************************************************************
//* Allocate receiving file with the correct format if not allocated
//**********************************************************************
//*
//STEP01 EXEC PGM=IEFBR14
//SYSUT2 DD DSN=mvs.file,
// DISP=(MOD,CATLG,DELETE),
// MGMTCLAS=MCTEMP,
// UNIT=SYSDA,SPACE=(CYL,(pri,sec),RLSE),
// DCB=(RECFM=recfm,LRECL=lrecl,BLKSIZE=blksize)
//**********************************************************************
//* Copy the transferred file to the z/OS dataset name
//* The steps performed are:
//* 1. Set the processing to end when a command fails.
//* 2. Convert the file from ASCII to EBCDIC.
//* 3. Copy the converted file to a MVS dataset with the Format
//* carriage return new line option. (Windows uses two characters
//* for new lines (CRLF) while z/OS only uses one character. Note
//* that crnl is used instead of crlf. z/OS considers the line
//* feed character a new line).
//* 4. Delete the temporary file.
//**********************************************************************
//*
//STEP02 EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//STDPRINT DD SYSOUT=*
//STDPARM DD *
sh
set -o errexit;
iconv -t IBM-1047 -f ISO8859-1
/unix/path/sftp.file > /unix/path/sftp.file.ebcdic ;
cp –F crnl /unix/path/sftp.file.ebcdic "//'mvs.file'";
rm /unix/path/sftped.file.ebcdic;
/*
//
Optionally, instead of running the IEFBR14 step to allocate a new MVS dataset with the correct
format, the –W option of the cp command can be used. See the modified cp command below:
cp –F crnl
-W "seqparms='RECFM=recfm,LRECL=lrecl,SPACE=(CYL,(pri,sec)),BLKSIZE=blksize'"
/unix/path/sftped.file.tr "//'mvs.file'";
- Pull from a remote server
Copy the MVS dataset to the z/OS Unix Directory using a batch job.
//valid job card
//**********************************************************************
//* This job copies a MVS dataset to a file in z/OS Unix that will
//* later be SFTPed to a Windows machine.
//**********************************************************************
//* Instructions
//* 1. Add a valid job card.
//* 2. Change mvs.file to the Mainframe file name. Leave the quotes
//* as is.
//* 3. Change sftped.file to the name of the file that was SFTPed. The
//* file name is case sensitive.
//* 4. Change /unix/path to the UNIX directory where the SFTPed file
//* is located. The path is case sensitive.
//*
//**********************************************************************
//* Copy the z/OS dataset to the Unix folder and convert to ASCII.
//* The steps performed are:
//* 1. Copy the MVS dataset to the UNIX folder with the crnl format
//* option. This options adds a carriage return to each line for
//* Windows. ICONV does not change UNIX NL to ASCII CRLF.
//* (Windows uses two characters for new lines (CRLF) while UNIX
//* only uses one character for new lines (NL)). Note that crnl is
//* used instead of crlf. z/OS considers the line feed character a new
//* line).
//* 3. Convert the file from EBCDIC to ASCII using ICONV.
//**********************************************************************
//*
//STEP02 EXEC PGM=BPXBATCH
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//STDPRINT DD SYSOUT=*
//STDPARM DD *
sh
cp -F crnl "//'mvs.file'" /unix/path/sftped.file.ebcdic;
iconv -t ISO8859-1 -f IBM-1047
/unix/path/sftped.file.ebcdic > /unix/path/sftped.file.ascii ;
rm /unix/path/sftped.file.ebcdic;
//
Next, SFTP the UNIX file to the remote server. The following is an example using PuTTY’s SFTP.
Bold print is what is entered, italic print is the response from the system.
psftp> open userid@nwrdc.fsu.edu
Using username "userid".
userid@nwrdc.fsu.edu's password: password (password not displayed)
Remote working directory is /u/userid
psftp> get /u/sftptest/sftped.file.ascii c:\Users\reese_harrington\Desktop\sftped.file.txt
remote:/u/sftptest/sftped.file.ascii => local:c:\Users\reese_harrington\Desktop\sftped.file.txt
psftp>