Aspera Enterprise Server Admin Guide 3.7.3

207 Pages·2017·1.96 MB·English

Enterprise Server Admin Guide
AIX
Revision:1222 Generated:03/29/2019 12:17
| Contents | ii
Contents
Introduction...............................................................................................................5
Standard Installation................................................................................................5
Requirements.........................................................................................................................................................5
Before Upgrading or Downgrading......................................................................................................................6
Product Setup........................................................................................................................................................7
Upgrade Follow up...................................................................................................................................8
Configuring the Firewall......................................................................................................................................8
Securing Your SSH Server...................................................................................................................................9
Changing and Securing the TCP Port......................................................................................................9
Restricting User Access..........................................................................................................................11
Configuring Transfer Server Authentication..........................................................................................13
Testing a Locally Initiated Transfer...................................................................................................................14
Updating the Product License............................................................................................................................15
Uninstalling.........................................................................................................................................................15
Managing Users from the Command Line..........................................................15
Setting Up Transfer Users..................................................................................................................................15
Setting Up Transfer Groups...............................................................................................................................18
Configuration Precedence...................................................................................................................................19
Setting Up a User's Public Key on the Server...................................................................................................21
Testing a User-Initiated Remote Transfer..........................................................................................................21
Managing Global Transfer Settings from the Command Line.......................... 22
Global Bandwidth Settings (Command Line)....................................................................................................22
Setting Up Virtual Links (Command Line).......................................................................................................23
Configuring the Transfer Server (Command Line)............................................................................................26
General Configuration Reference.........................................................................27
Configuration Methods.......................................................................................................................................27
aspera.conf - Authorization................................................................................................................................27
aspera.conf - Transfer.........................................................................................................................................29
aspera.conf - File System...................................................................................................................................41
aspera.conf - Filters to Include and Exclude Files............................................................................................45
aspera.conf - Server-Side Encryption at Rest (EAR)........................................................................................48
Reporting Checksums.........................................................................................................................................50
Overview of Inline File Validation....................................................................................................................52
Inline File Validation with URI..........................................................................................................................53
Inline File Validation with Lua Script...............................................................................................................56
ascp: Transferring from the Command Line.......................................................58
Ascp Command Reference.................................................................................................................................58
Ascp General Examples......................................................................................................................................70
Ascp File Manipulation Examples.....................................................................................................................72
| Contents | iii
Ascp Transfers with Object Storage and HDFS................................................................................................73
Configuring Multi-Session Transfers.................................................................................................................76
Applying Filters to Include and Exclude Files..................................................................................................85
Creating SSH Keys.............................................................................................................................................90
Ascp FAQs..........................................................................................................................................................91
Pre- and Post-Processing (Prepost).......................................................................93
Setting Up Pre/Post Processing..........................................................................................................................94
Pre/Post Variables...............................................................................................................................................95
Pre/Post Examples..............................................................................................................................................97
Setting Up Email Notification............................................................................................................................98
Email Notification Examples............................................................................................................................100
Managing the Node API...................................................................................... 102
Overview: Aspera Node API............................................................................................................................102
Node API Setup................................................................................................................................................102
Node Admin Tool.............................................................................................................................................103
Configuring the Server for the Node API........................................................................................................105
Redis DB Backup/Restore................................................................................................................................108
Setting up SSL for your Nodes........................................................................................................................109
Understanding the Watchfolder Architecture................................................... 111
Aspera Watch Service and Aspera Watchfolders.............................................................................................111
The Aspera Watch Services Manager ( )...................................................................................112
asperarund
Migrating Watch Services on Upgrade............................................................................................................113
Working With Aspera Watch Service................................................................ 114
Starting the Aspera Watch Service...................................................................................................................114
asperawatchd Configuration.............................................................................................................................115
Transferring and Deleting Files with the Aspera Watch Service.....................................................................117
Working With Aspera Watchfolders..................................................................119
Creating a Watch Folder...................................................................................................................................119
Configuring Watchfolder Services........................................................................................................119
Creating a Watch Folder with the Utility.................................................120
aswatchfolderadmin
Managing Watch Folders with aswatchfolderadmin........................................................................................122
Configuring the Watch Folder JSON Configuration File................................................................................123
Watchfolder Configuration in ................................................................................................135
aspera.conf
Choosing User Accounts to Run Watchfolder Services...................................................................................137
Creating a Watch Folder with the Node API...................................................................................................138
Configuring Watchfolder Services........................................................................................................138
Configuring the Node API...................................................................................................................139
Creating a Watch Folder with the Node API.......................................................................................139
Managing Watch Folders with the Node API..................................................................................................141
Updating the Docroot of a Running asperawatchfolderd Service....................................................................144
Managing Watch Folder Services.....................................................................................................................145
Configuring for Other Aspera Products............................................................145
Configuring for Aspera Files............................................................................................................................145
Set Up and Configure a Transfer User................................................................................................145
Configure the Server as a Files Node..................................................................................................147
Configure Server Certificates...............................................................................................................148
Adding the Node to Files.....................................................................................................................148
Authentication and Authorization......................................................................148
Installing SSL Certificates................................................................................................................................148
Setting Up Token Authorization.......................................................................................................................151
Token Generation (Node API).........................................................................................................................152
Token Generation (astokengen)........................................................................................................................155
Configuring Token Authorization in aspera.conf.............................................................................................157
Access Key Authentication..............................................................................................................................158
Asconfigurator Reference.................................................................................... 165
The asconfigurator Utility................................................................................................................................165
Syntax and Usage.............................................................................................................................................165
Examples...........................................................................................................................................................167
Reading Output.................................................................................................................................................168
User, Group and Default Configurations.........................................................................................................169
Trunk (Vlink) Configurations...........................................................................................................................175
Central Server Configurations..........................................................................................................................175
HTTP Server Configurations............................................................................................................................176
Database Configurations...................................................................................................................................178
Server Configurations.......................................................................................................................................179
Client Configurations........................................................................................................................................183
Troubleshooting.....................................................................................................183
Clients Can't Establish Connection..................................................................................................................183
Error: Session Timeout During Ascp Transfers...............................................................................................185
Logs Overwritten Before Transfer Completes.................................................................................................186
Disabling SELinux............................................................................................................................................186
Appendix................................................................................................................186
Restarting Aspera Services...............................................................................................................................186
Testing and Optimizing Transfer Performance................................................................................................187
Log Files...........................................................................................................................................................188
Product Limitations...........................................................................................................................................188
Technical Support.................................................................................................188
Legal Notice...........................................................................................................189
| Introduction | 5
Introduction
IBM® Aspera® Enterprise Server is a universal file transfer server that uses FASP® transfer technology to maximize
upload and download speeds. Enterprise Server includes the following features:
Feature Description
FASP transport technology The file transfer protocol that maximizes transfer speeds over IP networks by
eliminating the fundamental bottlenecks in conventional technologies. FASP features
bandwidth control, resume, transfer encryption, content protection, and data integrity
validation.
Transfer server Allows an unlimited number of concurrent client transfers. Uses virtual links to
manage aggregate bandwidth usage.
Pre- and Post-Processing Execute customizable actions when transfer events, such as the start and end of
(Prepost) transfer sessions, occur. An email notification script is included.
Ascp The command-line file transfer program.
Aspera Watch Service A service that detects file system change and manages snapshots, optimized for speed,
scale, and distributed sources. It discovers changes (new files and directories, deleted
items, and renames) in source file systems immediately as they occur, eliminating the
need to scan the file system.
More features are available with additional licenses:
Feature Description
Aspera Sync A software application that provides high-speed and highly scalable, multi-directional,
file-based replication and synchronization.
Watchfolders A service to automate large-scale file and directory transfers, with support for ultra-
large (10 million-item) directories and "growing file" sources.
Standard Installation
Requirements
System requirements for IBM Aspera Enterprise Server:
• Product-specific Aspera license file.
• AIX 6.1 or 7.1
• SSH Server. Version 5.2 or higher is recommended.
If you plan to set up and use the Node API, you must also meet the following requirements on each node machine:
• Identify a directory that you plan to use for sharing data. When you set up the Aspera transfer user (Node API
Setup on page 102), you set their absolute path (docroot) to this directory.
• The line must appear in the file. See
127.0.0.1 localhost hosts
/etc/hosts
| Standard Installation | 6
• For UNIX-based nodes, SELinux must be set to "permissive" or "disabled", not "enforced". To check the status of
SELinux, run the following:
# sestatus
If SELinux is set to "enforced", see Disabling SELinux on page 186.
Before Upgrading or Downgrading
Upgrading
• The Aspera Enterprise Server installer automatically checks for an older version of the product on your system. If
an older version is found, the installer automatically removes it before installing the new version.
• Although the installer performs your upgrade automatically, Aspera highly recommends completing the tasks
below before upgrading. If you do not follow these steps, you risk installation errors or losing your former
configuration settings.
• You cannot upgrade directly between different Aspera transfer products (such as from Point-to-Point to Desktop
Client, or from Point-to-Point to Enterprise Server). To upgrade, you need to back up the configuration, uninstall
the product, and perform a fresh install of the new version of the product.
Downgrading
Older installers do not check for newer versions of the application. You must prepare your machine as described
below then uninstall the newer version before continuing with your downgrade.
Newer versions of the Redis database are not compatible with older versions of the application. Your downgrade
process depends on whether a backup of the older Redis DB is available, either as a separate backup file or as part of
your backup of the directory from the older version.
var
• With a backup: Follow the steps below to prepare your machine. Uninstall the application (for instructions, see
Uninstalling on page 15). Copy the older Redis DB file into the directory before installing the older
var
(downgrade) version.
/opt/aspera/var/
• Without a backup: Follow the steps below to prepare your machine. Uninstall the application (for instructions,
see Uninstalling on page 15) and delete the and directories from your machine. Then do a fresh
var etc
installation of the older version. The configuration files in the directory may be compatible with older
etc
versions, but not all configurations may be read.
/opt/aspera/var/
/opt/aspera/etc/
Preparing for an Upgrade or Downgrade
1. Verify the version of your existing product.
The steps required to prepare for an upgrade may differ depending on your current product version. To view the
current product and version, run the following command:
# ascp -A
2. If you were using the Aspera Watch Service or Watchfolders, set a docroot for the user running those services, if it
is not already set.
For more information on setting docroots for users, see Updating the Docroot of a Running asperawatchfolderd
Service on page 144. Ensure that the filepath being watched, or the configured for the watch
source_dir
folder, is included in the docroot of the user.
3. Stop all FASP transfer-related applications, services and connections.
• connections
ascp
• SSH connections
| Standard Installation | 7
4. If you were using the asperanoded service, manually stop asperanoded and ensure the process is no longer
running.
• To stop the asperanoded service, run the following command:
# /etc/rc.d/init.d/asperanoded stop
• To determine if any asperanoded processes are running, run the following command:
# ps -Af | grep asperanoded
5. Back up configuration and settings files.
These files are found in the and folders.
etc var
• (contains server configuration, web configuration, user settings, license info)
/opt/aspera/etc/
• (contains Pre- and Post-Processing scripts, Connect Server)
/opt/aspera/var/
6. Back up the Redis database.
The Redis database is backed up as part of backing up the directory, but Aspera recommends backing it up
var
separately as well, particularly if it is stored on a different machine.
filepath database.backup
# sudo /opt/aspera/bin/asnodeadmin -b / /
Product Setup
To install Enterprise Server, log into your computer with root permissions, and follow the steps below.
1. Download the IBM Aspera product installer.
Use the credentials provided to your organization by Aspera to access:
http://downloads.asperasoft.com/en/downloads/1
If you need help determining your firm's access credentials, contact your Aspera account manager.
2. Run the installer as root:
version
# bash aspera-entsrv- -release.sh
An example of version is:
3.6.0.119806-aix-6.1-ppc32
3. Install the license.
To install the license from the command line, create the following file:
/opt/aspera/etc/aspera-license
Copy and paste your license key string into it, then save and close the file. To verify the license information, run
the following command:
# ascp -A
4. Review or update OpenSSH authentication methods.
Open your SSH Server configuration file from with a text editor.
/etc/ssh/sshd_config
To allow public key authentication, set to . To allow password authentication,
PubkeyAuthentication yes
set to , for example:
PasswordAuthentication yes
...
PubkeyAuthentication yes
PasswordAuthentication yes
...
| Standard Installation | 8
Save the file then run the following to reload SSH:
$ sudo stopsrc -s sshd
$ sudo startsrc -s sshd
To further secure your SSH Server, see Securing Your SSH Server on page 9.
5. Troubleshooting
If the installer freezes during installation, another Aspera product might be running on your computer. To stop all
FASP transfer-related applications and connections, see Before Upgrading or Downgrading on page 6.
Upgrade Follow up
If upgrading and the Redis DB is run on another system: Update the KV store keys to the latest format.
The local Redis DB schema is automatically updated by the installer, but non-local redis DB must be manually
updated by running the following command as root:
# /opt/aspera/bin/asnodeadmin --db-update
Configuring the Firewall
Your Aspera transfer product requires access through the ports listed below. If you cannot establish the connection,
review your local corporate firewall settings and remove the port restrictions accordingly.
Enterprise Server
Configure your firewall to allow the following ports:
• Inbound TCP/22 (or other TCP port set for SSH connections): The port for SSH connections.
Important: Aspera strongly recommends running the SSH server on a non-default port (allowing inbound SSH
connections on TCP/33001, and disallowing inbound connections on TCP/22) to ensure that your server remains
secure from SSH port scan attacks. For instructions on how to change your SSH port, see Securing Your SSH
Server on page 9.
If you have a legacy customer base utilizing TCP/22, then you can allow inbound connections on both ports.
Please refer to the topic Securing Your SSH Server on page 9 for details.
The firewall on the server side must allow the open TCP port to reach the Aspera server. No servers are listening
on UDP ports. When a transfer is initiated by an Aspera client, the client opens an SSH session to the SSH server
on the designated TCP port and negotiates the UDP port over which the data transfer will occur.
• Inbound UDP/33001: The port for FASP transfers, which use UDP/33001 by default, although the server may
also choose to run FASP transfers on another port.
• Local firewall: If you have a local firewall on your server (like ), verify that it is not blocking your
iptables
SSH and FASP transfer ports (such as TCP/UDP 33001). If you are using Vlinks, you will need to allow the Vlink
UDP port (55001, by default) for multicast traffic. For additional information on setting up Vlinks, see Setting Up
Virtual Links (Command Line) on page 23.
Remote Client Machines
Typically, consumer and business firewalls allow direct outbound connections from client computers on TCP and
UDP, and no configuration is required for Aspera transfers. In the special case of firewalls blocking direct outbound
connections, usually with proxy servers for web browsing, the following ports must be allowed:
• Outbound TCP/33001: Allow outbound connections from the Aspera client on the TCP port (TCP/33001 by
default, when connecting to a Windows server, or on another non-default port for other server operating systems).
• Outbound UDP/33001 (or a range, if required): Allow outbound connections from the Aspera client on the
FASP UDP port (33001, by default).
| Standard Installation | 9
• Local firewall: If you have a local firewall on the client (such as ), verify that it is not blocking your
iptables
SSH and FASP transfer ports (such as TCP/UDP 33001).
Important: Multiple concurrent clients cannot connect to a Windows Aspera server on the same UDP port.
Similarly, multiple concurrent clients that are utilizing two or more user accounts cannot connect to a Mac OS X,
FreeBSD, or Isilon Aspera server on the same UDP port. If connecting to these servers, you will need to allow a range
of outbound connections from the Aspera client (that have been opened incrementally on the server side, starting at
UDP/33001). For example, you may need to allow outbound connections on UDP/33001 through UDP/33010 if 10
concurrent connections are allowed by the server.
Securing Your SSH Server
Keeping your data secure is critically important. Aspera strongly recommends taking additional steps to set up and
configure your SSH server to protect against common attacks. These steps include the following:
1. Changing the TCP port.
2. Restricting user access.
3. Configuring transfer server authentication.
Changing and Securing the TCP Port
Generally, SSH servers listen for incoming connections on TCP Port 22. As such, Port 22 is subject to countless,
unauthorized login attempts by hackers who are attempting to access unsecured servers. An effective deterrent
is simply to turn off Port 22 and run the service on a seemingly random port above 1024 (and up to 65535). To
standardize the port for use in Aspera transfers, Aspera recommends setting the TCP port to 33001.
The OpenSSH suite included in the installer uses TCP/22 as the default port for SSH connections. Remote Aspera
clients attempt to establish an SSH connection with the server on port 33001. However, if the connection fails, the
client retries the connection on port 22. Aspera recommends opening TCP/33001 and disabling TCP/22 to prevent
security breaches of your SSH server.
• Open TCP/33001 and keep TCP/22 open until users are notified they should switch to TCP/33001.
• Once users are notified, block TCP/22 and allow traffic only on TCP/33001.
Prerequisites:
• Before changing the default port for SSH connections, verify with your network administrators that TCP/33001 is
open.
• Before closing port TCP/22, notify users of the change.
Aspera recognizes that disabling the default SSH connection port (TCP/22) may affect your client users. When
you change the port, ensure that you advise your users on how to configure the new port number, from the GUI (if
available and used) and from the command line.
GUI: To change the SSH port in Desktop Client, click Connections and select the entry for the server whose ports are
changing. On the Connection tab, click Show Advanced Settings and enter the SSH port number in the SSH Port
(TCP) field.
Command line: Clients running FASP transfers from the command line can specify the port by using the
-P 33001
option.
The following steps require root privileges.
1. Open the SSH configuration file.
/etc/ssh/sshd_config
2. Add the TCP/33001 SSH port.
| Standard Installation | 10
SSHD can listen on multiple ports, so you can have both TCP/33001 and TCP/22 open. To enable TCP/33001, add
the port to your file, as in the following example:
sshd_config
Port 22
Port 33001
Once your client users have been notified of the port change to TCP/33001, disable TCP/22 and use only
TCP/33001 by commenting out "Port 22" in your file. For example:
sshd_config
#Port 22
Port 33001
3. Disable non-admin SSH tunneling.
These instructions require that OpenSSH 4.4 or newer is installed on your system in order to use the
Match
directive. allows you to selectively override certain configuration options when specific criteria (based on
Match
user, group, hostname, or address) are met.
Open your SSH Server configuration file, , with a text editor. Add the following lines to the end
sshd_config
of the file (or modify them if they already exist):
AllowTcpForwarding no
Match Group root
AllowTcpForwarding yes
Depending on your file, you might have additional instances of that
sshd_config AllowTCPForwarding
are set to the default . Review your file for other instances and disable if necessary.
Yes sshd_config
Disabling TCP forwarding does not improve security unless users are also denied shell access, because they can
still install their own forwarders. Review your user and file permissions, and see the steps below for instructions
on modifying user shell access.
4. Update authentication methods
Public key authentication can prevent brute-force SSH attacks if all password-based authentication methods are
disabled. For this reason, Aspera recommends disabling password authentication in the file and
sshd_config
enabling private/public key authentication. To do so, add or uncomment and
PubkeyAuthentication yes
comment out .
PasswordAuthentication yes
PubkeyAuthentication yes
#PasswordAuthentication yes
PasswordAuthentication no
Note: If you choose to leave password authentication enabled, be sure to advise account creators to use strong
passwords and set to "no".
PermitEmptyPasswords
PermitEmptyPasswords no
5. Disable root login.
By default, OpenSSH allows root logins. However, disabling root access helps maintain a more secure server.
Aspera recommends disabling root access by commenting out in the
PermitRootLogin yes sshd_config
file and adding .
PermitRootLogin No
#PermitRootLogin yes
PermitRootLogin no
Administrators can use the command when root privileges are necessary.
su
6. Restart the SSH server to apply new settings.
To apply your new SSH server configuration settings, you must restart the server. Restarting your SSH server does
not affect currently connected users.