NPtcp - Online in the Cloud

PROGRAM:

NAME

NetPIPE - Network Protocol Independent Performance Evaluator

SYNOPSIS

NPtcp [-h receiver_hostname] [-b TCP_buffer_sizes] [options]

mpirun [-machinefile hostlist] -np 2 NPmpi [-a] [-S] [-z] [options]

mpirun [-machinefile hostlist] -np 2 NPmpi2 [-f] [-g] [options]

NPpvm [options]

See the TESTING sections below for a more complete description of how to run NetPIPE in
each environment. The OPTIONS section describes the general options available for all
modules. See the README file from the tar-ball at
http://www.scl.ameslab.gov/Projects/NetPIPE/ for documentation on the InfiniBand, GM,
SHMEM, LAPI, and memcpy modules.

DESCRIPTION

NetPIPE uses a simple series of ping-pong tests over a range of message sizes to provide a
complete measure of the performance of a network. It bounces messages of increasing size
between two processes, whether across a network or within an SMP system. Message sizes
are chosen at regular intervals, and with slight perturbations, to provide a complete
evaluation of the communication system. Each data point involves many ping-pong tests to
provide an accurate timing. Latencies are calculated by dividing the round trip time in
half for small messages ( less than 64 Bytes ).

The communication time for small messages is dominated by the overhead in the
communication layers, meaning that the transmission is latency bound. For larger
messages, the communication rate becomes bandwidth limited by some component in the
communication subsystem (PCI bus, network card link, network switch).

These measurements can be done at the message-passing layer (MPI, MPI-2, and PVM) or at
the native communications layers that that run upon (TCP/IP, GM for Myrinet cards,
InfiniBand, SHMEM for the Cray T3E systems, and LAPI for IBM SP systems). Recent work is
being aimed at measuring some internal system properties such as the memcpy module that
measures the internal memory copy rates, or a disk module under development that measures
the performance to various I/O devices.

Some uses for NetPIPE include:

Comparing the latency and maximum throughput of various network cards.

Comparing the performance between different types of networks.

Looking for inefficiencies in the message-passing layer by comparing it to the
native communication layer.

Optimizing the message-passing layer and tune OS and driver parameters for optimal
performance of the communication subsystem.

NetPIPE is provided with many modules allowing it to interface with a wide variety of
communication layers. It is fairly easy to write new interfaces for other reliable
protocols by using the existing modules as examples.

TESTING TCP

NPtcp can now be launched in two ways, by manually starting NPtcp on both systems or by
using a nplaunch script. To manually start NPtcp, the NetPIPE receiver must be started
first on the remote system using the command:

NPtcp [options]

then the primary transmitter is started on the local system with the command

NPtcp -h receiver_hostname [options]

Any options used must be the same on both sides.

The nplaunch script uses ssh to launch the remote receiver before starting the local
transmitter. To use rsh, simply change the nplaunch script.

nplaunch NPtcp -h receiver_hostname [options]

The -b TCP_buffer_sizes option sets the TCP socket buffer size, which can greatly
influence the maximum throughput on some systems. A throughput graph that flattens out
suddenly may be a sign of the performance being limited by the socket buffer sizes.

TESTING MPI and MPI-2

Use of the MPI interface for NetPIPE depends on the MPI implementation being used. All
will require the number of processes to be specified, usually with a -np 2 argument.
Clusters environments may require a list of the hosts being used when each job is run.
Put the list of hosts in hostlist then, for OpenMPI, run NetPIPE using:

mpirun --hostfile hostlist -np 2 NPmpi [NetPIPE options]

For MPICH2 use instead:

mpirun -machinefile hostlist -np 2 NPmpi [NetPIPE options]

To test the 1-sided communications of the MPI-2 standard, compile using:

make mpi2

Running as described above and MPI will use 1-sided MPI_Put() calls in both directions,
with each receiver blocking until the last byte has been overwritten before bouncing the
message back. Use the -f option to force usage of a fence to block rather than an
overwrite of the last byte. The -g option will use MP_Get() functions to transfer the
data rather than MP_Put().

TESTING PVM

Start the pvm system using:

pvm

and adding a second machine with the PVM command

add receiver_hostname

Exit the PVM command line interface using quit, then run the PVM NetPIPE receiver on one
system with the command:

NPpvm [options]

and run the TCP NetPIPE transmitter on the other system with the command:

NPpvm -h receiver hostname [options]

Any options used must be the same on both sides. The nplaunch script may also be used
with NPpvm as described above for NPtcp.

TESTING METHODOLOGY

NetPIPE tests network performance by sending a number of messages at each block size,
starting from the lower bound on the message sizes.

The message size is incremented until the upper bound on the message size is reached or
the time to transmit a block exceeds one second, which ever occurs first. Message sizes
are chosen at regular intervals, and for slight perturbations from them to provide a more
complete evaluation of the communication subsystem.

The NetPIPE output file may be graphed using a program such as gnuplot(1). The output
file contains three columns: the number of bytes in the block, the transfer rate in bits
per second, and the time to transfer the block (half the round-trip time). The first two
columns are normally used to graph the throughput vs block size, while the third column
provides the latency. For example, the throughput versus block size graph can be created
by graphing bytes versus bits per second. Sample gnuplot(1) commands for such a graph
would be

set logscale x

plot "np.out"

OPTIONS

-a asynchronous mode: prepost receives (MPI, IB modules)

-b TCP_buffer_sizes
Set the send and receive TCP buffer sizes (TCP module only).

-B Burst mode where all receives are preposted at once (MPI, IB modules).

-f Use a fence to block for completion (MPI2 module only).

-g Use MPI_Get() instead of MPI_Put() (MPI2 module only).

-h hostname
Specify the name of the receiver host to connect to (TCP, PVM, IB, GM).

-I Invalidate cache to measure performance without cache effects (mostly affects IB
and memcpy modules).

-i Do an integrity check instead of a performance evaluation.

-l starting_msg_size
Specify the lower bound for the size of messages to be tested.

-n nrepeats
Set the number of repeats for each test to a constant. Otherwise, the number of
repeats is chosen to provide an accurate timing for each test. Be very careful if
specifying a low number so that the time for the ping-pong test exceeds the timer
accuracy.

-O source_offset,dest_offset
Specify the source and destination offsets of the buffers from perfect page
alignment.

-o output_filename
Specify the output filename (default is np.out).

-p perturbation_size
NetPIPE chooses the message sizes at regular intervals, increasing them
exponentially from the lower boundary to the upper boundary. At each point, it
also tests perturbations of 3 bytes above and 3 bytes below each test point to find
idiosyncrasies in the system. This perturbation value can be changed using the -p
option, or turned off using -p 0 .

-r This option resets the TCP sockets after every test (TCP module only). It is
necessary for some streaming tests to get good measurements since the socket window
size may otherwise collapse.

-s Set streaming mode where data is only transmitted in one direction.

-S Use synchronous sends (MPI module only).

-u upper_bound
Specify the upper boundary to the size of message being tested. By default,
NetPIPE will stop when the time to transmit a block exceeds one second.

-z Receive messages using MPI_ANY_SOURCE (MPI module only)

-2 Set bi-directional mode where both sides send and receive at the same time
(supported by most modules). You may need to use -a to choose asynchronous
communications for MPI to avoid freeze-ups. For TCP, the maximum test size will be
limited by the TCP buffer sizes.

Use NPtcp online using onworks.net services