multifile(1)

NAME
multifile - multiplexing file server

SYNOPSIS
multifile -t <tlsconfigfile> -r <rootdir>

DESCRIPTION
Multifile is a multiplexing file server with Transport Layer Security. Mutifile delivers up to 255 multiplexed streams of file data per connection. You can turn off multiplexing to conserve resources. If available, multifile uses Kernel TLS for maximum performance and minimum memory footprint.

A companion library, libmultifile(3), provides client functions.

An example interactive client program (mclient) is installed with the server. The client does not use multifile's multiplexing capability.

  ADVERTISEMENT
When a client connects to multifile, multifile greets the client with a frame containing a line of UTF-8 text that advertises multifile's version and the maximum number of simultaneous requests multifile is configured to service per connection (the value of the -n option). The values are separated by a colon. The text is zero-terminated. The advertisment has the following form:

    Multifile-4.5:255

After they read the advertisement, clients send request frames.

  REQUEST FRAMES

Multifile exchanges data with clients in frames of up to 1056 bytes of payload preceded by a 2 byte unsigned payload length header.

A client requests up to 255 (or the value of the -n option) resources in a request frame. It is unlikely that you will be able to fit 255 requests in 1056 bytes, but you can piggyback further requests in subsequent request frames as described later in this document.

The payload of a request frame consists of fully-qualified Unix file system paths separated by zero bytes and terminated with a zero byte. Text must use the encoding that the server's file system uses.

    /path/to/file1\0
    1561208728/path/to/file2\0
    1561208728:4096/path/to/file3\0
    1561208728:4096:1024/path/to/file4\0

Components of paths must be separated by virgules (UTF-8 47). Multiple contiguous virgules are not an error in a path. Multiple virgules are collapsed into a single virgule by the kernel and have been since at least UNIX Version 6 of 1975.

If the sequence ".." appears as an element in a path, multifile drops the connection. If a path begins with ".", multifile drops the connection. If a path is empty, multifile drops the connection.

Paths must begin with a virgule or the UTF-8 represententation of 1 to 3 numbers separated by colons.

  UNCONDITIONAL REQUESTS
A path that begins with a virgule is an unconditional request. If the path describes a file, multifile returns the data in the file in one or more response frames. If the path describes a directory, multifile returns a content listing for the directory.

  CONDITIONAL REQUESTS
A path that begins with numbers is a conditional request.

The first number is a time value expressed as the number of seconds since the UNIX epoch (00:00:00 January 1, 1970 UTC). The server only returns data for the requested entity if its modification time is more recent than the time value.

The second number is optional. If present, it must be separated from the time value by a colon: ":" (UTF-8 58). The second number is an offset into the file. The server returns data for the file starting from the offset byte. The first byte in a file is numbered 0. This value is ignored with directory requests.

The third number is optional. It cannot be present unless the offset is present. If the third number is present, it must be separated from the offset by a colon. The third number is a byte count. The server delivers at most the byte count number of bytes from the file. A request with a byte count of 0 is identical to a request with no count. This value is ignored with directory requests.

If a client wants to request a range of bytes unconditionally, the client specifies a time value of 0.

If a client wants to request multiple ranges from the same file, the client prefixes each range to a separate request. This enables the client to multiplex the response for single files.

  RESPONSE FRAMES
Response data are delivered in multiplexed frames.

The first byte of payload of response frames is an unsigned numerical response ID. The value of this number corresponds to the ordinal position of a path in its request frame. The stream of response frames is terminated with a frame with a response ID of 0.

In the first response frame for a request, the remaining bytes subsequent to the response ID are the response header. A response header is UTF-8 text describing the number of data bytes to be delivered and the last modification time expressed in seconds since the UNIX epoch. The two values are separated by a colon. The text is zero-terminated. A non-zero count indicates that multifile will deliver data for that request.

•   If the count is zero and the time is zero, the server could not access the requested entity. Multifile delivers no further response frames for the path.

•   If the count is zero and the time is non-zero, the conditions of a file or directory request could not be satisfied, or the file is empty. Multifile delivers no further response frames for the path.

•   If the count is non-zero and the time is non-zero, a file request is successful and the file's data are delivered in subsequent response frames.

•   If the count is non-zero and the time is zero, the path is a directory path. Multifile successfully opened the directory. The count is inaccurate and should be ignored. The length of the response data is not known in advance. It will be zero if the directory is empty.

In response frames subsequent to the first for a request, the remaining bytes subsequent to the response ID are response data.

  DIRECTORY RESPONSES
The payload data returned for directory requests consists of lines of zero-terminated text. The text will be encoded in the encoding used by the server's file system.

The payload of the first frame delivered contains the directory's modification time and the directory's name separated by a colon. Each subsequent frame's payload contains the last modification time, size, and name of a directory entry, separated by colons. The name is appended with a virgule if the named entity is a directory.

  PIGGYBACKED REQUESTS
A client can send request frames while it receives response frames. The response IDs for the new requests are the same as they would be if all the active requests had been concatenated into one (potentially oversized) request frame. If a client submits more than 255 total requests (see the -n option), the excess requests are discarded. Note that each request path must fit entirely inside one request frame. Paths cannot be split across frames.

If a client submits 255 or fewer total requests and does not receive response frames for piggybacked requests before receving the terminator frame, piggybacking did not occur. The piggybacked requests arrived too late to be piggybacked. Multifile delivers the new requests in a new response stream.

Consider the following situation. A multifile server advertises that it accepts 4 simultaneous requests. The client submits a request frame that contains 2 requests. The server begins to deliver response frames. The client then submits another request frame that contains 2 requests.

If piggybacking occurs, the 3rd and 4th requests have the response IDs of 3 and 4 in the response stream. If piggybacking does not occur, the client must subtract the number of requests in the first request frame (2) from the would-be response IDs of the 3rd (3) and 4th (4) requests to determine the response IDs for those requests in the second response stream. This results in response IDs of 1 and 2 for the 3rd and 4th requests in the second response stream.

If piggybacking occurs, the client cannot submit more requests while the response stream is active, or the requests in excess of 4 will be discarded. Note that slots cannot be reused. If the client receives all of the data for one of the requested entities in a response stream, that does not free a slot for reuse. The client cannot submit another request until the entire response stream for all the requested entities has ended.

If piggybacking does not occur, the client is free to submit 2 more requests because only 2 slots are occupied in the second response stream. The 5th and 6th requests will either be piggybacked onto the second response stream with response IDs of 3 and 4, or they will arrive in a third response stream with response IDs of 1 and 2.

When piggybacking fails, the client must consider the number of requests in the previous request to be the sum of the number of initial and successfully piggybacked requests in the previous response stream. For example, if the multifile server advertises that it accepts 6 simultaneous requests, and the client submits the three request frames described previously, but only the second frame is successfully piggybacked onto the first, the client must subtract 2 + 2 = 4 from the would-be response IDs for the 5th and 6th requests, which results in response IDs of 1 and 2 in the second response stream.

  STOP FRAMES
If a client wants to stop all or part of a response stream, the client sends a stop frame. Then the client continues reading response frames until it reads the terminator frame. Stopped responses cannot be restarted.

The payload of a stop frame consists of a series of stop pairs. A stop pair consists of a UTF-8 full-stop character "." (UTF-8 46) followed by the UTF-8 representation of a number. The number must correspond to the ordinal position of a path in its request frame. A stop frame must be zero-terminated.

A stop pair with a zero index stops all requests from being delivered.

  CONNECTION CULLING
Multifile closes idle connections every 60 seconds. An idle connection is one that has not generated a response frame for 60 or more seconds.

  CONFIGURATION
Place the following lines in /etc/sysctl.conf and reboot. The 2 files values determine the maximum number of open files system-wide and per-process. Multifile uses a file descriptor for every connection and one for every file or directory in the current request frame. The 3 subsequent options turn on Kernel TLS.

    kern.maxfiles=262144
    kern.maxfilesperproc=131072
    kern.ipc.mb_use_ext_pgs=1
    kern.ipc.tls.ifnet.permitted=1
    kern.ipc.tls.enable=1

Multifile writes its pid into /var/run/multifile.pid. You stop multifile with a SIGTERM.

A rc.d script is installed in /usr/local/etc/rc.d/. Add the following lines to /etc/rc.conf to start multifile on system boot. Replace the items in brackets with values appropriate for your system. These are the minimal set of options you should start with. The available options are described in full at the end of this manual page.

    multifile_enable="YES"
    multifile_flags="-t <tls-config-file> -r <server-root> -u <user>"

Start, stop, or restart multifile, or determine if it is running with the following commands.

    service multifile start
    service multifile stop
    service multifile restart
    service multifile status

If you do not want multifile started on system start, set

    multifile_enable="NO"

and use the following commands.

    service multifile onestart
    service multifile onestop
    service multifile onerestart
    service multifile onestatus

  COMMAND-LINE OPTIONS
The following command line options are recognized. The -t and -r options are mandatory. The rest are optional.

-r  The -r option specifies the root directory of the server. The server invokes chdir(2) and chroot(2) on the value of this option before changing to the user and group specified by the -u and -g options.

-p  The -p option specifies the port to listen on. This defaults to 8000.

-i  By default, multifile accepts connections on all interfaces it can find capable of IPv4 or IPv6. The -i option limits the server to accepting connections from a specified interface. Pass the IP address of the desired interface as argument.

-m  By default, multifile maintains no more than 1024 simultaneous connections. The -m option changes this value.

-n  By default, multifile multiplexes up to 255 concurrent responses per connection. To lower this value, pass a positive value to -n.

-s  The -s option turns off multiplexing. Multifile delivers all response frames for a requested entity before delivering response frames for subsequent requested entities. Multifile queues only one frame for delivery each time the server processes a request queue.

-u

-g  The -u and the -g options are used to specify the user and group for the server. Both values default to "nobody".

-t  The -t option specifies the fully qualified path to the TLS configuration file. The file must contain 3 lines of text. The first line is the fully qualified path to the file holding the server key in PEM format. The second line is the password for the key. This line can be blank if there is no password. The third line is the fully qualified path to the file containing the certificate chain in PEM format.

AUTHORS
James Bailie <bailie9@icloud.com>
mammothcheese.ca