SecureBlackbox 16: Ensuring Correct Data Exchange in FTP

When the data exchange is built in a way that requires one party to upload the file to the server and another file to download that file, it is a very common problem that the downloading party starts the downloading operation before the file is fully uploaded.

The issue here is detecting the moment when the file is fully uploaded. Unfortunately, there exists no universal one-size-fits-all solution -- the server software differs, and the details of file transfer scenarios differ as well.

Servers normally have no way to indicate to the downloading party that the file is incomplete. The reason is simple: from a technical point of view, in most cases, the server cannot know the expected file size or when the file is uploaded and cannot indicate the status of the file to other parties. In addition, most servers open local files in "reading allowed" share mode, which means that the file can be written by one party and read from other parties. If the files are opened in exclusive mode (no access until the writer closes the file), the problem does not exist, but in most cases, servers do not do this for multiple reasons.

Any solutions to the above issues involve a certain level of support by the uploading party and forward thinking by the designer of the data exchange scheme.

What you can do to ensure correct downloads:

• Use a server that locks files exclusively when they are written to and ensure that if the server opens files in the mode requested by the client that the uploading party uses exclusive mode when opening the server-side file for writing.
• Have the uploading client upload the files with a different name or extension that will not match the rules (file mask, etc.) according to which downloading parties detect files that they need to download.
• Have the uploading client create the zero-length "upload completion" indicator file after the original file is uploaded. Downloading parties would have to check for this dedicated flag file, which would have the same name and extension as the file that may be downloaded, with the appended special extension (for example, the flag file for "filename.ext" would be named "filename.ext.dlok"). If the flag file exists, these downloading parties would download the main file, then delete the flag.
• Have the uploading client create the flag file, similar to the one described in the previous option, to signal an incomplete upload. The downloading parties would not attempt to download the file if such a flag exists. The uploading party would remove the flag after completion of the upload.

As you can see, each of these approaches requires that you have at least some control over the uploading parties.

Another solution is possible when files are uploaded into one directory (the "upload directory"), then moved to another directory (the "download directory") by the server. Downloading parties do not have read access to the upload directory, and uploading parties cannot write to the download directory. This approach has the advantage of not requiring much control over uploading clients. The drawbacks are that (a) the server should be scriptable; i.e., you should be able to define the rules for moving the files, and (b) if the file upload is interrupted, the server, as said above, has no way to know the intended size of the file, so the partial upload is treated as complete, and the server would move the incomplete file for downloading.

As you see, the variety of options gives you some flexibility in implementing the data transfer. However, it is almost impossible to find a reliable solution in the case of conventional file transfer protocols (FTP/FTPS and SFTP) without the server and the data exchange protocol being designed specifically with these issues in mind.