The basic idea of using a directory as a queue is to place new "work" as a file in a "queue" directory. When the queue-processor is ready to process the new work-files it first moves the queue-directory's files to the queue-processor's "working" directory and then processes them.
The assumption here is that the work-files have all their content. But we all know that writing to a file takes time and during this writing time the queue-processor might awake and start processing the new, incomplete work-files. This is not good. If you are using a directory as a queue it is likely your queue-processor is a script and, unfortunately, most script solutions have poor error handling. The result is that the incomplete work-file gets ignored.
The good news is that there is an easy solution. I first saw this solution used within the qmail MTA. A feature of Unix file systems is that adding to or removing files from a directory is an atomic action. That is, all processes wanting to alter the set of files in a directory are queued up and only one at a time is allowed to make changes. So, when you write the work-content file make sure to write it first outside of the queue-directory and then move the completed work-file to the queue-directory. It is critical that the the work-content creator do the moving.
The archetype for this is to create a queue-directory containing two sub-directories
mkdir queue/new/Add the work-content file to queue/tmp with a globally unique file name. Once writing is finished and the file closed, then move it from queue/tmp/ to queue/new/. This even works nicely across networks, for example
mkdir queue/tmp/
QUEUE=/x/y/zSee also Using inotifywait with queue directories.
SOURCE="/a/b/c/foo.txt"
TARGET="$QUEUE/tmp/$(basename $SOURCE)-$(uuid)"
scp -q $SOURCE user@host:$TARGET
ssh -q user@host "mv $TARGET $QUEUE/new/$(basename $SOURCE)"
