COUNTER ORCHESTRATION

Kyuu Example

This is an example script of using the kyuu message queue for organizing two "worker" threads.

The setup

Running ./setup.sh will create three kyuus - one, two, and results.

Next, ./start_pipeline will spin up two worker processes: one polling on kyuu one for work and writing its output to kyuu two, the other polling on kyuu two for work and writing its results to kyuu results.

The workflow

The individul scripts called by orchestrate.sh send a message through a pipeline that runs the sed command on the message, replacing the second and third word in the input commands with "hello" and "world."

The beginning of a 'job' starts with sending a message into kyuu one, on which the first worker process listens. ./feed_inputs.sh, in our case, echoes a set of inputs from the file inputs into kyuu one.

As the first worker polls for work, it takes these messages and runs the shell script dowork1.sh on them. dowork1.sh uns a simple sed 's/one/two/ on the messages, then takes the outpt and routes it to kyuu two.

Worker two picks up these messages off of two, and runs a similar script dowork2.sh. It runs the output of this into the kyuu results.

Observing the output

cat results will show the final messages, one at a time. kyuu -fv results will dump all of the messages at once. The input will have transformed from 1 one one, 2 one one, etc., into 1 hello world, 2 hello world, etc. as a results of having the dowork* scripts run on it.

Testing manually

Running setup.sh will start up the kyuuD and creates the kyuus used by the pipeline we're using.