Blender:Flamenco:Shaman

In this writeup, I'll show how to submit a scene file to Shaman using bash cli commands. Bash allows us to be expressive but concise helping teach the steps required.

Shaman is a staging library of arts assets stored not by name but by file hash and size and is consumed by the Flamenco renderfarm. Artist follow normal workflows of referencing textures and meshes from shared network drives or local drives in their 3d program but must do so only with relative paths.

We'll present bash snippets, to query and ultimately submit an asset to Shaman.

Step 1. Analyze file

FILE_PATH="derby-car.bsz"
if [[ "$OSTYPE" == "darwin"* ]]; then
    FILE_SIZE=$(stat -f%z "$FILE_PATH")
elif [[ "$OSTYPE" == "linux-gnu"* ]] || [[ "$OSTYPE" == "msys" ]]; then
    FILE_SIZE=$(stat -c%s "$FILE_PATH")
fi
FILE_HASH=$(shasum -a 256 "$FILE_PATH" | awk '{print $1}')

Step 2 Ask Shaman, you got this file? Usually you send a list of files

MANAGER_URL="http://10.88.0.1:8080/api/v3"
curl -s -X POST "$MANAGER_URL/shaman/checkout/requirements" \
     -H "Content-Type: application/json" \
     -d '{
       "files": [
         {
           "path": "'$REMOTE_PATH'",
           "sha": "'$FILE_HASH'",
           "size": '$FILE_SIZE'
         }
       ]
     }'

It returns a list of all files it does not already have

Step 3 Submit to Shaman

curl -s -X POST "$MANAGER_URL/shaman/files/$FILE_HASH/$FILE_SIZE" \
     -H "Content-Type: application/octet-stream" \
     -H "X-Shaman-Original-Filename: $FILE_PATH" \
     --data-binary @"$FILE_PATH"

Running the "requirements" query again will return

At any time you can verify storage status with the hash and the size

curl -s -X GET "$MANAGER_URL/shaman/files/$FILE_HASH/$FILE_SIZE"

Step 4 Create Virtual Filesystem Checkout, a symlink in the network share /mnt/oomerfarm/flamenco/my_favourite_dir/and_subdir/derby-car.bsz to the blob in Shaman's library. This virtual file path is what is passed to the command line renderer.

REMOTE_PATH="my_favourite_dir/and_subdir/derby-car.bsz"
CHECKOUT_NAME="test-$(date +%s)"
curl -s -X POST "$MANAGER_URL/shaman/checkout/create" \
     -H "Content-Type: application/json" \
     -d '{
       "checkoutPath": "'$CHECKOUT_NAME'",
       "files": [
         {
           "path": "'$REMOTE_PATH'",
           "sha": "'$FILE_HASH'",
           "size": '$FILE_SIZE'
         }
       ]
     }'

Step 5 Worker asset materialize, does not exist

Workers gets jobs that have one or more tasks. Workers currently must start their command line renderers by inputing all assets directly from the shared network drive. This simple renderfarm works but once we scale to 100 workers, we end up using a lot of bandwidth from a single server across either a robust network or a slow one. Since each invocation of blender or Bella cli renderer has the scene file as an input, it has to download from the shared network drive every frame.

A shared network drive is simple, but the next logical step is to have the flamenco-worker create a local cache or blob library. a mini Shaman. We can already do this by creating pre-render task that checks the manifest, checks local cached assets and downloads 0 or more files from the shared network drive as needed. Intuitively this means the first frame will get the big download hit, but frame 2-100 end up downloading nothing and are jump right to the path tracing stage.