Deconstructing veilid-wasm's wasm_build.sh

Veilid | Veilid WASM

The wasm build script can be found at veilid/veilid-wasm/wasm_build.sh. Before I run it, I want to understand what it does.

SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

${BASH_SOURCE[0]} contains the name, including the (possibly relative) directory of the current script. Dirname gets the directory name of the script, and then cd changes directories to said directory.

If CD fails, the && short-circuits, and everything explodes. Otherwise, pwd returns the new current directory, and that is written to SCRIPTDIR. Since cd was run from a subshell, it does not impact the rest of the running script.

Long story short, $SCRIPTDIR now contains the absolute directory path to wasm_build.sh.

set -eo pipefail

This causes the script to exit (-e) if the pipefail option triggers (-o pipefail). If anything from any pipeline explodes, the script explodes.

get_abs_filename() {
    # $1 : relative filename
    echo "$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"
}

You can actually find that exact chunk in this stackoverflow answer. It takes a filename ($1) and returns its absolute path.

pushd $SCRIPTDIR &> /dev/null

pushd is the same as cd, except it prints stuff (thus the &> /dev/null), and remembers where you’re coming from. We’ll see popd later. It’s a stack!

This will essentially cd to the directory where the script is. This lets us run the script from somewhere else, and yet everything meaningful inside the script can rely on a consistent working dir.

Finding DwarfDump

if [ -f /usr/local/opt/llvm/bin/llvm-dwarfdump ]; then
    DWARFDUMP=/usr/local/opt/llvm/bin/llvm-dwarfdump
elif [ -f /opt/homebrew/llvm/bin/llvm-dwarfdump ]; then
    DWARFDUMP=/opt/homebrew/llvm/bin/llvm-dwarfdump
else 
    DWARFDUMP=`which llvm-dwarfdump`
    if [[ "${DWARFDUMP}" == "" ]]; then
        echo llvm-dwarfdump not found
    fi
fi

This chunk sets $DWARFDUMP to the path to the llvm-dwarfdump executable. I’m not sure what dwarfdump does (yet), but given that there is no explosion if it’s not found, I suspect it’s an optional helper.

Detecting Build Profile

if [[ "$1" == "release" ]]; then  
    # ... build for release
else
    # ... build for debug
fi

This if statement matches off the first argument to the script. We’ll build for release on ./wasm_build.sh release, and build for debug on anything else.

Building for Release

OUTPUTDIR=../target/wasm32-unknown-unknown/release/pkg
INPUTDIR=../target/wasm32-unknown-unknown/release

These define two directories, relative to the script, for the context of wasm-bindgen. Cargo’s output will be wasm-bingen’s input.

cargo build --target wasm32-unknown-unknown --release

This tells Cargo to build for wasm, with release settings.

To get that to run manually, I had to install my distro’s capnproto package (compilation requirement for veilid-core). Details for that are in my finalized Compiling for Wasm notes.

For me, the generated wasm file came out to about 8.5MB.

Generating Bindings

mkdir -p $OUTPUTDIR
wasm-bindgen --out-dir $OUTPUTDIR --target web $INPUTDIR/veilid_wasm.wasm

1. Create the pkg directory
2. Run wasm-bindgen to generate the JS bindings in pkg
   • --target web generates a standalone ESModule.

The wasm-bindgen binary is provided by the wasm-bindgen-cli package, which you can get with cargo install.

Here’s what I ran to manually replicate the above:

WASMDIR=../target/wasm32-unknown-unknown/release
mkdir $WASMDIR/pkg
wasm-bindgen --out-dir $WASMDIR/pkg --target web $WASMDIR/veilid_wasm.wasm

This creates a number of files:

veilid_wasm_bg.wasm      - An optimized wasm blob
veilid_wasm_bg.wasm.d.ts - TS definitions for said wasm
veilid_wasm.js           - JS bindings / wasm wrapper
veilid_wasm.d.ts         - TS definitions for the JS bindings

This pkg dir can be dumped just about anywhere, and behaves like any other ESM module.

wasm-bindgen appears to have knocked size down to about 5.6MB.

Optimizing Size

wasm-strip $OUTPUTDIR/veilid_wasm_bg.wasm

The wasm-strip binary comes from “wabt”, aka the WebAssembly Binary Toolkit. This comes via distro package managers, ie sudo pacman -S wabt on arch.

Keeping my $WASMDIR variable from above, here’s what I ran to replicate:

wasm-strip $WASMDIR/pkg/veilid_wasm_bg.wasm

Final wasm size is 4.4MB.

Building for Debug

OUTPUTDIR=../target/wasm32-unknown-unknown/debug/pkg
INPUTDIR=../target/wasm32-unknown-unknown/debug

This is much the same as for release; just using the debug directory.

RUSTFLAGS="-O -g $RUSTFLAGS" cargo build --target wasm32-unknown-unknown

This adds two flags to the RUSTFLAGS environment variable, prior to running cargo build without the release flag. RUSTFLAGS are passed through to the rustc compiler.

• -O enables level-2 optimizations
• -g adds debug information

Cargo’s default behavior is to build with debug settings.

Here’s what I ran to replicate:

RUSTFLAGS="-O -g" cargo build --target wasm32-unknown-unknown

Note that RUSTFLAGS impacts dependencies as well, so this will be a complete from-scratch recompile.

Wasm blob comes out to 197MB for me. That’s huge!

Generating Bindings

mkdir -p $OUTPUTDIR
wasm-bindgen --out-dir $OUTPUTDIR --target web --keep-debug --debug $INPUTDIR/veilid_wasm.wasm

This is very close to the release build, with two extra tags:

• --debug adds some extra debug code (both js and wasm)
• --keep-debug does not strip “DWARF” debug sections

Looks like llvm-dwarfdump (in the $DWARFDUMP variable) from above is capable of printing DWARF debug information.

Here’s what I ran to replicate:

WASMDIR=../target/wasm32-unknown-unknown/debug
mkdir $WASMDIR/pkg
wasm-bindgen --out-dir $WASMDIR/pkg --target web --keep-debug --debug $WASMDIR/veilid_wasm.wasm

veilid_wasm_bg.wasm comes out to 195MB, a very small decrease.

Generating a Sourcemap

./wasm-sourcemap.py $OUTPUTDIR/veilid_wasm_bg.wasm -o $OUTPUTDIR/veilid_wasm_bg.wasm.map --dwarfdump $DWARFDUMP

This is a python3 file that generates a sourcemap. It appears to come from the emscripten project. The original script can be found here.

The --dwarfdump flag is always passed, even if $DWARFDUMP is blank - if you don’t have llvm-dwarfdump installed, this script will explode.

Here’s what I ran to replicate, keeping my $WASMDIR variable from above:

python3 wasm-sourcemap.py $WASMDIR/pkg/veilid_wasm_bg.wasm -o $WASMDIR/pkg/veilid_wasm_bg.wasm.map --dwarfdump /usr/bin/llvm-dwarfdump

The .map file comes out to about 3.5MB.

Wrapping Up

popd &> /dev/null

This “undoes” the pushd/“cd“ from earlier, returning the working directory to wherever you were before you ran the script.

# Print for use with scripts
echo SUCCESS:OUTPUTDIR=$(get_abs_filename $OUTPUTDIR)

Prints the absolute path to the output directory (could be either release or debug). That’s it!

My condensed notes on pre-reqs and tool dependencies are on the compiling for wasm page.