This tutorial explains how to embed a Python interpreter in a Rust-based Wasm module. It allows you to leverage existing Python code from your Wasm module with considerably less effort than by using CPython's C API directly.

We will demonstrate how to use the pyo3 crate to easily expose Rust code to Python and the wlr-libpy crate to quickly get a libpython build for wasm32-wasi.

Fast track

If you are already knowledgeable about Rust and Wasm and are the impatient type, you can take a look at the sample code in wasi-py-rs-pyo3 or wasi-py-rs-cpython. These are small, readable programs that cover a few basic cases to get you started. The main points are explained below, so you can return to this article if you run into issues.

To see the code in action you will need to have installed cargo with support for the wasm32-wasi target (rustup target add wasm32-wasi) as well as wasmtime.

Then you can execute the following commands from a terminal:

export TMP_WORKDIR=$(mktemp -d)
git clone --depth 1 \
  https://github.com/vmware-labs/webassembly-language-runtimes \
  -b rust-wasi-py-examples ${TMP_WORKDIR}/wlr
(cd ${TMP_WORKDIR}/wlr/python/examples/embedding/wasi-py-rs-pyo3; ./run_me.sh)

and you should see a result similar to the following:

   Compiling ...
    Finished dev [unoptimized + debuginfo] target(s) in 20.60s

Calling a WASI Command which embeds Python
  (adding a custom module implemented in Rust) and calls a custom function:
+ wasmtime \
  --mapdir /usr::target/wasm32-wasi/wasi-deps/usr \
  target/wasm32-wasi/debug/py-func-caller.wasm

Hello from Python (libpython3.11.a / 3.11.3
  (tags/v3.11.3:f3909b8, Apr 28 2023, 09:45:45) [Clang 15.0.7 ])
  in Wasm(Rust).
args=(
  ('John', 21, ['funny', 'student']),
  ('Jane', 22, ['thoughtful', 'student']),
  ('George', 75, ['wise', 'retired']))

Original people:
[Person(Name: "John", Age: 21, Tags:["funny", "student"]),
 Person(Name: "Jane", Age: 22, Tags:["thoughtful", "student"]),
 Person(Name: "George", Age: 75, Tags:["wise", "retired"])]
Filtered people by `student`:
[Person(Name: "John", Age: 21, Tags:["funny", "student"]),
 Person(Name: "Jane", Age: 22, Tags:["wise", "student"])]


Calling a WASI Command which wraps the Python binary
  (adding a custom module implemented in Rust):
+ read -r -d '\0' SAMPLE_SCRIPT
+ wasmtime \
  --mapdir /usr::target/wasm32-wasi/wasi-deps/usr \
  target/wasm32-wasi/debug/py-wrapper.wasm \
  -- -c \
'import person as p
pp = [p.Person('\''a'\'', 1), p.Person('\''b'\'', 2)]
pp[0].add_tag('\''X'\'')
print('\''Filtered: '\'', p.filter_by_tag(pp, '\''X'\''))'

Filtered:  [Person(Name: "a", Age: 1, Tags:["X"])]
+ set +x

This example features a person Python module defined entirely in Rust as part of the Wasm module's code. The person module is then used by Python scripts, which are interpreted by libpython embedded in the Wasm module. The Python code deals with creating person.Person instances with name, age and tags and then filtering those by tag with the person.filter_by_tag method.

Step-by-Step

Next, let's discuss how you can build something similar on your own, from scratch.

1. The end goal

The goal of this exercise is to have

• a Wasm module written in Rust
• which implements a Python module in Rust,
• embeds libpython,
• and when called, executes a Python script that can use the Python module.

This can be a stepping stone for other projects, where host functions (Wasm imports) or Rust logic can be exposed to the Python code or one could define a list of Wasm exports to expose an existing Python application as a Wasm module.

2. Defining the person module in Rust

For this example we use the pyo3 project. It allows us to create Rust bindings for Python and to run and interact with Python code from a Rust binary. This is very similar to what cbindgen provides for Rust-to-C communication or wh at wasmbindgen provides for Wasm host-to-module communication.

The pyo3 project has great documentation where you can find details about all the macros, types and functions shown below. The full code of the person module is available here.

To define a class that can be used in Python you need to add the respective pyo3 annotations like this:

#[derive(Clone)]
#[pyo3::pyclass]
#[pyo3(name = "Person")]
pub struct Person {
    #[pyo3(get, set)]
    name: String,
    #[pyo3(get, set)]
    age: u16,
    #[pyo3(get)]
    tags: Vec<String>,
}

Then for the class methods you should annotate the impl section. If necessary, you could implement special Python methods (e.g. __str__).

#[pyo3::pymethods]
impl Person {
    #[new]
    fn new(name: String, age: u16) -> Self {
        Self {
            name,
            age,
            tags: vec![],
        }
    }

    fn add_tag(&mut self, tag: String) {
        self.tags.push(tag);
    }

    fn __str__(&self) -> PyResult<String> {
        Ok(format!(
            "Person(Name: \"{}\", Age: {}, Tags:{:?})",
            self.name, self.age, self.tags
        ))
    }
}

Exposing a function to Python is also straightforward

#[pyo3::pyfunction]
pub fn filter_by_tag(people: Vec<Person>, tag: String) -> Vec<Person> {
    let mut result: Vec<Person> = vec![];
    for p in people.iter() {
        if p.tags.iter().any(|t| t == &tag) {
            result.push(p.clone());
        }
    }
    result
}

And finally, to define the module itself you have to implement a builder function that bundles all classes and functions together.

#[pyo3::pymodule]
#[pyo3(name = "person")]
pub fn make_person_module(
    _py: Python<'_>,
    module: &PyModule,
) -> PyResult<()> {
    module.add_function(pyo3::wrap_pyfunction!(filter_by_tag, module)?)?;
    module.add_class::<Person>()
}

As you can see, it's just typical Rust code with a few extra annotation lines (and occasional use of Py* types).

To expose this module to any Python script that will run in the embedded Python we can add it to the list of builtin modules, before initializing the CPython interpreter. This will allow Python scripts to simply import person. Here is the line needed to achieve this.

pyo3::append_to_inittab!(make_person_module);

With all of the above we can have something like this in Python

import person
people = [person.Person('a', 1), person.Person('b', 2)]
people[0].add_tag('X')
print('Filtered: ', person.filter_by_tag(people, 'X'))

which will give us the expected result:

Filtered:  [Person(Name: "a", Age: 1, Tags:["X"])]

3. Link to libpython via wlr-libpy

For pyo3 to work we will need a libpython build for wasm32-wasi. At Wasm Language Runtimes we publish pre-built assets for Python so you could get the one from there. As wasm32-wasi is universally portable you can always use the same asset no matter the build platform.

To make things easy we have created a wlr-libpy crate, which will download all necessary pre-built assets and setup the linker flags during the build process. Apart from libpython itself it also takes care of properly linking to the wasi-sdk libraries needed by CPython.

To use this crate you will need to add a build dependency to Cargo.toml

[build-dependencies]
wlr-libpy = {
  git = "https://github.com/vmware-labs/webassembly-language-runtimes.git",
  features = ["build"]
}

Then, invoke it in the build.rs file in your project root like this

fn main() {
    use wlr_libpy::bld_cfg::configure_static_libs;
    configure_static_libs().unwrap().emit_link_flags();
}

Effectively, this will generate a list of linker flags that looks like:

cargo:rustc-link-search=native=..../wasi-deps/wasi-sysroot/lib/wasm32-wasi
cargo:rustc-link-search=native=..../wasi-deps/lib/wasi
cargo:rustc-link-search=native=..../wasi-deps/lib/wasm32-wasi
cargo:rustc-link-lib=wasi-emulated-signal
cargo:rustc-link-lib=wasi-emulated-getpid
cargo:rustc-link-lib=wasi-emulated-process-clocks
cargo:rustc-link-lib=clang_rt.builtins-wasm32
cargo:rustc-link-lib=python3.11

These flags add the link paths where the libraries got downloaded and configures the linker to link to the wasi-emulated-* and clang_rt.builtins-wasm32, which are used by libpython3.11.

4. Combining pyo3 with wlr-libpy

You just need to add the typical pyo3 dependency in your Cargo.toml:

[dependencies]
pyo3 = { version = "0.19.0", features = ["abi3-py311"] }

Note, that by default pyo3 will look for a python3 installation on the build machine and try to get libpython from there. Even if it is installed, we will still use the wasm32-wasi libpython downloaded in wasi-deps/lib/wasm32-wasi. However, if you don't have a python3 installation you will need to set the PYO3_NO_PYTHON=1 variable when building.

PYO3_NO_PYTHON=1 cargo build --target=wasm32-wasi

5. Delegating to the python interpreter

CPython's API allows us to invoke the Py_Main method and delegate further execution to the Python interpreter. This is very useful if you want to produce a python.wasm WASI command which behaves like the original CPython interpreter (using command line arguments, stdin, stdout, etc.), but also has access to your Rust-define Python modules.

An example of doing that can be seen in py-wrapper.rs.

To make it easier we've exposed Py_Main via a Rust wrapper in the wlr-libpy crate. Just add this dependency in your Cargo.toml

[dependencies]
wlr-libpy = {
  git = "https://github.com/vmware-labs/webassembly-language-runtimes.git",
  features = ["py_main"]
}

Then you can add your Rust-defined Python module to the builtins and delegate to Py_Main:

use pyo3::append_to_inittab;
use wasi_py_rs_pyo3::py_module::make_person_module;
use wlr_libpy::py_main::py_main;

pub fn main() {
    append_to_inittab!(make_person_module);

    py_main(std::env::args().collect());
}

6. Calling your Wasm module

For CPython to work the Wasm module which embeds it will have to pre-open and properly map the folder with CPython's standard libraries. With wlr-libpy this folder is downloaded and extracted at target/wasm32-wasi/wasi-deps/usr and should be mapped as /usr

So, if we want to run py-wrapper.rs mentioned above we can do it like this

wasmtime \
  --mapdir /usr::target/wasm32-wasi/wasi-deps/usr \
  target/wasm32-wasi/debug/py-wrapper.wasm \
  -- -c 'import sys; print(sys.version)'

3.11.3 (tags/v3.11.3:f3909b8, Apr 28 2023, 09:45:45) [Clang 15.0.7 ]

Related work

Here is a list of resources about using Python in a Wasm module. The list is neither exhaustive nor organized in any way. The purpose is to present more information that can give you better understanding or spark new ideas about how Python can be used within Wasm modules.

• Joel Dice's work on componentize-py, which offers programmatic generation of bindings that wrap a Python app into a WASI Preview 2 component.

• Our own article for Extending Web apps with Wasm+Python, which shows how one can write up Python bindings manually with the C API.

• Brett Cannon's tutorial on running an existing Python app with python.wasm.

• A Fermyon article on using Python in WebAssembly.

• A Suborbital article on adding Python to their SE2 engine.

If you are looking for a place to discuss with people actively interested in Python on Wasm take a look at:

• The #python channel on the WebAssembly Discord server.
• The Python guest runtime and bindings stream on the ByteCode Alliance Zulip chat.

Summary

We went through the steps of embedding Python in a Rust-based Wasm module and pointed to some of the benefits and use-cases for doing that.

With pyo3 and wlr-libpy it's really easy to do this, so people can quickly experiment with new ideas.

Try out the wlr-libpy crate and let us know what you think. We would be happy to add more features and publish to crates.io if this is helpful to the community.