robertodr / totaltrash.xyz

Update Rust+Python+Nix blog

By Roberto Di Remigio on December 18, 2019
This patch is not signed.
6A7oomyVmGMzxfy63ewxFPnGZgpfn84KTeKnp2Q5mLDEGm6mA8Co5swW2U7PKUg4uHGcQ8BSkFUtzzdSn9v83JBM
This patch is in the following branches:
master
48
49


50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101

102
103









104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159

```bash
curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path --default-toolchain nightly
source "$HOME"/.cargo/env
$ curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path --default-toolchain nightly
$ source "$HOME"/.cargo/env
```
Other dependencies will be specified in the `Cargo.toml` file and installed when we build the extension.
Note that we chose `nightly` as default Rust toolchain. As was mentioned, [PyO3] is bleeding edge and uses features on the Rust language only available in nightly builds.

### Python dependencies

These are both for final users (_e.g._,
[Click](https://click.palletsprojects.com/en/7.x/) for the command-line
interface) and for developers (_e.g._,
[pytest](https://docs.pytest.org/en/latest/) for testing). Preferably one should
only install Python dependencies in virtual environments, **never globally**. I
am partial towards [Pipenv], but [Conda] could work as well.

Running `pipenv install --dev` with the following `Pipfile`:
```
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
maturin = ">=0.7.6"
pytest = ">=4.0"

[packages]
click = ">=7.0"
```
will create the virtual environment and install all needed packages. With `pipenv shell`, you can jump in the virtual environment.
The [maturin] package is **essential**: this is the tool that will orchestrate building the extension and packaging for PyPI.

## Workflow

OK, now we have a working development environment! To build the extension:

a. Make sure that the Rust toolchain is on `PATH`:
```bash
$ source "$HOME"/.cargo/env
```
b. Make sure that you are in the Python virtual environment:
```bash
$ pipenv shell
```
c. Invoke the build tool:
```bash
$ maturin develop
```
The `Cargo.toml` lists the Rust dependencies of our project and these will be
automatically downloaded and compiled if not already available.

**And DONE!** [maturin] will compile the module and link it into a dynamic
shared object (DSO) alongside the Python code:
shared object (DSO) alongside the Python code resulting in the following directory tree:
```bash
$ tree rustafarian

  rustafarian/
  ├── cli.py
  ├── __init__.py
  ├── module
  │   ├── double.py
  │   ├── __init__.py
  └── rustafarian.cpython-37m-x86_64-linux-gnu.so
rustafarian/
├── cli.py
├── __init__.py
├── module
│   ├── double.py
│   ├── __init__.py
└── rustafarian.cpython-37m-x86_64-linux-gnu.so
```

We can thus run tests with `pytest`:
```bash
$ pytest -x -rws -vv tests/test_*
```
or import `rustafarian` and use it in an interactive Python session spawned in our
development environment.

Running `ldd` on the DSO shows that there is no Rust runtime to link to:

```bash
$  ldd rustafarian/rustafarian.cpython-37m-x86_64-linux-gnu.so

  linux-vdso.so.1 (0x00007ffe979ac000)
  libdl.so.2 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libdl.so.2 (0x00007efd23df5000)
  librt.so.1 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/librt.so.1 (0x00007efd23deb000)
  libpthread.so.0 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libpthread.so.0 (0x00007efd23dca000)
  libgcc_s.so.1 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libgcc_s.so.1 (0x00007efd23bb4000)
  libc.so.6 => /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib/libc.so.6 (0x00007efd239fe000)
  /nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007efd23e63000)
```

## Packaging and deployment

It is now time to distribute this little Rust+Python package by deploying it to
PyPI. [PEP 517] introduced the `pyproject.toml` file to specify build systems
for Python packages. Ours will specify [maturin] as our build tool and thus look
like:

```toml
[build-system]
requires = ["maturin"]
build-backend = "maturin"
```

As you can see, there is no mention of user dependencies (Click, in our case) in
this file. Nor of console scripts to be installed alongside the module.
This information is contained in the `Cargo.toml`:

```toml
[package.metadata.maturin]
requires-python = ">=3.6"
requires-dist = ["click>=7.0"]
scripts = {rustafarian = "rustafarian.cli:cli"}
classifier = ["Programming Language :: Python"]
```
Why is this so? [maturin] will generate the appropriate Python packaging information based on the contents of `Cargo.toml`, thus replacing any Python-related packaging tool or configuration file.

164
165

166
167
It is now time to allow [direnv] to do its thing:
```bash
direnv allow
$ direnv allow
```