Benchmarks¶

Everything here was measured against real MongoDB by the unpublished bench/ package. Raw result files are in bench/results/<git sha>/; the numbers below are from 6dc837f.

Environment. MongoDB 8.0.32 in Docker (single-node replica set; a two-shard cluster where stated), Python 3.13.5, pymongo 4.18.3, msgspec 0.22, pydantic 2.14, Apple Silicon (8 cores), client and server on the same machine.

How to read the numbers.

• Absolute times depend on the machine. The comparisons that travel are ratios against the raw driver in the same process.
• Run-to-run variation on this setup is large: the same driver baseline moved by ±30% between runs. Treat a ratio within about 0.8–1.25 as "no difference". Conclusions below rest only on effects that were larger than that and repeated across runs.
• Cross-runtime absolute times (Python vs Node) are not comparable; overhead ratios are.

Reproduce with make up and uv run mongospecs-bench run, run LOAD, stress.

Summary¶

Requests in a real server¶

The storefront application runs under uvicorn (one worker) and is driven over TCP with a mix of 60% product listings (filter, sort, cursor page of 20), 30% product detail with reviews and a review count, and 10% orders (a multi-document transaction). The baseline implements the same three endpoints by hand with pymongo and Starlette, including the transaction.

Latency is measured with 2 virtual users (an unloaded server), capacity with 64. The primary metric is CPU time of the server process per request, which does not depend on the load generator.

No request failed in any run, and after every run the stock invariant held: for every product, stock plus units on placed orders equals the initial stock, and no stock is negative.

What the extra 0.13 ms buys: validated typed models, the field policy, allowlisted filters, scopes, ETags, problem documents and OpenAPI. In a profile of a listing the driver itself is the largest part of the request; decoding and serialising the 20 documents comes next.

A first version of this benchmark used httpx as the load generator and reported both servers at about 330 requests/s. That was the client's ceiling, not the servers': the server sat at 10–30% CPU. The generator was replaced with a minimal keep-alive client and CPU per request became the primary metric.

Reads¶

10,000 documents fetched with find(), median of 15 runs, ratio to the driver returning dicts in the same loop. "Nested" documents have a sub-document, a list of 8 sub-documents, a list and a map.

raw() is the driver. The cost of lean() is the backend's own validation: msgspec converts a nested document in 3.3 µs, attrs (through msgspec) in 8.3 µs, pydantic in 15.9 µs; the driver's BSON-to-dict step before that takes 10.1 µs.

Change tracking¶

5,000 nested documents:

A tracked document keeps the raw BSON bytes it was read from (about 1.2 KB here) and nothing else; the comparable snapshot is built only if the document is saved. Changing one field and saving sends a 38-byte update where a full $set would send 659 bytes.

This is the result of an optimisation made during this work. The first implementation re-encoded every loaded document to build the snapshot up front:

The same pass replaced one-await-per-document cursor iteration with batch fetches and removed a "trusted read" option whose pydantic fast path measured slower than validating (17 µs against 5 µs per wide document).

Compared with Mongoose¶

Mongoose 8 on Node 22, same MongoDB, same document shapes, ratios against the raw Node driver.

The two are not doing identical work: Mongoose's lean objects are untyped, while a mongospecs lean read validates into typed models, which is why Mongoose is ahead on lean reads. On tracked documents, the default in both, mongospecs' overhead over its driver is lower.

include(): $lookup or batched queries¶

Parents each referencing fan_out distinct children; ratio of $lookup time to batched time (below 1 means $lookup is faster). Two full runs on the replica set are shown because they disagree.

Decision: strategy="auto" means batched queries. $lookup is never clearly ahead, is 2.5–3× slower when many parents share few targets (the server repeats each target per parent), and is 5–9× slower through mongos for small fan-outs. It stays available as strategy="lookup" for the case where a single round trip matters more than time. This is also what Mongoose does.

A $lookup whose result document would exceed 16 MB fails with a typed DocumentTooLarge; the batched strategy loads the same data without trouble (stress scenario X5).

Pagination¶

50,000 documents, pages of 20, sorted on an indexed field.

• Cursor pagination sorts by the requested key and then by _id. With an index on the key alone the first page took 28.8 ms, because the server sorted in memory; with an index on (key, _id) it took 0.58 ms. Index listings accordingly.
• A total from a concurrent count_documents took 6.0 ms; the same page and total through $facet took 28.5 ms and is additionally bound by the 16 MB document limit. $facet is not used.

Writes and plugins¶

Ratios to the driver; these are single round trips where the network dominates and noise is highest.

Per-operation cost with 0, 3 and 10 no-op plugins on a registry differed by less than the noise (0.94–1.20× with no trend), as did the three built-in plugins.

Stress¶

All scenarios pass at full size.

X7 ran on a standard (GIL) build. The unit suite runs on the free-threaded build in CI as an experimental job.

End-to-end¶

bench/tests/test_e2e_storefront.py boots the storefront under uvicorn and talks to it over HTTP:

• a full customer journey on each of the five plugs, and ordering on each model backend
• 40 concurrent buyers for 10 units: exactly 10 orders succeed, 30 are refused, stock ends at 0
• multi-item orders are all-or-nothing, including opposing lock orders that must not deadlock
• the change-stream projector counts every sale exactly once across a restart and a period with no projector running
• requests in flight finish when the server is asked to stop
• under traffic, a primary step-down produces only declared errors (503/504), and afterwards every accepted order has moved stock exactly once

A separate check (mongospecs-bench trace) sends requests through the Sentry SDK to a local rustrak server and asserts that each request arrives as one trace containing the HTTP transaction and its Model.op database spans.

These tests found defects the unit and contract suites had not: aggregation paths failed whenever a default time limit was configured, ids given as strings to reference fields were stored as strings on msgspec and attrs, and unexpected server errors were not logged.

The Rust question¶

The plan asked whether a Rust component is justified, to be decided from these measurements.

Where the time goes when reading 10,000 nested documents with msgspec models:

• For bulk reads, decoding BSON into dicts is essentially all of the driver's time: measured alone it takes 101 ms for 10,000 documents, and the driver's whole read took 93 ms in the same run (equal within noise). It is about two thirds of a lean read. Decoding BSON straight into models, skipping the dict, is the one change that could make mongospecs faster than the raw driver rather than slower. That is a native-code job.
• For request-sized reads (20 documents) that step is an estimated 10–20% of the server CPU per request (20 documents at 3–7 µs against 0.63 ms); the driver's asynchronous machinery dominates. A native decoder would gain little there.
• Nothing else measured (query building, plugins, tracking, serialisation) is large enough to move to Rust.

Recommendation: no Rust in 0.2. If bulk reads matter to you, the worthwhile project is a BSON-to-model decoder as a separate optional package behind Codec.decode(bytes). The seam is already there: tracked reads receive undecoded BSON from the driver today. A Rust driver is not indicated by anything here.

Hot paths after the performance review¶

A review profiled the library and listed thirteen places where work could be saved. Workload B9 measures them through public entry points: CPU time of the client process, and commands sent. Before is bench/results/49a32a9/B9.json, after is bench/results/fb543de/B9.json (msgspec rows).

What this shows:

• The large wins are where work was avoided altogether: children that were fetched to be thrown away, a read before a delete, a scan over every route.
• Serialising from the stored document is 1.7 to 2.6 times cheaper than going through the model (pydantic 6.7 → 2.9 µs, attrs 6.4 → 2.5 µs), but a whole list request improves by under 10% here: the driver and the event loop are most of a request, as the Rust section above already found.
• The last three rows are within run-to-run variation on this machine. The cheaper weak reference per tracked document, the cheaper query clone and concurrent includes are correct and tested, but this benchmark cannot show their effect; do not read the tracked row as a regression or the get row as a gain.

Not done, and why: compiling encode plans and skipping reference normalisation on decode (review items 2 and 3) change the decode and encode core for a gain B1 puts at a few microseconds per document; they are left until a workload shows per-document CPU, rather than the driver, is the limit.

Limits of these measurements¶

• One machine, client and server together, MongoDB in Docker on macOS. Network cost is unrealistically low and variance is high.
• One uvicorn worker; no TLS, no proxy.
• The free-threaded build has not been benchmarked.
• The Mongoose comparison uses default settings on both sides and no tuning of either.