Summer notes

Getting back to writing notes! Here's a dump of what's been going on.

First class docs

The notion of making docs a 'first class' feature of OCaml seemed pretty important to me, so I applied to present it at the OCaml Workshop, and it got accepted! I was really looking forward to going and discussing with everyone important in OCaml development how to progress this, and then life intervened and I found out that my daughter Skylark had a rather major surgery scheduled for exactly the day of the workshop. I stayed here to be with her, and happily it all went well. Meanwhile in France they played a recording I had made of my talk, and Paul-Elliot answered questions. The slides are here

It has managed to start a discussion, so we'll see where that ends up.

Odoc fixes

As part of this, I needed to make sure that installing the odoc files wasn't going to be just a massive waste of space. An initial investigation showed the odoc files were roughly as large as the entire lib directory - so installing them doubled its size. This was clearly unacceptable, so I spent some time looking at saving space. We'd not even looked at this before, so I was pretty confident we'd find some good space savings.

There were 3 straightforward changes that made a big difference. Firstly, and most obviously, enabling compression. This was straightforward as OCaml itself has been compressing files since 5.1.0. Secondly, removing some optimisation in the representation of Identifiers that's become redundant since it was put in. This was a larger PR, but mostly mechanical. The one moderately interesting part of it was modifying the hash function of the identifiers to ensure we always traverse right to the Root constructor in deeply nested identifiers. The last fix was just in the AST representing comments. In it, a paragraph is an inline_element with_location list, and the inline elements are [Word of string], [Space], [`Code_span _] and so on. So each word and each space carried its own start and end location in the AST, which we almost never use. The third optimisation was simply to join runs of words and spaces into a single entry. With these three in place, the space usage went down by 6-7 times to a much more reasonable size.

While in an upstreaming mood I've also pulled out several other optimisations that were made for handling OxCaml-produced heavily templated code. My favourite: base uses ppx_template so heavily that it ends up with around 100,000 doc comments, of which only 33 are unique, so memoizing the parsing of comments saves a great deal of repeated work.

Odd

Odd is my tool to simulate the "First Class Docs" world by installing odoc files with odd_driver (a mini fork of odoc_driver) as an opam hook. It includes a CLI that can search docs, show them in markdown format, and complete references. I thought I might try getting an AI agent to use it to help write some OCaml programs, and then it could critique the tool and suggest improvements. This turned out to be very useful. I set up a few tasks that required using the Jane Street libraries, as these tend to have mli files that are less easily read than many others in the OCaml ecosystem. Not only did it end up suggesting a number of useful bugfixes and improvements to odd, but it also found several bugs in odoc that I submitted fixes for. A fix for extended opens, a fix to prevent hidden items leaking into the docs, some inconsistent strengthening of module types. It also spurred me on to fixing a long-standing issue - how to handle a hand-written wrapper module that doesn't expose everything.

The obvious next step was to see whether odd was actually helpful to the agents. To this end I changed the tasks to have 3 different prompts - one with just the task, one suggesting to look in the opam lib dir for the mli files, and one suggesting using odd. I set these all going with the same two models. I've not completely analysed the results yet but it does seem that odd isn't significantly helpful! That's a surprising enough result that I'm going to dig into why in more detail, and I'll report back.

For these tests I used a mix of Claude and DeepSeek-V4-Flash-0731, the latter running entirely locally on my laptop with antirez/ds4 and driven by pi.dev as the agent. It was slower than Claude, but perfectly capable of passing every task, which I still find quite amazing. For the three-prompt comparison I ran it through openrouter instead, purely for speed.

Docs CI

I switched over ocaml.org to the new docs CI running on dill.caelum.ci.dev. This is using "day11", which is a variant of Mark Elvers's day10. The switch-over was relatively smooth, but there were two issues. The first was that the rendering of package-wide markdown docs disappeared, leading to a PR to revert the switch. Fortunately I was able to fix it and rebuild everything within a day, so the PR was closed without merging.

The second issue was a bit more insidious. The docs CI machine, dill, has a layered storage solution - a pair of mirrored 7TB spinning disks with a 2TB SSD dm-cache in front. At that point I was pruning old layers by hand, and I had let them build up. When I finally triggered a prune, a big rebuild came in within an hour or so, and between them they completely filled the 2TB dm-cache disk. The whole machine ground to a halt for a few hours.

The way it's supposed to work is that when someone requests a documentation page on ocaml.org, the ocaml.org server requests a json representation of the page from dill, then renders that as the body back to the client. In order to make this as fast as possible, it's not the ocaml-docs-ci server that actually sends the page back, but a caddy server sitting in front of it. This works nicely most of the time, but when the machine was busy with flushing the dm-cache blocks it was taking quite a long time to respond to these requests. This led to the ocaml.org server running out of fds and crashing!

The two-fold solution was firstly to increase the fd limit on the ocaml.org server, and secondly to stick the layer pruning in a cron job that runs in the middle of the night. That keeps the size of the layers used in the docs down to around 1TB, and so we shouldn't run out of cache blocks.

Monitoring

The old docs-ci machine was quite difficult to maintain. It was hard to see what it was doing, why it was doing it, and to get to the relevant logs. As a consequence, the new incarnation had much more of a focus on being able to figure out what's going on. This manifests in two ways - firstly being able to browse the packages, the build and docs logs and classifying failures correctly, and secondly by having a much more useful grafana dashboard telling me how healthy the machine is.

The interesting thing about this dashboard is that Claude created it entirely, and via config files rather than driving the UI. It's not perfect - the 'Uptime' graph is a plot of x=y - but it certainly shows me almost everything that's interesting. The screenshot only shows the top of the dashboard - there's a lot more if you scroll down.

OxCaml VMM

A new project. I had a catchup with Dave Scott and while reminiscing about the good ol' XenServer days he mentioned the project libkrun. An OCaml version of this sounded like fun - and an OxCaml one even more so, as we should be able to use some of the OxCaml features to make it really quite efficient.

It's very early days for this now, but it's capable of booting a linux VM on my macbook and on my Raspberry Pi 4. It's already a useful playground for looking at how modes can help us write idiomatic looking code that doesn't allocate. Right now there's nothing to be gained with the multicore modes, but that may happen as I move beyond one vCPU. I'll do a bit more of a write-up of this soon. Code is here

A bit of fun

While I was waiting in the hospital I needed something to take my mind off the surgery. I can't remember what got me on this track, but I was thinking about old Amiga demos, and I remember one in particular - ARTE by Sanity that was spectacular. One effect I enjoyed particularly was a texture mapped sphere. It was something I had tried to write myself back in the day, involving, if I recall correctly, a lookup table that was created using AMOS basic. I wondered if Claude could take a look into it and figure out precisely how it worked.

It was fascinating watching Claude work at the problem. It pulled down a copy of amiberry, which exposes a socket over which you can control the emulator. I had bought a copy of Amiga Forever a few years back so I had all the kickstart roms and workbench disks around, so it used them, downloaded the ARTE adf and booted it up. I then paused it at the effect I was interested in, and Claude interrogated the state of the VM over the socket. It then searched around and found a copy of the Sanity Operation System which matched what was used to assemble the demo, in particular figuring out where local state was stored. It took quite a while to figure out the bitplane trickery, and there were several round trips where the output was messed up in various ways, but it got there in the end.

Of course, most of the interesting bits about the effect is that it worked at all on the Amiga hardware, and we miss out on all that in reproducing in modern OCaml, but nevertheless it's quite fun to see it running!

Here is the ocaml-arte-sphere repository, containing a terminal-based renderer and a Claudius renderer.