Astonishingly, it's already been four whole months since starting back at the university, which I find incredibly hard to believe. I'm utterly convinced that it was only a couple of weeks ago that I walked back into the Computer Laboratory as an SRA for the first time since 2021, but here we are, at the end of term already. Time to do a bit of a retrospective and forward-looking plan for the next 3-4 months!
On wednesday this week, I had a chance to sit down with Anil, supposedly to talk about the upcoming lecturing of 1A Foundations of Computer Science, but we ended up talking about what I've been doing for the past few months, and where it fits into the broader picture of the group as a whole. It was a really useful conversation, and I thought it would be good to outline it here while it's fresh in my mind.
So then, to start, what have I been doing? What have I achieved? What have I learnt? It's been a bit of a daunting experience, landing in a team that are already working one hundred miles an hour on things well out of my comfort zone. I've been going to group meetings and having lots of interesting conversations, but I've found it difficult to make the next steps happen. One area where I've had some success is in working with Sadiq on LLMs - in particular, getting local LLMs to solve programming exercises that we both wrote up. I've also been working with him on taking the output from the docs CI and summarising it with LLMs in order to create an MCP server that would help tools like Claude Code to choose OCaml packages to solve users' problems.
It's been somewhat easier, partly due to inertia, to carry on with projects that had been in flight at the time I started. Things like getting the Odoc 3 generated docs onto ocaml.org, which is finally complete only as of this week!. This has taken a whole lot of time, but I'm really pleased with the end results. There's still an awful lot of improvements that I'd like to see made, which, after drawing breath for a couple of weeks, I'll be writing down.
An itch I'd been wanting to scratch for a long time has been to look at client-side ocaml notebooks. I decided to make this an integral feature of this blog, and I've learnt an awful lot doing it. An important feature of this that I've been keeping in mind is the idea that we could use the ocaml-docs-ci tool to build the libraries, which would allow us to host a toplevel for every single package in opam-repository - allowing at best interactive examples, and at bare minimum merlin for live type-checking and autocompletion. The important principles to keep in mind for this are that:
I have this all working on my blog, where I have both an oxcaml worker and a standard ocaml worker and they both dynamically load in libraries and cmis as specified on the page.
I've also supervised a 1A course for the first time - Introduction to Probability, and I've done some marking for the 1A Foundations of Computer Science.
Something that I'd been expecting to do a lot on was work with oxcaml, but as the release happened later than anticipated and it coinciding with the marking and supervising, I've not done quite as much of this as I had thought I would. In addition, I had anticipated working on Odoc to start implementing the new features of oxcaml, but to avoid duplicating effort I've been waiting for the patches that have already been written at Jane Street to at least get odoc to compile, which have taken longer than I had hoped to get to me.
With that in mind, Anil and I then talked about the bigger picture, as those of you who know Anil will be entirely unsurprised to hear! In particular, how will we be weaving the various threads of these activites - the teaching of OCaml, the large-scale (for OCaml) CI work, the LLMs and Oxcaml work together to form a coherent whole? How do I find a balance between them and ensure that we find synergies as opposed to pulling in different directions? How do make sure what we're doing helps us navigate the upending of the nature of development that agentic coding is bringing?
A clear and obvious area where we'll be able to see real progress is to extract from docs CI the logic that I've been using to do efficient builds of packages. As I previously wrote about, the new CI system is far more efficient than some of the other ocurrent-based pipelines, and it would save a huge amount of compute time if we were to take this tech and apply it elsewhere.
So, how might we take what we've got and produce something useful to everyone? We need to take a hammer to the fracture points of the docs CI service and split it into individually useful parts. Here are some next steps as I see them now. Let's take the solver out of docs CI, and have a service whose sole job is to create a repository of up-to-date solutions for all versions of all packages in opam-repository. These are the data that allow us to build the tree of package builds.
Next, turn these solutions into one giant build. Perhaps a script? Maybe a giant buildkit dockerfile? This is very similar to Mark Elvers' day10 project. We can get this running on a big machine and just see how fast we can build everything. The key thing here is that it should be trivial to run this on a linux box. A raspberry pi or a 768-core behemoth with 3TiB of ram. Just how fast can we get it going? It's already building in a couple of days using sage, but that's using ocurrent/obuilder, which isn't quite the right tool for the job, and on a relatively puny machine. Can we do it in an hour? 10 minutes? Certainly the incrememntal builds ought to be done in seconds. What's the limit?
These tools can then be used as the foundation for other CI systems. For opam-repo-ci, where we should be able to do the builds for a new package incredibly quickly. For opam-health-check, where we currently build foundational packages like dune and findlib thousands of times per run.
Once we've got the packages built, docs CI is simply a pass over the top of the built artifacts. ocaml-docs-ci already demonstrates this - it only takes a few hours to rebuild all the docs when a new version of odoc is released, but in a way that only benefits docs! All the CI systems should be able to use this.
We should also then be able to run js_of_ocaml on the libraries to build to infrastructure needed for the per-package toplevels for ocaml.org that I mentioned above. Each of these steps should be separate stages in a pipeline - one where each step produces artifacts for the next to consume.
When we mix in some of the projects that other people in the team are working on, like David's work on relocatable OCaml, we've got something that might be able to form a basis for a binary cache for Dune Package Management, particularly when we involve Ryan's Hyperres paper so we might check that dependencies from outside of the OCaml universe are correct. Can we use Patrick and Michael's shark to do the build steps? Can we use these images to serve up toplevels for ocaml.org that are real toplevels rather than javascript toplevels? Can we use these build environments to do help with reinforcement learning to train LLMs on OCaml code? There are a lot of interesting directions to take this work.
There are, of course, other responsibilities that I have. Some of these I'll be able to fit in with the theme above, and some - well - maybe I'll have to figure out how to delegate them, a skill that I am not particularly good at, but one that I feel I should learn!
A looming, terrifying, but tremendously exciting opportunity is teaching of 1A Foundations of Computer Science. This is amongst the first courses we teach our incoming undergraduates, currently lectured by Anil. As he's on sabbatical this year, he has asked me to step up and lecture it. This is definitely not one for delegation!
The immediate question, partly raised by my work with Sadiq, is: what do we do about LLMs? How should we adjust our teaching to take into account the existence of these tools? We had a very interesting chat earlier in the term with Professor Viraj Kumar from IISc who was visiting Cambridge earlier this year. He's been working on this question for a while now, and I hope to have some more conversations with him over the summer.
An area where I've really made a shockingly small amount of progress is to write up all the work that's gone into Odoc over the past 6 (!!!) years.
This needs to be tidied up and a v0.1 released. In particular, the work on js_top_worker might well be shared with Arthur's x-ocaml for a unified toplevel experience.
I'd like to carry on the work I've started with Sadiq on the interaction of LLMs with OCaml. Getting the package search to work sensibly for an MCP server is first on the list, but also doing some reinforcement learning to improve specifically the perfomance on OCaml is very interesting, but not something I've managed to carve out the time for yet. Something along the lines of swesmith but adapted for OCaml.
Odoc needs to have some work done on it to support the new work that's gone into oxcaml, for example documenting of the modes. This is something I do expect to be working on soon.
Work needs to be done on the dune rules for odoc, which currently only support the feature-set in odoc 2.x. Paul-Elliot has done some work on this, but much more needs to be done.
There are a lot of things in the above list. I'm not sure yet how I manage to figure out what I actually end up doing, and how that helps me to help Tarides, to fit in as a useful member of the EEG group, and to make sure I'm doing what's right for my own future. I feel the core project of the CI work will help everyone, but slotting the other work into the bigger picture will require some careful thought.