As of today, Odoc 3 is now live on OCaml.org! This is a major update to odoc, and has brought a whole host of new features and improvements to the documentation pages.
Some of the highlights include:
A huge amount of work went into the Odoc 3.0 release, and I'd like to thank my colleagues at Tarides, in particular Paul-Elliot and Jules for the work they put into this.
But the odoc release happened months ago, so why is it only going live now? So, the doc tool itself is only one small part of getting the docs onto ocaml.org. Odoc works on the cmt and cmti files that are produced during the build process, and so part of the process of building docs is to build the packages, so we have to, at minimum, attempt to build all 17,000 or so distinct versions of the packages in opam-repository. The ocurrent tool ocaml-docs-ci, which I've previously written about, is responsible for these builds and in this new release has demonstrated a new approach to this task, where we attempt to do the build in as efficient a way as possible by effectively building binary packages once for each required package in a specific 'universe' of dependencies. For example, many packages require e.g. cmdliner.1.3.0 to build, and some require a specific version of OCaml too. So we'll build cmdliner.1.3.0 once against each version of OCaml required -- but only once, which is in contrast to how some of the other tools in the ocurrent suite work, e.g. opam-repo-ci. Once the packages are built, we then run the new tool odoc_driver to actually build the HTML docs. In addition to this, a new feature of Odoc 3 is to be able to link to packages that are your direct dependencies - so for example, the docs of odoc contain links to the docs of odoc_driver, even though odoc_driver depends upon odoc. This, whilst sounding easy enough, required some radical changes in the docs ci, which I promise I will write about later!
The builds and the generation of the docs is all done on a single blade server, called sage with 40 threads, 2 8TiB spinning drives and a 1.8TiB SSD cache, and it produces about 1 TiB of data over the course of a couple of days. The changes required to this part of the process since odoc 2.x were primarily myself and Mark Elvers
Once the docs are built, how do they get onto ocaml.org? Odoc itself knows nothing about the layout and styling of ocaml.org, so the HTML it produces isn't suitable to be just rendered when a user requests particular docs. What happens is that odoc produces, as well as a self-contained HTML file, a json file with the body of the page, the sidebars, the breadcrumbs and so on as structured data, one per HTML page, which are then served from sage over HTTP. When a user requests a particular docs page, the ocaml.org server will request that json file from sage, then render this with the ocaml.org styling, then send it back to the user.
As odoc 3 moved a fair bit of logic from ocaml.org into odoc itself, there were quite a few changes that needed to be made to the ocaml.org server to integrate this into the site. This work was mostly done by Paul-Elliot and myself, with a lot of help from the ocaml.org team, in particular Sabine Schmaltz and Cuihtlauac Alvarado.
So, quite a lot of integration and infrastructure work was required to get the new docs site up and running, and I'm very happy to see this particular task concluded!