Opam 103: Starting a new Project

---

Welcome back to the opam deep-dives series!

Ever since the first episode in the series have our readers asked for us to explore the dev side of the opam experience. Needless to say that it has always been in our plans to do so, however, for the sake of clarity and accessibility, we had to deal with user-facing scenarios first as neat introductions to the general subject matter.

We warmly thank you for your patience! Your wait was not in vain, because today, we start a new project in an opam-encompassing workflow! 🚀🚀

That being said, expect this first dev-side tutorial on opam to be most useful to the newer OCaml devs out there. 😇

Be sure to read the other episodes available at this point in time: Opam 101: The First steps, which introduces you to the fundamentals of opam, from installation to exploration, and Opam 102: Pinning Packages which already dives quite deep into package pinning, one of the first keys to tailoring your workflow and environment to your exact needs.

Also, check out each article's tags to get an idea of the entry level required for the smoothest read possible!

New to the expansive OCaml sphere? As said on the official opam website, opam has been a game changer for the OCaml distribution, since it first saw the day of light here, almost a decade ago.

---

Contextual Requisites

Our goal for this post and the next one is to take you on a journey which starts with creating a directory for your new OCaml project on your filesystem and ends with publishing it on the Official opam-repository.

We will aim at clarifying at every step of the way what is good to keep in mind and try to paint a most realistically exhaustive and concise opam dev-side scenario. We will cover switch creation, package selection and incorporation, code sharing, dependencies locking and distribution.

As far as the minimum required familiarity with opam goes for you to be able to fully enjoy this tutorial, we recommend only for you to have read our first article: Opam 101: The First Step and especially the section that explains what a switch is.

Nevertheless, here's a quick TL;DR for those of you who would rather get started:

What is an opam switch?

Conceptually, it's the environment of the OCaml dev. Opam provides you with a command-line interface for you to customise, and maintain a safe and stable environment. It's defined by all the possible combinations and valid operations between a specific version of the OCaml compiler, and any set of versioned packages.

Functionally, it is set of environment variables that are user-updated and point to the different locations of installed versions of packages, binaries and other utilities either in a ~/.opam directory for global switches or in the current _opam directory for local ones.

Link to the official documentation here.

Since there are two different kinds of switches, and since today's subject matter is bounded to making a new project, to keep things simple, we will use local switches in our examples.

Ready? Let's go!

Setting up the environment

Unsurprisingly, the first step on the journey to publishing your very own OCaml package, is to set it up to be developed... 🤯

This encompasses everything from creating the working directory of your new project, to setting up a custom, local switch for it.

We will consider that you have created a new directory for your project and have since moved into it in order to progress further in the setup process.

Something like:

$ mkdir helloer
$ cd helloer

Here are the things that opam will help you accomplish at this stage of the development process, we will do our best to explain them in the upcoming section:

• Setting up a new switch (i.e, environment creation);
• Browsing the OCaml package distribution for libraries and tooling in general (i.e, technical exploration);
• Selection and installation of OCaml software inside a switch (i.e, environment setup and tailoring);
• Contraint-based package compatibility calculations, which entails automatic solving of package dependency trees (i.e, automatic environment safety verifications);

Creating a new local switch

$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.

A switch is the virtual environment in which opam will operate and assist you in taking all necessary steps when coming up with an optimal enough workflow.

As said previously, a switch is comprised of a specific version of the OCaml compiler and a set of packages that are compatible with that specific version.

So let's first create a local switch in our helloer directory.

$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

We let opam select the default switch invariant when creating a new switch which is OCaml compiler version >= 4.05.0. You can define any set of switch invariants that you wish.

In the call above, the . character is indicative that we are asking opam to create a switch inside the current directory, a local switch as opposed to a global one by giving opam a path.

The idea of switch invariants is quite simple, they are the parameters to the automatic solving of package dependency trees and they are immutable. Opam will never change invariants without notifying you first and will always consider the switch invariants when building the graph of available and compatible packages for your current switch, or for any other switch-altering operation for that matter.

So, back to our example:

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.

We can see that opam selected ocaml-system.4.14.1 as opposed to ocaml-base-compiler.4.14.1 as the OCaml compiler to install in your current local switch along with its dependencies.

The difference between these two compilers is that ocaml-system is a system-bound compiler, typically one installed outside of your opam installation; e.g. with the help of the package manager of your favourite OS. On the other hand, ocaml-base-compiler would be a new compiler installed within your opam installation, one that opam would have permission over.

If you recall this section of Opam 101, you should know that creating a switch can be a fairly time-consuming task depending on whether or not the compiler version you have queried from opam is already installed somewhere on your machine, every time you ask opam to install a version of the compiler, it will first scour your installation for a locally available version of that compiler to save you the time necessary for downloading, compiling and installing a brand new one. This is the reason why opam has selected an ocaml-system.4.14.1 compiler instead of installing a brand new ocaml-base-compiler.4.14.1.

A quick look at our current directory will show that an _opam directory can now be found and a quick call to opam switch will also show that its existence has been registered by opam:

$ ls
_opam
$ opam switch
#  switch                        compiler             description
→  /home/ocamler/dev/helloer     ocaml.4.14.1         /home/ocamler/dev/helloer
   my-switch                     ocaml-system.4.14.1  my-switch

[NOTE] Current switch has been selected based on the current directory.
       The current global system switch is my-switch.

opam indicates that it has selected the local switch as the currently active one with the → character and then tells us that the currently active global switch outside of this directory is still a previously created one called my-switch.

Local switches were explained in detail in this section of Opam 101. We learned in it that opam automatically selects the local switch as the currently active one as soon as we move inside the directory in which it was created.

A quick call to opam list will show us what packages are currently installed in our switch. For now, all we have are the dependencies of the OCaml compiler:

$ opam list
# Packages matching: installed
# Name        # Installed # Synopsis
base-bigarray base
base-threads  base
base-unix     base
ocaml         4.14.1      The OCaml compiler (virtual package)
ocaml-config  2           OCaml Switch Configuration
ocaml-system  4.14.1      The OCaml compiler (system version, from outside of opam)

A quick call to which ocaml will confirm that the path to our compiler points to a location outside of the opam installation in our filesystem. The path does not point to either the global ~/.opam nor /home/ocamler/dev/helloer/_opam directories.

$ ocaml -vnum
4.14.1
$ which ocaml
/usr/bin/ocaml

In the case of a global switch, the following would be true:

$ ocaml -vnum
4.14.1
$ which ocaml
/home/ocamler/.config/opam/my-global-switch/bin/ocaml

---

Since we have a compiler, let's compile a small program.

$ cat helloer.ml
let () =
  print_endline "Hello OCamlers!!"
$ ocamlc -o hello helloer.ml
$ ./hello
Hello OCamlers!!

Granted, this is not a very interesting project to distribute yet.

In order for us to make things a bit more interesting, we should look into installing and incorporating external utilities, like libraries, into helloer.

Package Selection: opam search

Selecting an adequate external library to facilitate an engineering effort is one of the dev's most important skill. This step will take less and less time as you get familiar with the distribution and pick your own favourite tools for all kinds of engineering goals.

All these solutions are available for you to browse with the opam CLI (opam search, opam show), or on either of the official opam or ocaml websites.

Choose a build system

The first tool to look for would arguably be a build system.

If you are already familiar with the OCaml distribution, the first one to come to mind is most likely Dune, since it's the most ubiquitous.

We choose Dune this time around, but keep in mind that, as well as for the broader OCaml Distribution, alternative solutions and complementary tooling are maintained year-round by an active community of dedicated Camleers. Make sure to experiment and reach out to the community.

We believe that introducing you to the current most common practices of the OCaml Community is a solid way to get you going.

If you happen to look for guidance or any kind of support for your OCaml developments, keep in mind that the Discuss OCaml Community Forum is the best place to engage with your peers!

Find command-line libraries

Let's browse the distribution for packages that will help us in implementing a neat command-line interface for helloer.

We know that the OCaml Standard Library ships an Arg module which aims at allowing the parsing of command-line arguments. However, this module is quite basic. Using only it will make it very tedious for us to tailor our CLI to our needs, we have no choice but to browse the OCaml Distribution for an external CLI library.

Using opam, a simple opam search with your keywords might help you greately:

$ opam search "command line interface"
# Packages matching: match(*command line interface*)
# Name                  # Installed # Synopsis
bap-byteweight-frontend --          BAP Toolkit for training and controlling Byteweight algorithm
clim                    --          Command Line Interface Maker
cmdliner                --          Declarative definition of command line interfaces for OCaml
dream-cli               --          Command Line Interface for Dream applications
hg_lib                  --          A library that wraps the Mercurial command line interface
inquire                 --          Create beautiful interactive command line interface in OCaml
kappa-binaries          --          Command line interfaces of the Kappa tool suite
minicli                 --          Minimalist library for command line parsing
ocal                    --          An improved Unix `cal` utility
ocamline                --          Command line interface for user input
wcs                     --          Command line interface for Watson Conversation Service

cmdliner is one of our favourite libraries for that matter so let's use it in helloer.

Use test libraries

Finally, before we get to coding our little project, we should consider adding a test library to our project. This will make writing tests much easier, less time consuming and tedious.

Again, calling opam search with one or several keywords will yield many packages that pertain to testing OCaml binaries. Our selection for today will be alcotest, a well-known and wide-spread option for conducting tests on ocaml binaries.

$ opam search "test"
# Packages matching: match(*test*)
# Name                              # Installed # Synopsis
afl-persistent                      --          Use afl-fuzz in persistent mode
ahrocksdb                           --          A binding to RocksDB
alcotest                            --          Alcotest is a lightweight and colourful test framework
alcotest-async                      --          Async-based helpers for Alcotest
alcotest-js                         --          Virtual package containing optional JavaScript dependencies for Alcotest
alcotest-lwt                        --          Lwt-based helpers for Alcotest
alcotest-mirage                     --          Mirage implementation for Alcotest
[...]

---

Now that we have found the few packages we needed for our current project to grow smoothly, we can install them and start learning how they work!

A call to opam install will change the state of our current switch by installing these three new packages:

$ opam install dune cmdliner alcotest

Getting started

helloer is a toy project -- with it we can play around with the tools at hand and learn a thing or two about them.

We will learn the fundamentals about how to build an OCaml project with dune and we will see how integrating external libraries and running tests are done.

However, if you're curious about the source code of the project, you can check it out right here.

Do keep in mind that this is rather remote from what you will encounter in the wild. Both the code base and the structure of the repository are made intentionally bare-bones. This will allow us to smoothly introduce what we believe are the most common features a beginner OCaml dev should get familiar with.

---

Building, running and testing your project with Dune

dune-project

Let's first start with the dune-project file since every Dune-driven project should have one at its root.

This file is the entry point for your project — it's how Dune knows you're working in a structured project.

$ cat dune-project
(lang dune 3.15)
(package (name helloer))

Purposes:

• Declares your project name, version, and many more things.
• Specifies which Dune language version you're using (minimal compatibility).
• Can configure features like package names for opam, documentation, and more.
• It can generate the opam file for you. Lookout for the first line of some package.opam files: # This file is generated by dune, edit dune-project instead

---

dune file

A dune file is a build specification file, typically found in each subdirectory of your project. Since our toy helloer project needs no subdirectory, we will put it at the root of the repository.

$ cat dune
(library
 (name helloer_lib)
 (modules helloer_lib)
)

(executable
 (public_name helloer)
 (name helloer)
 (libraries cmdliner helloer_lib)
 (modules helloer)
)

(test
 (name test)
 (libraries alcotest helloer_lib)
 (modules test)
)

What it does:

• Tells dune how to build the OCaml files in that directory.
• Defines libraries, executables, test targets, and more.

This dune file defines three stanzas: a library stanza, an executable stanza and a test stanza.

We could write an entire blogpost for each section of this file but for the sake of brievety, you will have to settle for quick breakdowns for the moment.

library stanza:

(library
 (name helloer_lib)
 (modules helloer_lib)
)

What it does:

• Defines a library named helloer_lib.
• This will build it from the file named helloer_lib.ml.
• Only the exposed modules should be listed here (in this case, just helloer_lib).

OCaml module names should match the filename, so helloer_lib.ml is expected to exist in this directory.

executable stanza:

(executable
 (public_name helloer)
 (name helloer)
 (libraries cmdliner helloer_lib)
 (modules helloer)
 (promote (until-clean))
)

What it does:

• name: builds an executable named helloer.
• Needs libraries cmdliner (for CLI parsing) and helloer_lib (our own library).
• public_name helloer: This makes the executable available publicly. It is used for dune install helloer in the opam file for instance
• promote (until-clean): If you use expect tests, this causes output files to be "promoted" into the source tree until the next clean, useful in test output comparison.

test stanza:

(test
 (name test)
 (libraries alcotest helloer_lib)
 (modules test)
)

What it does:

• Declares a test target named test, defined in the file test.ml.
• Uses the alcotest testing library.
• Also uses helloer_lib to test its functionality.

---

dune build

As you can see below, the dune build @all command will build all targets defined in your dune files.

$ tree
.
├── dune
├── dune-project
├── helloer_lib.ml
├── helloer.ml
├── helloer.opam
└── test.ml
$ dune build @all

$ tree -L 2
.
├── _build
│   ├── default
│   │   ├── helloer.exe      // executable in its build dir
│   │   ├── helloer_lib.cmxs // built library
│   │   ├── test.exe         // test executable
│   │   └── [...]
│   ├── install
│   └── log
├── dune
├── dune-project
├── helloer.exe    // executable at the root 
├── helloer_lib.ml
├── helloer.ml
├── helloer.opam
└── test.ml

What it does:

• @all is an alias that includes all buildable targets defined in your dune files: executables, libraries, tests, docs, etc.
• Useful for doing a full build to ensure everything compiles.

You can also use custom aliases (like @doc, @runtest, etc.), or define your own in your dune files.

---

dune exec --

This command is used to run executables that are part of your OCaml project:

dune exec -- ./path/to/executable

So, something like:

$ dune exec -- ./helloer.exe
Hello OCamlers!!                   
$ dune exec -- ./helloer.exe --gentle
Welcome my dear OCamlers.          

Will tell dune to build the executable if it's not already built, and then run it. The -- separates the dune options from the executable and its arguments. Everything after -- is passed to the program you’re running.

---

dune runtest

This command runs your test suite(s).

$ dune runtest
Testing `Tests'.                 
This run has ID `N39NJ5ZE'.

  [OK]          messages          0   normal.
  [OK]          messages          1   gentle.

Full test results in `~/ocamler/dev/helloer/_build/default/_build/_tests/Tests'.
Test Successful in 0.000s. 2 tests run.

What it does:

• Builds test targets defined in your project.
• Looks for files ending in .t or .ml files marked as tests.
• Executes the tests, often using expect style testing (like ppx_expect or alcotest).

It's quite straight forward: if you have an inline_tests stanza or an expect test, it will run them and tell you if anything failed.

---

At this point in the development process, we can assume that you know how to use the most basic command-lines to make your OCaml projects a reality!

All that is left for us to do is learn about what opam needs for you to make your project opam compliant. This means: how to write an opam file and what information goes into it, and in time (in the next blogpost), how to distribute your newly developed package to the rest of the OCaml Community on the opam-repository!

Your first opam file

So how exactly does one write an opam file?

You can find all the relevant documentation about all fields found in a proper opam file right here, on the opam file documentation page.

Minimal requirements for a functional opam file

You'll find below a minimal opam file for the helloer project.

This file is minimal in the sense that it is complete enough for you to work with your package on your local environment, but there remains a few fields that we will explain in a moment and that are necessary for you to distribute your code.

$ cat helloer.opam
opam-version: "2.0"
depends: [
  "cmdliner"
  "ocaml"
  "alcotest" {with-test}
]
build: [
 [ "dune" "build" "-p" name ]
 [ "dune" "runtest" ] {with-test}
]
install: [ "dune" "install" ]

For now, there is enough information that opam to install your OCaml project locally : how to build and install your project.

What are these fields for?

opam-version: "2.0":

• Specifies that this opam file uses syntax compatible with opam 2.0 or later.
• Required at the top of every opam file.

depends: [...]:

• Lists the packages that your project depends on. It lists all necessary information to build your project, and to help opam suggest what other packages have to be installed prior to it.
  • You could optionally set lower or upper bound for specific version range (e.g., cmdliner {>= "1.0.0"}), but omitting that is fine for a minimal file like this one.

build: [...]:

• Tells opam how to build your project.
• ["dune" "build" "-p" name]: Builds the package. -p name means "build the part of the project with the same name as the opam package".

  name here should match the actual opam package name; it’s often replaced automatically by dune when generating the file.

• ["dune" "runtest"] {with-test}: Runs the test suite, only if --with-test is passed (e.g. during CI or development).
• You can fill up with any command here, it will be launched by opam in a sandboxed environment.

install: [...]:

• This tells opam how to install the built binaries and libraries into the opam environment.
• Dune will install any libraries, executables, and other files you've marked as public_name.

That's it for the essential fields of an opam file. Now onto the metadata fields which are required for you to later distribute your package through the Official OCaml opam-repository.

An opam file supports about thirty-ish valid fields to specify a package, again, you can look them all up here.

Requirements for a distributable opam package

What fields are mandatory for a proper opam package?

A good question, and the answer is simple since opam features a linting command with opam lint.

This means that running this command on our small opam file will yield the following message to help us make sense of what more is required to make our newly developped package distributable:

$ opam lint .
/home/ocamler/dev/helloer/helloer.opam: Errors.
    error 23: Missing field 'maintainer'
  warning 25: Missing field 'authors'
  warning 35: Missing field 'homepage'
  warning 36: Missing field 'bug-reports'
    error 57: Synopsis must not be empty
  warning 68: Missing field 'license'

As you can see, some of these missing fields are considered errors, and others are considered a mere warnings. Each also come with a designated error code. You can find all warning and error codes by either running the opam lint --help command in your terminal, or going to the corresponding opam man page on the interwebs.

Lets break each of them down and see how these linting errors can be fixed.

• Error 23: Missing field maintainer
  What: Contact email for the package maintainer.
  Example:

  maintainer: "hell@er.com"
  

• Error 57: Synopsis must not be empty
  What: One-line description of your project.
  Example:

  synopsis: "A simple command-line greeter"
  

• Warning 25: Missing field authors
  What: List of project authors.
  Example:

  authors: ["Your Name"]
  

• Warning 35: Missing field homepage
  What: URL to your project's website or repository.
  Example:

  homepage: "https://github.com/OCamlPro/opam_bps_examples"
  

• Warning 36: Missing field bug-reports
  What: URL for reporting issues.
  Example:

  bug-reports: "https://github.com/OCamlPro/opam_bps_examples/issues"
  

• Warning 68: Missing field license
  What: License under which your project is distributed.
  Example:

  license: "MIT"
  

---

TODO:

You can also generate an opam file with dune

Conclusion and what's next

Mon code compile, mes tests passent, ma doc génère, hourray, tout va bien.

Maintenant notre projet est paré. On peut être sûr que notre projet va fonctionner avec opam en local, qu'il pourra etre pinné dans notre switch sans soucis. On a fait le tour des commandes de base pour compiler un projet ocaml avec dune et construire un opam file qui déchire.

La prochaine étape ? Le distribuer via l'opam-repository OCaml officiel.

On parlera de:

• community driven collaboration
  • ouvrir une pr, collaborer, itérer et patienter
• la forme d'un fichier opam dans le repository
  • versionnage, compatibilité, opam-lock et que sais-je
• l'opam ci et la maintenance du repo par la fondation

Merci d'avoir lu jusqu'a maintenant et on espère que les plus néophytes s'y retrouveront pour embarqer confortablement la caravane OCaml !

---

About OCamlPro:

OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.

• We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
• We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
• We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
• We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.

Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.