Opam 103: Starting a new Project

---

Welcome back to the opam deep-dives series!

Ever since the first episode in the series have our readers asked for us to explore the dev side of the opam experience. Needless to say that it has always been in our plans to do so, however, for the sake of clarity and accessibility, we had to deal with user-facing scenarios first as neat introductions to the general subject matter.

We warmly thank you for your patience! Your wait was not in vain, because today, we start a new project in an opam-encompassing workflow! 🚀🚀

That being said, expect this first dev-side tutorial on opam to be most useful to the newer OCaml devs out there. 😇

Be sure to read the other episodes available at this point in time: Opam 101: The First steps, which introduces you to the fundamentals of opam, from installation to exploration, and Opam 102: Pinning Packages which already dives quite deep into package pinning, one of the first keys to tailoring your workflow and environment to your exact needs.

Also, check out each article's tags to get an idea of the entry level required for the smoothest read possible!

New to the expansive OCaml sphere? As said on the official opam website, opam has been a game changer for the OCaml distribution, since it first saw the day of light here, almost a decade ago.

---

Contextual Requisites

The goal of this specific blogpost is to take you on a journey which starts with creating a directory for your new OCaml project on your filesystem and ends with publishing it on the Official opam-repository.

We will aim at clarifying at every step of the way what is good to keep in mind and try to paint a most realistically exhaustive and concise opam dev-side scenario. We will cover switch creation, package selection and incorporation, code sharing, dependencies locking and distribution.

As far as the minimum required familiarity with opam goes for you to be able to fully enjoy this tutorial, we recommend only for you to have read our first article: Opam 101: The First Step and especially the section that explains what a switch is.

Nevertheless, here's a quick TL;DR for those of you who would rather get started:

What is an opam switch?

Conceptually, it's the environment of the OCaml dev. Opam provides you with a command-line interface for you to customise, and maintain a safe and stable environment. It's defined by all the possible combinations and valid operations between a specific version of the OCaml compiler, and any set of versioned packages.

Functionally, it is set of environment variables that are user-updated and point to the different locations of installed versions of packages, binaries and other utilities either in a ~/.opam directory for global switches or in the current _opam directory for local ones.

Link to the official documentation.

Since there are two different kinds of switches, and since today's subject matter is bounded to making a new project, to keep things simple, we will use local switches in our examples.

Ready? Let's go!

Setting up the environment

Unsurprisingly, the first step on the journey to publishing your very own OCaml package, is to set it up to be developed... 🤯

This encompasses everything from creating the working directory of your new project, to setting up a custom, local switch for it.

We will consider that you have created a new directory for your project and have since moved into it in order to progress further in the setup process.

Something like:

$ mkdir my-project
$ cd my-project

Here are the things that opam will help you accomplish at this stage of the development process, we will do our best to explain them in the upcoming section:

• Setting up a new switch (i.e, environment creation);
• Browsing the OCaml package distribution for libraries and tooling in general (i.e, technical exploration);
• Selection and installation of OCaml software inside a switch (i.e, environment setup and tailoring);
• Contraint-based package compatibility calculations, which entails automatic solving of package dependency trees (i.e, automatic environment safety verifications);

Creating a new local switch

$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.

A switch is the virtual environment in which opam will operate and assist you in taking all necessary steps when coming up with an optimal enough workflow.

As said previously, a switch is comprised of a specific version of the OCaml compiler and a set of packages that are compatible with that specific version.

So let's first create a local switch in our my-project directory.

$ opam switch create .

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><><><>
Switch invariant: ["ocaml" {>= "4.05.0"}]

We let opam select the default switch invariant when creating a new switch which is OCaml compiler version >= 4.05.0. You can define any set of switch invariants that you wish.

In the call above, the . character is indicative that we are asking opam to create a switch inside the current directory, a local switch as opposed to a global one by giving opam a path.

The idea of switch invariants is quite simple, they are the parameters to the automatic solving of package dependency trees and they are immutable. Opam will never change invariants without notifying you first and will always consider the switch invariants when building the graph of available and compatible packages for your current switch, or for any other switch-altering operation for that matter.

So, back to our example:

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-system.4.14.1
∗ installed ocaml-config.2
∗ installed ocaml.4.14.1
Done.

We can see that opam selected ocaml-system.4.14.1 as opposed to ocaml-base-compiler.4.14.1 as the OCaml compiler to install in your current local switch along with its dependencies.

The difference between these two compilers is that ocaml-system is a system-bound compiler, typically one installed outside of your opam installation; e.g. with the help of the package manager of your favourite OS. On the other hand, ocaml-base-compiler would be a new compiler installed within your opam installation, one that opam would have permission over.

If you recall this section of Opam 101, you should know that creating a switch can be a fairly time-consuming task depending on whether or not the compiler version you have queried from opam is already installed somewhere on your machine, every time you ask opam to install a version of the compiler, it will first scour your installation for a locally available version of that compiler to save you the time necessary for downloading, compiling and installing a brand new one. This is the reason why opam has selected an ocaml-system.4.14.1 compiler instead of installing a brand new ocaml-base-compiler.4.14.1.

A quick look at our current directory will show that an _opam directory can now be found and a quick call to opam switch will also show that its existence has been registered by opam:

$ ls
_opam
$ opam switch
#  switch                        compiler             description
→  /home/ocamler/dev/my-project  ocaml.4.14.1         /home/ocamler/dev/my-project
   my-switch                     ocaml-system.4.14.1  my-switch

[NOTE] Current switch has been selected based on the current directory.
       The current global system switch is my-switch.

opam indicates that it has selected the local switch as the currently active one with the → character and then tells us that the currently active global switch outside of this directory is still a previously created one called my-switch.

Local switches were explained with enough detail in this section of Opam 101. We learned in it that opam automatically selects the local switch as the currently active one as soon as we move inside the directory in which is was created.

A quick call to opam list will show us what packages are currently installed in our switch. For now, all we have are the dependencies of the OCaml compiler:

$ opam list
# Packages matching: installed
# Name        # Installed # Synopsis
base-bigarray base
base-threads  base
base-unix     base
ocaml         4.14.1      The OCaml compiler (virtual package)
ocaml-config  2           OCaml Switch Configuration
ocaml-system  4.14.1      The OCaml compiler (system version, from outside of opam)

A quick call to which ocaml will confirm that the path to our compiler points to a location outside of the opam installation in our filesystem. The path does not point to either the global ~/.opam nor /home/ocamler/dev/my-project/_opam directories.

$ ocaml -vnum
4.14.1
$ which ocaml
/usr/bin/ocaml

In the case of a global switch, the following would be true:

$ ocaml -vnum
4.14.1
$ which ocaml
/home/ocamler/.config/opam/my-global-switch/bin/ocaml

---

Since we have a compiler, let's compile a small program.

$ cat helloer.ml
print_endline "Hello OCamlers!!"
$ ocamlc -o hello helloer.ml
$ ./hello
Hello OCamlers!!

Granted, this is not a very interesting project to distribute yet.

In order for us to make things a bit more interesting, we should look into installing and incorporating external utilities, like libraries, into my-project.

Package Selection

Build system

The first tool to look for would arguably be a build system.

If you are already familiar with the OCaml distribution, the first one to come to mind is most likely Dune, since it's the most ubiquitous.

We choose Dune this time around, but stay aware that, as well as for the broader OCaml Distribution, alternative solutions and complementary tooling are maintained year-round by an active community of dedicated Camleers. Make sure to experiment and reach out to the community.

We believe that introducing you to the current most common practices of the OCaml Community is a solid way to get you going.

If you happen to look for guidance or any kind of support for your OCaml developments, keep in mind that the Discuss OCaml Community Forum is the best place to engage with your peers!

Command-line tools

Let's browse the distribution for packages that might help us in implementing a neat command-line interface.

Selecting an adequate external library to facilitate an engineering effort is one of the dev's most important skill. This step will take less and less time as you get familiar with the distribution and pick your own favourite tools for all kinds of engineering goals.

All these solutions are available for you to browse with the opam CLI (opam search, opam show), or on either of the official opam or ocaml websites.

Let's resume our first engineering goal, which is to implement a simple command-line interface for my-project.

We know that the OCaml Standard Library ships an Arg module which aims at allowing the parsing of command-line arguments. However, this module is quite basic. Using only it will make it very tedious for us to tailor our CLI to our needs, we have no choice but to browse the OCaml Distribution for an external CLI library.

Using opam, a simple opam search with your keywords might help you greately:

$ opam search "command line interface"
# Packages matching: match(*command line interface*)
# Name                  # Installed # Synopsis
bap-byteweight-frontend --          BAP Toolkit for training and controlling Byteweight algorithm
clim                    --          Command Line Interface Maker
cmdliner                --          Declarative definition of command line interfaces for OCaml
dream-cli               --          Command Line Interface for Dream applications
hg_lib                  --          A library that wraps the Mercurial command line interface
inquire                 --          Create beautiful interactive command line interface in OCaml
kappa-binaries          --          Command line interfaces of the Kappa tool suite
minicli                 --          Minimalist library for command line parsing
ocal                    --          An improved Unix `cal` utility
ocamline                --          Command line interface for user input
wcs                     --          Command line interface for Watson Conversation Service

cmdliner is one of our favourite libraries for that matter so let's use it in my-project.

---

A call to opam install will change the state of our current switch by installing these two new packages:

$ opam install dune cmdliner
The following actions will be performed:
  ∗ install dune     3.15.0
  ∗ install cmdliner 1.2.0
===== ∗ 2 =====

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved cmdliner.1.2.0  (https://opam.ocaml.org/cache)
⬇ retrieved dune.3.15.0  (https://opam.ocaml.org/cache)
∗ installed cmdliner.1.2.0
∗ installed dune.3.15.0
Done.

Note pour raja: Récupérer des versions des libs qui ne sont pas les dernières. Pour que ça puisse donner un exemple non-déterminisme dans la version effectivement installée localement a la machine du dev, ça justifiera le passage par la section opam lock

Package Incorporation

Un mot sur l'integration de paquet au dev par dune pour dev tout seul

Je commence à coder.

dune-project, peut-être quelques conseils pour mettre le pied à l'étrier dune. le dossier _build etc.

$ cat src/dune
(executable
 (name main)
 (libraries some-lib)
 (modules module_one module_two main))

(env
 (dev
  (flags
   (:standard -warn-error -A))))

• opam switch
  • focus on locally available packages to make sure you can compile your own code.

J'ai besoin d'un librairie. opam search topic + lookup dans ocaml.org Note pour raja: récup des versions de libs pas les dernières

Juste pour trouver et utiliser les paquets désirés (overlapping avec ocbp): - faire un switch local: toutes les commandes idoines, ocaml.org

J'ai besoin de rajouter des test & de la doc (lookup de lib)

Je cherche une lib de test de la même manière que pour la CLI. Je prend alcotest. J'ecris mon test et là mon code compile, mes tests passent, hourray, tout va bien.

$ ls
dune  dune-project  helloer_lib.ml  helloer.ml  _opam  test.ml
$ dune build
$ ./helloer.exe
Hello OCamlers!!
$  ./helloer.exe --gentle
Welcome my dear OCamlers.
$ dune runtest
Testing `Tests'.
This run has ID `PNLOZG72'.

  [OK]          messages          0   normal.
  [OK]          messages          1   gentle.

Full test results in `/home/ocamler/dev/my-project/_build/default/_build/_tests/Tests'.
Test Successful in 0.000s. 2 tests run.

Je veux commencer à bien former mon projet, j'ecris un opam file Avec ses deps, ses deps tests/doc et instructions. * introduction de opam lint

• populate opam file to make sure other people can compile your code
  • By hand
    • expliquer chaque champs et leur utilisation par opam
    • parler des champs de dependance de test, et de generation de doc ( You can also generate an opam file with dune )

Juste pour trouver et utiliser les paquets désirés (overlapping avec ocbp): - faire un switch local: toutes les commandes idoines, ocaml.org

J'ai besoin de rajouter des test & de la doc (lookup de lib)

Mon code compile, mes tests passent, ma doc génère, hourray, tout va bien.

Sharing the code

4. Another dev want to participate o/

   prérequis: avoir le projet d'exampls sur github/ocp

• Un autre dev arrive:
  • Récupération du code avec git clone
  • Creates a compatible dev environment with a single command:
    • opam install . --deps-only (option pour les devs)
    • opam install . --deps-only --with-test (option pour les devs)
    • opam install . --deps-only --with-doc (option pour les devs)

5. Lock your dependencies, for a better compatibility dev setup

Comme vous le voyez, les dépendances ne sont exactement les mêmes installées, c'est possible de l'approcher avec le magnifique opam lock. Lors de la mise en place du fichier opam, il est possible de générer à tout moment un fichier de lock. L'idée derrière opam lock, c'est que : je suis entrain de développer. J'ai tout qui marche comme je veux dans mon switch. Opam lock me génère un fichier opam, qui contient toutes les dépendances de mon projet à la version exacte. * opam lock . ou opam lock my-package Du coup, lors du 4. le dev peut prendre le fichier lock avec opam install . --deps --locked et installer les mêmes versions de paquet.

Distributing the code

6. Publication

   Note: À voir si on introduit opam lint ici ou en 3. avec la description du fichier opam Derniers avec opam lint

   • publier avec opam-publish, il y a besoin de
     • une url
     • une archive
     • et opam publish d'installé opam publish xxx
     • explication de ce qui se passe dans le détails
   • est-ce qu'on parle de dune release ? Non, car trop spécifique à dune.

---

About OCamlPro:

OCamlPro is a R&D lab founded in 2011, with the mission to help industrial users benefit from experts with a state-of-the-art knowledge of programming languages theory and practice.

• We provide audit, support, custom developer tools and training for both the most modern languages, such as Rust, Wasm and OCaml, and for legacy languages, such as COBOL or even home-made domain-specific languages;
• We design, create and implement software with great added-value for our clients. High complexity is not a problem for our PhD-level experts. For example, we helped the French Income Tax Administration re-adapt and improve their internally kept M language, we designed a DSL to model and express revenue streams in the Cinema Industry, codename Niagara, and we also developed the prototype of the Tezos proof-of-stake blockchain from 2014 to 2018.
• We have a long history of creating open-source projects, such as the Opam package manager, the LearnOCaml web platform, and contributing to other ones, such as the Flambda optimizing compiler, or the GnuCOBOL compiler.
• We are also experts of Formal Methods, developing tools such as our SMT Solver Alt-Ergo (check our Alt-Ergo Users' Club) and using them to prove safety or security properties of programs.

Please reach out, we'll be delighted to discuss your challenges: contact@ocamlpro.com or book a quick discussion.