Formalising external APIs

Gleam v1.13.0 released

a soft wavey boundary between two sections of the website
Return Icon

by Louis Pilfold

Gleam is a type-safe and scalable language for the Erlang virtual machine and JavaScript runtimes. Today Gleam v1.13.0 has been published. Let's go over at the highlights now.

External API for Gleam data

One of Gleam's strengths is that it is part of the BEAM and JavaScript ecosystems, enabling Gleam programs to take advantage of code written in Erlang, Elixir, JavaScript, and more. This is a large part of how Gleam was able to become a practical production-ready language so quickly, by not restricting Gleam programmers to just the comparatively young Gleam package ecosystem.

A function written in one of these other languages can be imported into a Gleam module as an "external function", and then called without any additional performance overhead.

@external(erlang, "moon_base", "launch_spaceship")
pub fn lift_off(countdown: Int) -> Result(Spaceship, LaunchError)

Here the Erlang function launch_spaceship from the module moon_base is being imported.

One restriction to external functions is that they will need to return data types that Gleam can understand. Gleam's "external type" feature lets the programmer refer to types defined in other languages, but if the programmer wants to be able to directly construct and pattern match data from external functions it will need to be in a format compatible with Gleam data types.

Often a programmer will write a small wrapper function in the external language to convert the data, and also to make the interface adhere to Gleam patterns and conventions.

-module(moon_base).
-export([launch_spaceship/1]).

launch_spaceship(Countdown) ->
    try
        Spaceship = launch_control:launch_spaceship(Countdown),
        {ok, Spaceship}
    catch
        error:no_fuel -> {error, no_fuel};
        error:bad_weather -> {error, bad_weather}
    end.

This launch_spaceship Erlang function wraps the function from the launch_control module, converting the exception-based API into a Result type, the Erlang representation of Gleam's Ok and Error variants.

One thing that a Gleam programmer may find challenging is knowing how to construct Gleam data in these wrapper functions. A lack of detailed documentation made it unclear what the correct approach should be.

This lack of clarity makes learning how to use external code more challenging, and worse, it may result in programmers using internal APIs that are intended only to be used by the compiled Gleam code. If the Gleam ecosystem were to grow with many packages using these internal APIs, it would force the Gleam core team to support them as if they were public APIs. Committing to these APIs would greatly limit what changes we can make to the internal representation of Gleam data, and make many potential performance improvements impossible.

To fix this we have done two things. First, we have created a guide on Gleam externals, detailing how to correctly write and use externals.

Secondly, a dedicated API is now provided for JavaScript based code to work with Gleam data, both making usage clearer and giving the Gleam core team maximum freedom to improve performance in future. Each data type defined in Gleam will have a set of functions defined to work with it, for example:

// In src/person.gleam
pub type Person {
  Teacher(name: String, subject: String)
  Student(name: String)
}

// In src/my_javascript_code.mjs
import {...} from "./person.mjs";

// Constructing custom types
let teacher = Person$Teacher("Joe Armstrong", "Computer Science");
let student = Person$Student("Louis Pilfold");

let randomPerson = Math.random() > 0.5 ? teacher : student;

// Checking variants
let randomIsTeacher = Person$isTeacher(randomPerson);

// Getting fields
let teacherSubject = Person$Teacher$subject(teacher);

// The `name` field is shared so can be accessed from either variant
let personName = Person$name(randomPerson);

There will be a migration period where existing JavaScript externals will need to migrate over to the new API. We have created tooling to analyse the Gleam package ecosystem to identify code that is in need of updating, and we will be helping with this process.

Further additions will be made to the externals guide detailing useful patterns, how to avoid common problems, and advising when and how to use externals.

Thank you Surya Rose for taking the lead role in implementing these new APIs, and for the Gleam team more widely for the design of this addition!

Improved bit array exhaustiveness checking

Gleam's bit array syntax allows you to declaratively construct and parse binary data in a way that may be easier to understand than using binary operators.

The compiler now applies an optimisation known as "interference based pruning" when compiling bit array pattern matching where matches are performed at the start of bit arrays. This optimisation drastically reduces compile times, memory usage and the compiled code size, removing many redundant checks.

It is particularly impactful for programs that pattern match on some fixed patterns at the start of the bit array. For example, network protocol parsers.

pub fn parser_headers(headers: BitArray, bytes: Int) -> Headers {
  case headers {
    <<"CONTENT_LENGTH" as header, 0, value:size(bytes), 0, rest:bytes>>
    | <<"QUERY_STRING" as header, 0, value:size(bytes), 0, rest:bytes>>
    | <<"REQUEST_URI" as header, 0, value:size(bytes), 0, rest:bytes>>
    // ...
    | <<"REDIRECT_STATUS" as header, 0, value:size(bytes), 0, rest:bytes>>
    | <<"SCRIPT_NAME" as header, 0, value:size(bytes), 0, rest:bytes>>
      -> [#(header, value), ..parse_headers(rest)]
  }
}

Additionally, the compiler now raises a warning for unreachable branches that are matching on bit array segments that could never match. Consider this example:

pub fn get_payload(packet: BitArray) -> Result(BitArray, Nil) {
  case packet {
    <<200, payload:bytes>> -> Ok(payload)
    <<404, _:bits>> -> Error(Nil)
    _ -> Ok(packet)
  }
}

There's a subtle bug here. The second branch can never match since it's impossible for the first byte of the bit array to have the value 404. The new error explains this nicely:

warning: Unreachable pattern
  ┌─ /src.gleam:4:5
  │
4 │     <<404, _:bits>> -> Error(Nil)
  │     ^^^^^^^^^^^^^^^
  │       │
  │       A 1 byte unsigned integer will never match this value

This pattern cannot be reached as it contains segments that will never
match.

Hint: It can be safely removed.

Thank you Giacomo Cavalieri for these improvements! Exhaustiveness checking is a very complex field, so these additions are very impressive.

Unused argument detection

Gleam's unused code detection and purity tracking emits a warning any time some code is unused and could be removed without changing the behaviour of the program.

This has been extended to be able to identify function arguments that are used when the function calls itself recursively, but never actually used in the function's implementation. For example:

import gleam/io

pub fn greet(x, times) {
  case times {
    0 -> Nil
    _ -> {
      io.println("Hello, Joe!")
      greet(x, times - 1)
    }
  }
}

In this piece of code the x argument is unused, so the compiler will raise the following warning:

warning: Unused function argument
  ┌─ /Users/giacomocavalieri/Desktop/prova/src/prova.gleam:3:14
  │
3 │ pub fn greet(x, times) {
  │              ^ This argument is unused

This argument is passed to the function when recursing, but it's never
used for anything.

Thank you Giacomo Cavalieri!

Better meaningless opaque type error

A public custom type can be marked as "opaque", meaning that while other modules can import and reference the type, they are unable to construct or pattern match on values of that type. This is useful for restricting the ways that a data type can be used in order to provide a more robust API.

pub opaque type Permission {
  SuperUser
  Regular
  Restricted
}

It is invalid to mark a private type as opaque. Previously this would result in a slightly cryptic syntax, but now a specific helpful error has been added for this case.

error: Private opaque type
  ┌─ /src/one/two.gleam:2:1
  │
2 │ opaque type Wibble {
  │ ^^^^^^ You can safely remove this.

Only a public type can be opaque.

The language server now also offers a "quick fix" code action to remove opaque from a private type:

opaque type Wibble {
// ^^^ This is an error!
  Wobble
}

If you hover over the type and trigger the quick fix, the language server will automatically remove the opaque keyword:

type Wibble {
  Wobble
}

Thank you Giacomo Cavalieri!

More fault tolerance

Gleam's compiler implements fault tolerant analysis. This means that when there is some error in the code the compiler can still continue to analyse the code to the best of its ability, ignoring the invalid parts. Because of this, the Gleam language server can have a good understanding of the code and provide IDE features even when the codebase is in an invalid state.

Giacomo Cavalieri and sobolevn) have improved the compiler to be fault tolerant for errors relating to analysis of labeled fields in variant patterns, parsing of private opaque type definitions, and parsing of type names followed by (), further improving the experience of using the Gleam language server.

Thank you both!

Redundant pattern alias warning

_ as x is a valid pattern in Gleam. The _ means "don't assign any name to this value", and the as x part means "assign the name x to this value".

As you can see, this is quite a silly pattern. The alias as pattern makes the discard _ pattern redundant, and it would always be better to use the pattern x, which means "assign this value to the name x".

// Redundant
let _ as x = something()

// Recommended
let x = something()

Using an alias pattern with a discard pattern has been deprecated, and the Gleam code formatter will rewrite any instances of it to the recommended syntax.

Thank you eutampieri for this!

More inefficient list check warnings

Gleam's basic sequence type is an immutable linked list with structural sharing, a data type inherited from Erlang and one common in functional programming languages.

The correct way to check if a list is empty is to pattern match on it with the empty-list pattern, or to compare it to an empty list literal.

pub fn both_empty(list1: List(a), list2: List(b)) -> Bool {
  // Pattern matching approach.
  // More verbose, but can be part of a larger pattern match.
  let list1_empty = case list1 {
    [] -> True
    _ -> False
  }

  // Comparison approach.
  let list2_empty = list2 == []

  list1_empty && list2_empty
}

The standard library's list.length function returns the length of a given list. Gleam and Erlang lists don't store the length on them as a static property, so this function has to traverse the full list, and count the number of elements, making it a very wasteful way to determine if a list is empty.

This behaviour may be suprising to programmers familiar with languages with a different core sequence data type, so they might not realise this is not a good way to check for an empty list. To remove this confusion the compiler would emit a warning for code like list.length(items) == 0, informing the programmer of better alternatives.

With this release the warning will also be emitted for more inefficient use of list.length, including checks for non-empty lists using operators like > and <.

warning: Inefficient use of `list.length`
  ┌─ /data/data/com.termux/files/home/test_gleam/src/test_gleam.gleam:5:13
  │
5 │     let _ = 0 < list.length(numbers)
  │             ^^^^^^^^^^^^^^^^^^^^^^^

The `list.length` function has to iterate across the whole
list to calculate the length, which is wasteful if you only
need to know if the list is empty or not.

Hint: You can use `the_list != []` instead.

Thank you Andrey Kozhev!

A helpful syntax error for JavaScripters

In Gleam names are assigned to values within functions using the let keyword. There is a const keyword too, but it is used to declare module-level constants, and it is a syntax error to use it within functions.

The compiler now provides an helpful error message for any programmers accustomed to languages where const is used within functions, such as JavaScript.

pub fn deep_thought() -> Int {
  const the_answer = 42
  the_answer
}

error: Syntax error
  ┌─ /src/file.gleam:2:3
  │
3 │   const the_answer = 43
  │   ^^^^^ Constants are not allowed inside functions

All variables are immutable in Gleam, so constants inside functions are not
necessary.
Hint: Either move this into the global scope or use `let` binding instead.

Thank you Surya Rose!

A helpful error for Rustaceans, C#ers, and friends

Gleam has a consistent syntax for constructors at the type and value level, using () for both. It does not use () for value constructors and <> for type constructors, as is common in some other languages.

To help folks coming from other languages a set of helpful errors have been added for when they try to use this non-Gleam syntax in their Gleam code.

error: Syntax error
  ┌─ /src/parse/error.gleam:2:12
  │
2 │ type Either<a, b> {
  │            ^ I was expecting `(` here.

Type parameters use lowercase names and are surrounded by parentheses.

    type Either(a, b) {

See: https://tour.gleam.run/data-types/generic-custom-types/

Notice how the error message includes the correct syntax for the specific code that the programmer has written. The programmer could copy/paste the correct version into their code, if they so desired. A link to the documentation is also provided, linking to whichever feature the syntax error is for.

Thank you Aaron Christiansen!

A helpful syntax error for Pythonistas, Elixirists, and friends

Similar to the last two new error messages, there's now a helpful error message for programmers trying to use # instead of // to write a comment. Thank you sobolevn!

Displaying dependency version information

Gleam's built tool integrates with Hex, the package management system and primary package repository for the BEAM ecosystem. Gleam implements dependency version locking to ensure that builds are deterministic, and to prevent unaudited code from unexpectedly becoming part of your application. Dependency code is just as much of a risk and responsibility as code directly written by the programmer, so it must be treated with great care and consideration.

The only time the build tool will select new versions of dependency packages is if the programmer adds or removes a dependency, if the programmer changes the package's dependency version requirements, or if the programmer requests the dependency versions be upgraded using the gleam update command.

In these cases when dependencies are changed, added, or removed the build tool will now print the changes, to help the programmer understand and go on to audit the new code.

$ gleam add lustre
  Resolving versions
Downloading packages
 Downloaded 3 packages in 0.04s
      Added gleam_json v3.0.2
      Added houdini v1.2.0
      Added lustre v5.3.5

hex owner transfer

The hex owner transfer command has been added to the build tool, allowing Gleam programmers to transfer ownership of existing Hex packages to another account. Thank you Giacomo Cavalieri!

Improved type displaying

When a Gleam package is published to Hex HTML documentation is generated and published to the HexDocs documentation hosting website. This documentation has now been improved to now print the names of public type aliases instead of internal type names when annotating functions and types. This makes the documentation more likely to use the APIs that the package author intends for their users.

For example, for the following code:

import my_package/internal

pub type ExternalAlias = internal.InternalRepresentation

pub fn do_thing() -> ExternalAlias { ... }

This is what the build tool used to generate:

pub fn do_thing() -> @internal InternalRepresentation

This is technically correct, but not very useful for the reader of the documentation, as they cannot learn anything about these internal types. Now it will not use the internal name, allowing the programmer to click-through to its documentation, and understand how they should refer to this type in their code.

pub fn do_thing() -> ExternalAlias

This improvement also applies to the language server, both in information displayed on hover, and in code actions such as "add annotations".

import lustre/html
import lustre/element
import lustre/attribute

pub fn make_link(attribute, element) {
  html.a([attribute], [elements])
}

If the "add annotations" code action is run on this function the language server will identify that the imported lustre/attribute module has public export of the Attribute type, and that the imported lustre/element module has public export of the Element type, so they will both be used instead of the internal definition.

pub fn make_link(
  attribute: attribute.Attribute,
  element: element.Element(a)
) -> element.Element(a) {
   html.a([attribute], [elements])
}

Thank you Surya Rose!

Improved type naming in code actions

Another way in which code actions need to consider type names is with type parameters. The "add type annotations" and "generate function" code actions must find names for any type variables that do not clash with any already in use. Previously the language server would track names in-use at the module level which could result in correct but unexpected names being used.

Take this code, for example.

fn something(a: a, b: b, c: c) -> d { todo }

fn pair(a, b) { #(a, b) }

Previously, when triggering the "Add type annotations" code action on the pair function, the language server would have used these names:

fn pair(a: e, b: f) -> #(e, f) { #(a, b) }

However in 1.13, it will now use these names:

fn pair(a: a, b: b) -> #(a, b) { #(a, b) }

Thank you Surya Rose!

Tangled support

Tangled is a new open source source forge, and the Gleam build tool now has support for it. When specified the HTML documentation will include links to the source code definitions for each type and value in the package, as it would for previously supported source forges such as GitHub and Forgejo.

repository = { type = "tangled", user = "me", repo = "my_project" }

Thank you to Naomi Roberts for this. As part of this work she added support for links to multi-line sections of code in Tangled itself!

Further language server support

The language server was missing support for a few syntaxes, this has now been fixed. You can now go to definition, rename, etc. from alternative patterns in case expressions:

case wibble {
  Wibble | Wobble -> 0
  //         ^- Previously you could not trigger actions from here
}

And hovering over a record field in a record access expression will now show the documentation for that field, if any exists.

Thank you Surya Rose and fruno!

Remove unreachable clauses code action

Gleam's pattern matching analysis can identify any clauses of a case expression are unreachable due to previous patterns already matching any values the redundant one could match.

pub fn main() {
  case find_user() {
    Ok(person) -> todo
    Ok(Admin) -> todo
    Ok(User) -> todo
    Error(_) -> todo
  }
}

Here the Ok(Admin) and Ok(User) patterns could never match as all the Ok values would be instead matched by the earlier Ok(person) pattern.

This is clearly a mistake in the code, so the compiler will emit a warning and highlight the unreachable clauses. Most commonly the programmer will want to edit the patterns to correct them, but some times the clauses are no longer needed, and can be deleted entirely. To help with this scenario the language server now offers a quick-fix code action to delete any redundant clauses.

Triggering it on the above code will result in the code being edited like so:

pub fn main() {
  case find_user() {
    Ok(person) -> todo
    Error(_) -> todo
  }
}

Thank you Giacomo Cavalieri!

Pattern match on value code action improvements

Gleam has a single flow-control feature: pattern matching with the case expression. Because of this there's a lot of pattern matching in Gleam programs! The language server offers a code action to quickly pattern match on a focused value, and with this release it has been further improved.

It can now be triggered on lists, with the default clauses including one for when the list is empty, and one for when it is non-empty.

pub fn is_empty(list: List(a)) -> Bool {
  //            ^^^^ Triggering the action here
}

Triggering the action over the list argument would result in the following code:

pub fn is_empty(list: List(a)) -> Bool {
  case list {
    [] -> todo
    [first, ..rest] -> todo
  }
}

The code action can now be triggered on variables introduced by other patterns. For example, here we have a let statement with a pattern defining the variables name and role.

pub fn main() {
  let User(name:, role:) = find_user("lucy")
  //              ^^^^ Triggering the action here
}

Triggering the action on role results in a case expression being inserted on the line below, with a clause for each of the possible variants of the Role type that variable held.

pub fn main() {
  let User(name:, role:) = find_user("lucy")
  case role {
    Admin -> todo
    Member -> todo
  }
}

If the variable was introduced within a case expression already then the behaviour is different. For example:

pub fn main() {
  case find_user() {
    Ok(user) -> todo
    Error(_) -> todo
  }
}

Triggering the code action on the user variable would cause the code to be rewritten to expand that clause of the case expression, replacing it with one specialised clause for each of the possible variants of the type of that variable.

pub fn main() {
  case find_user() {
    Ok(Admin) -> todo
    Ok(Member) -> todo
    Error(_) -> todo
  }
}

Thank you Giacomo Cavalieri! Pattern matching is such a important core feature of Gleam that these improvements make a big difference to the experience of writing and editing Gleam code.

Collapse nested case expressions code action

Another new code action is one to collapse nested case expressions into one, reducing nesting and enabling further optimisations in some situations.

case user {
  User(name:) ->
    case name {
      "Joe" -> "Hello, Joe!"
      _ -> "Hello there!"
    }
  Guest -> "You're not logged in!"
}

Triggering the code action on the first clause will result in it being replaced by multiple clauses that produce the same behaviour as the nested version.

case user {
  User(name: "Joe") -> "Hello, Joe!"
  User(name: _) -> "Hello there!"
  Guest -> "You're not logged in!"
}

Thank you Giacomo Cavalieri!

Add omitted labels code action

Function parameters and record fields can have labels, names that can be used at the call-site to make it clearer what each argument is, and to make the ordering of the arguments not matter. The language server now offers a code action to add the omitted labels in a call. For example:

pub type User {
  User(first_name: String, last_name: String, likes: List(String))
}

pub fn main() {
  let first_name = "Giacomo"
  User(first_name, "Cavalieri", ["gleam"])
}

Triggering the code action on the User constructor will result in the language server adding the labels to the arguments.

pub type User {
  User(first_name: String, last_name: String, likes: List(String))
}

pub fn main() {
  let first_name = "Giacomo"
  User(first_name:, last_name: "Cavalieri", likes: ["gleam"])
}

Inter-module generate function code action

The "Generate function" code action now works when the missing function is to be defined in another module. For example:

// src/maths.gleam
pub fn add(a: Int, b: Int) -> Int { a + b }

// src/app.gleam
import maths

pub fn main() -> Nil {
  echo maths.add(1, 2)
  echo maths.subtract(2, 1)
  Nil
}

The app module is calling a function called subtract from the maths module, but that function doesn't exist. Triggering the code action on the call to maths.subtract will edit the maths.gleam file to add the outline of the function, for the programmer to complete.

pub fn add(a: Int, b: Int) -> Int { a + b }

pub fn subtract(int: Int, int_2: Int) -> Int {
  todo
}

Thank you Surya Rose! This is a nice little quality-of-life improvement for Gleam programmers.

Extract function code action

And the last code action of this release, one that has been eagerly anticipated for some time by many Gleam programmers: extract function.

const head_byte_count = 256

pub fn get_head_of_file() {
  let assert Ok(contents) = read_file()
  case contents {
    <<head:bytes-size(head_byte_count), _:bits>> -> Ok(head)
    _ -> Error(Nil)
  }
}

If you were to select the case expression in your editor and trigger the code action, then it would be extracted to a new function, like so:

const head_byte_count = 256

pub fn get_head_of_file() {
  let assert Ok(contents) = read_file()
  function(contents)
}

fn function(contents: BitArray) -> Result(BitArray, Nil) {
  case contents {
    <<head:bytes-size(head_byte_count), _:bits>> -> Ok(head)
    _ -> Error(Nil)
  }
}

Unfortunately the language server protocol design is rather limiting, so the Gleam language server cannot prompt the programmer for a suitable name, it has to use a meaningless name instead. The "rename" feature of the language server can be triggered to give it a more appropriate name.

I believe that Microsoft's rewrite of the TypeScript toolchain will include it using the language server protocol instead of their custom protocol, so hopefully this will result in them expanding the LSP specification to include features their custom protocol has, such as code actions being able to ask for more information.

Thank you again Surya Rose!

Formatter improvements

Gleam has a code formatter that can clean up a file of Gleam code in an instant, freeing up time that would be otherwise spent on the busy-work of manually laying out code. Typically it is run by the programmer's text editor when a file is saved. Several improvements have been made to it with this release.

Bools can be negated with the ! operator, and ints can be negated with the - operator. Negating a value multiple times is redundant and does nothing, so the formatter now collapses duplicate negations.

pub fn useless_negations() {
  let lucky_number = --11
  let lucy_is_a_star = !!!False
}

The code is rewritten like so:

pub fn useless_negations() {
  let lucky_number = 11
  let lucy_is_a_star = !False
}

Additionally, the formatter no longer removes blocks from case clause guards, as the programmer may wish to include them to make code clearer, even if they are not required according to Gleam's operator precedency rules. This also makes the behaviour consistent with the formatting of regular expressions in Gleam functions.

Thank you Giacomo Cavalieri!

And the rest

And thank you to the bug fixers and experience polishers: Andrey Kozhev, Benjamin Peinhardt, Danielle Maywood, fruno, Giacomo Cavalieri, Joohoon Cha, Matias Carlander, Surya Rose, and Tristan-Mihai Radulescu).

For full details of the many fixes and improvements they've implemented see the changelog.

A call for support

Gleam is not owned by a corporation; instead it is entirely supported by sponsors, most of which contribute between $5 and $20 USD per month, and Gleam is my sole source of income.

We have made great progress towards our goal of being able to appropriately pay the core team members, but we still have further to go. Please consider supporting the project or core team members Giacomo Cavalieri and Surya Rose on GitHub Sponsors.

Thank you to all our sponsors! And special thanks to our top sponsors:

Try Gleam