Metals: A Meta Programming Language for Composable Systems

Metals is a library that provides the meta programming language foundation for domain modeling and schema management.

The powerful meta modeling capabilities provided by this library will power the next generation of distributed, compositional cloud computing environment.

Domains and Modeling in Computation

A domain is a field or an area under investigation. In a computational context, we are interested in developing computational models of domains in the real world and use these models to build solutions to a set of identified problems.

Modeling or model development is the first step in any project. They are essential to communication among stakeholders and used throughout the development process. They may be formal or informal, documented or not, explicit or implicit. In a way, they are like spoken language! We use it pretty much during the course of all our activities, without conscious awareness of their existence or use most of the time.

The power of models essentially decides the type of problems we can address and the quality of our solutions.

Key ideas

The domain modeling capabilities and the required meta programming support within metals are based on the following key ideas.

Simple models, complex domains: Models based on simple rules can exhibit complex behavior. Ability to model domains of arbitrary complexity does not necessarily mean that the models have to be unreasonably complex as well. We strive to promote simple models throughout.
Compositional models: Complex models are compositionally built from simpler ones. It is our fundamental belief that the compositionality of our mental models underlie the extraordinary human cognitive capabilities.
Context Aware Computing: Next to compositionality, context awareness or mode dependent computation is the most important aspect of the computational models supported by metals. This is especially important in the context of distributed, heterogeneous computing environments. The models are not used by a single system or environment as we traditionally assume when we build computational models.
Combinatorial generalization: the ability to learn increasingly complex concepts by synthesizing simple ideas, enabling both rapid learning and adaptation of knowledge

Our goals

The modeling capabilities provided by metals shall meet the following goals.

Enable Communication: The models should serve as the communication medium between different teams, especially with domain experts. We shall not expect any programming expertize from the domain experts or other teams outside of the development or engineering teams, nor shall we want the developers to leave their tools of choice and familiarity to build communication models.
Modeling Data and Behaviors: The models should be able to represent both data and their associated behaviors. The compositionality and combinatorial generalization shall apply to both data and behaviors.
Executable models: The models should be executable. It should be possible to convert the models developed through collaborative modeling process to executable artifacts with minimal effort.
Make Illegal States Unrepresentable: The models should be able to capture the domain constraints explicitly and make all illegal states unrepresentable. The models should fully and faithfully represent the domain constraints so that they can serve as a single source of truth about accuracy our requirements and the subsequent implementation.
Uniform Structured representations: The modeling language shall support structured representations that can be used across heterogeneous environments. The models should be able to provide uniform view of the domain across many boundaries that exists in the development and runtime environments such as programming languages, frameworks, platforms, networks, etc.
Lazy Evaluation and Transparency: A state represented in the model shall be evaluated lazily in a location transparent way. When one models a domain, the domain objects represent data that eventually resolve to appropriate values, not necessarily eagerly computed. For example, when one refers to a User in the model, it could be an object that is returned by a network call or a database query. The models should be able hide such details. The transparency also means that the model builders are not burdened by idiosyncrasies such as different query semantics, implementation details and access protocols of the underlying systems or environment.
Distributed and Independently Evolving Models: Building a uniform model for a given application does not necessarily mean a single global model. Global models are complex to build and a nightmare to maintain. Our goal is to support smaller, distributed models that can evolve independently. An application (or solution) context will be able import the relevant models and create a unified view of the domain as it pertains to that application.

We will look into some of the modeling challenges, prevailing techniques, and how we approach them in metals.

Modeling Data and Behaviors

There are several capabilities a powerful domain modeling language shall provide to model data and behaviors in heterogeneous environments. We shall look at each of these capabilities in some detail below.

Let's start with data.

Modeling Data

A Simple Example Model

Let us take a simple example of modeling a User. A naive approach would start with defining a type representing the model for User. This can be done in a programming language such as Rust. Even someone without any programming language experience can understand the model below.

#![allow(unused)]
fn main() {
struct User {
    first_name: String, // Required
    last_name: String,  // Optional
    age: u8,            // Optional. Must be > 18
    email: String,      // Required. Must follow valid email format
}
}

Refining the Model

From a domain modeling perspective, let us consider a few improvements to the above model which are right now communicated using comments in the snippet.

Only first_name and email are required fields, rest are optional.
All our users are aged above 18.
Not all Strings are valid email addresses. For starters, we want to guarantee they have the right email format (someone@somewhere.som).

Representing optional data

There are straightforward ways to incorporate optional information in most programming languages today. One can encode them in Rust as below.

#![allow(unused)]
fn main() {
struct User {
    first_name: String,         
    last_name: Option<String>,  
    age: Option<u8>,           // Must be > 18
    email: String,             // Must follow valid email format
}
}

Constraining the domain of values

Let us tackle the email format problem next. Essentially what we are saying is that a String is not the most appropriate representation of an email address. This where we use the type system to help us. Let us define a new type for email, EmailAddress, to clearly model the above idea.

#![allow(unused)]
fn main() {
// New type representing an email address
struct EmailAddress(String);

struct User {
    first_name: String,         
    last_name: Option<String>,  
    age: Option<u8>,           // Must be > 18
    email: EmailAddress,       // Must follow valid email format
}
}

This model is better. It is very clear that there is more to an email than just any plain String.

But that has not helped much with the definition of EmailAddress itself. It still seems to say the same, that it is just a String.

If you are a developer, you can think of several ways to code this up.

You can code up the whole thing in the constructor (or builder) for EmailAddress type. This is completely opaque from a model perspective.
You can explicitly capture the regular expression representing an email to ensure that everyone can see it (in the spirit of transparency).
You can also make it clear to everyone that the constructors can fail by using an appropriate return type such as Result that can either return a valid User or an error.

#![allow(unused)]
fn main() {
// New type representing an email address
struct EmailAddress(String);

// Email address errors
enum EmailAddressError {
   InvalidFormat(String),
}

// Regular expression used to validate email addresses
// Trust me, we use it in the code below.
const EMAIL_REGEX: &str = r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$";

impl EmailAddress {
   fn new(addr_string: String) -> Result<EmailAddress, EmailAddressError> {
      // Coming soon: The code for validating and constucting a new instance
      todo!()
  }
}

struct User {
    first_name: String,         
    last_name: Option<String>,  
    age: Option<u8>,           // Must be > 18
    email: EmailAddress,       // Must follow valid email format
}
}

While this may be an improvement, there are several shortcomings.

The regular expressions (RegEx) are not exactly very friendly (at least in my humble opinion). But they are a necessary evil and we may live with it. At least, there are standard expressions that you can use from web without fighting over their correctness during your modeling time for most common data types.
But as more and more validations are added, we are going to move more model aspects to code. What if we wanted to express that the email address should be less than 128 characters?
Things can easily get very complicated. What if the email address is to be validated by an external service or against a registered user database? In many schema based environments like GraphQL, they are addressed by field level resolvers. Now, both programmers and modelers have to learn a new modeling language to communicate with each other. We are not suggesting that it is wrong, but just pointing out how frictions arise in modeling even simple things. How implementation and environment details creep into the safe modeling space we were trying to create for ourselves!
Even with this simple example, one can see that the worlds of modeler and programmer are starting to diverge while their concerns remain entangled. As more model questions needing clarity emerge, more details seem to submerge in code. For all the slogans such as code is documentation and model is code that we embrace out there, we seem to have to resort to extraordinary measures to capture and communicate even the simplest of modeling requirements.

To be fair, there are aspects of model and computations that ultimately have to be coded and that require the specialized programming skills. But we should at least be able to separate them cleanly, keep them far away from the common concerns that impact diverse stakeholders and delay them till design and implementation time.

Getting back to our example, a common solution is to provide appropriate constructors or builders for a each type, so that their interfaces can represent the domain constraints better. These builders with combinators is a really powerful tool. The challenge is the broiler plate code that we end up developing and porting across different environments. We will definitely embrace builder patterns and combinators in our model. But we try to do that in a generic way without all the ceremony!

An important point that we might be glossing over here. The constructors and builders deal with constructing instances for a given type. In our discussion, we are starting to talking about constructing types themselves. We have casually entered the meta realm.

Issues of state

Before exploring the solutions along the lines we discussed above, let us discuss a few more challenges with these approaches.

We just modelled a general user in our system and boldly proclaimed that certain fields are required and others are optional. This represents a programmer bias, instance bias, where we take an instance creation to also represent a single instant in time.

A typical instance creation is coded as shown below.

#![allow(unused)]
fn main() {
// Hiding the User definition for brevity. Copied from previous snippet
struct EmailAddress(String);

struct User {
   first_name: String,         
   last_name: Option<String>,  
   age: Option<u8>,           // Must be > 18
   email: EmailAddress,       // Must follow valid email format
}

let user = User {
    first_name: "John".to_string(),
    last_name: Some("Doe".to_string()),
    age: Some(42),
    email: EmailAddress("john@doe.com".to_string())
};
}

For a developer, the mental association of an instance creation is that of invoking a constructor, which happens instantly. This is not how things happen in the real world.

For instance (no pun intended), we may have a guest user we want to track who may not have any associated data in our system until after the user registers with the system. Once a guest user completes the registration process, the rest of the information becomes available.

Now, the simple, nice, clean data model we developed for our User is starting to unravel! This is a problem state which is very common in models of any kind. The available information (aka data) and expected behaviors of domain object do depend on state.

These scenarios are so ubiquitous, there are many tricks of the trade that we have developed over time. Over one third of the use cases for Business Process Management (BPM) systems involve handling some form of this synchronization of state with data representation.

Most solutions involve abandoning the required fields altogether and adding additional fields either in the same model or creating separate User models corresponding to different user states. The model consideration have left the model space, disappearing behind a think fog of code and process diagrams.

The builder pattern does not address this modeling problem.

In addition to the representation problem in the time dimension, there is a problem in the spatial dimension too. Different microservices that constitutes our modern systems may have slightly different views of the model while referring to the same domain entity. Even in monolithic systems, there are representational differences across different layers or between frontend and backend subsystems.

For example, there may be a password field for User, which should not be available to any parts of the system, except the one responsible for user authentication perhaps.

User may have an ID field that is both required and non-null, not when it originates from a web form for new user creation, but definitely on its return journey from the server and all times thereafter.

What a User is depends upon who is asking and when in the application. The answer is context dependent! But there is still the essence of it all that is invariant in the system, a single culprit with different witness descriptions.

Relationships in Model

Relationships are part of life and models. They can be challenging to manage in both cases as well.

By representing email addresses as a different kind of model entity, we already introduced relationships into our model.

Every User has an email address

That seemingly innocent and straightforward sentence introduces a number of concepts relating to relations into our model.

Relatedness: Two entities are related to each other in some way.
Directionality: The relationship is from User entity to EmailAddress. There is nothing in our model the reverse relation from EmailAddress to User, at least in the model we developed so far.
Degree of Dependency: The User contains an EmailAddress_. We may express the nature of relationship between entities using different nomenclature depending on our professional upbringing. We strive to capture the degree and nature of these relationships in terms of association (weak form), aggregation, composition, containment etc. in some terminology.
Cardinality: Every User has one EmailAddress. Here it is expressed by single, required field embedded into the User model. BTW, it does not exclude the possibility of multiple users having same email, which is probably what we intended to model. This requires a uniqueness constraint. We have definitely rules out the possibility of a User having multiple EmailAddresses, whether that was our intention or not. As you can see, we want to be able to reason with one-to-one, one-to-many and many-to-many relations between model elements or entities.
Lifecycle: There are questions about the the life of related entities in our model. Is an EmailAddress valid even after the associated User is removed from the system? What happens when an EmailAddress is updated, invalidated or removed? Should that be allowed outside the context of a User? What if we were tracking different communications User had in our system which is linked to email?

Such modeling concepts and their primitives are predominant among database community. They are also very much part of formal modeling systems such as UML. But they are mostly incorporated into the programming interfaces through broiler plate code. In many cases, this means that we may express the relationship inconsistently across different parts of the system. The model of User for the database would contain this constraint, but the Application Programming Interface (API) would not! Shouldn't it be possible for all stakeholders and all systems to be able to have same understanding of model constraints without actually worrying about where and how they are implemented?

The moral of the story here is that modeling decisions in the presence of complex relations and their representation and communication is hard, even without the complexities of globally distributed, heterogeneous environments where modular subsystems can independently evolve.

Models of Data

Crossing the chasm

Data modeling has a long and storied history. It is a rich and well established field well supported by a thriving database community. The databases and their schemas have been powering most of the systems out there. When we refer to model in an application context (Model, View, Controller (MVC) pattern as example), we are most often referring to the data, usually stored in some database. Yet there seems to be a divide between the modeling aspects of data as it pertains to data layer and computation (or application logic) layer. One focuses more on data at rest while the other is concerned about the data in motion (transition). But both have to deal with the dynamics of the system (the changing states).

Data community does this by shoving more status fields into their tables and documents while the application community (both client and server side) deals with them by writing a truck load of code in the name of controllers and logical blocks. The very essence of our system live in the wild wild west of broiler plate code that is built to tie these disparate worlds together.

Can we have a unified model of a domain that crosses these boundaries of client, server, middleware and databases? Can we do this without all the ceremony and fanfare?

The quote below summarizes our troubles and tribulation with modeling so far.

"Domain entities are modal and dynamic, while our data models are static."

This is what we attempt to change. Before we can address the above challenges, let us also look at the challenges with adding behaviors to our model.

Modeling Behaviors

THIS SECTION IS STILL WORK IN PROGRESS

Metal Plating Models

Having looked at the variegated and vitalizing challenges of modeling, it is time to explore our approaches to solve them.

Metals - A Meta Programming Library for Modeling Composable Systems

Metals is a meta programming library that extends the existing programming environments to provide powerful modeling and schema management capabilities.

At a very high level, Metals provides the following:

A meta model: At the core of Metals is a composable model definition language that is easy to use, effective to communicate and mathematically accurate at the same time.
Programming Language Extensions: In order to build models using the standard programming tools and making them executable (able to convert them to working applications), Metals extends programming languages such as Rust, Typescript and Python. It is our goal to support as many of the popular programming languages as possible by making this process easy and by working with respective programming language communities.
A Schema Repository - Schemer: A repository to store and retrieve meta models.
Schema Federation - Xborder (pronounced cross-border): Ability to import and integrate models from external schemas sources such as GraphQL as well as to export schema from programs to repository.
A Runtime - Aprun (/ˈeɪ.prən/ pronounced like apron): The runtime guarantees interoperability of models across distributed, heterogeneous environments.

We will revisit our toy example and see how we can metal plate our problems in the coming sections. Please refer to the modeling guide for detailed model development and the reference for technical and mathematical details behind Metals.

THIS SECTION IS STILL WORK IN PROGRESS

Posting with Metals

In the Rust book, there is a chapter on Implementing an Object-Oriented Design Pattern which describes a stateful Posts API at some length. We will revisit this example in the context of metals.

Note: The primary objective of the original example and code used in the Rust book is educational, not building production ready code. This article should be read in the same spirit and not as a comparison of design decisions or superiority of technical choices.

Original Post Example from Rust Book

The book discusses the State pattern and how to go about implementing it in Rust. The crux of the pattern is that the behavior of a value can change based on a set of states which we track internally.

The book first implements this pattern closely following the object-oriented way first and then refines it using the Rust language capabilities. We will look at both approaches here.

Overview of functionality

Here is a summary of the functionality from the book for each reference:

A blog post starts as an empty draft.
When the draft is done, a review of the post is requested.
When the post is approved, it gets published.
Only published blog posts return content to print, so unapproved posts can’t accidentally be published.

Object oriented implementation

The code below shows the intended use of this API (reproduced from original article). This will not run yet.

fn main() {
    use blog::Post;

    // Create new post
    let mut post = Post::new();

    // Add text to the post
    post.add_text("I ate a salad for lunch today");
    // Since this post is still in draft and not yet approved, you will
    // not see the content of the post yet.
    assert_eq!("", post.content());

    // Request review of the post
    post.request_review();
    // Content is still not visible
    assert_eq!("", post.content());

    // Approve the post
    post.approve();
    // Now you have access to the content of the post
    assert_eq!("I ate a salad for lunch today", post.content());
}

Here is the module that implements the Post functionality as expected by the text and code above. Again, this is the same code from original book chapter, with a lot of added comments to make it easier to follow for those less familar with Rust syntax and also to bring out some of our observations.

#![allow(unused)]
fn main() {
pub mod blog {
    /// Post has an internal state and string content
    pub struct Post {
        state: Option<Box<dyn State>>,
        content: String,
    }

    // This is the Post interface. This shows everything you can do with
    // Post
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state
        /// - It has empty content
        pub fn new() -> Post {
            Post {
                state: Some(Box::new(Draft {})),
                content: String::new(),
            }
        }

        /// Add text to the post
        /// - It appends the new text to existing content
        /// - This is not a stateful operation. The way it is defined
        /// here implies that you can add text to posts that are
        /// pending review or approval.
        /// - This is not very difficult to alleviate. We just have to
        /// delegate this to state as we do with other stateful
        /// operations
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Returns the content
        /// - It delegates to the current state so that the state
        /// can decide the right thing to do (either return an content
        /// or the actual content)
        pub fn content(&self) -> &str {
            self.state.as_ref().unwrap().content(self)
        }

        /// Request review of the post
        /// - This may cause a state transition. Here we don't have to
        /// worry about the whole associated state transition, but the
        /// State implementation will take care of it.
        /// - But we have to check the code for each state to see what
        /// happens when we request a review.
        /// - If you look at the code, you will see the following valid
        /// transitions between state when we invoke request_review
        ///     Draft -> PendingReview
        ///     PendingReview -> PendingReview (no transition, a no-op)
        ///     Published -> Published (no transition, a no-op)
        /// - See comments on state implementation
        pub fn request_review(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.request_review())
            }
        }

        /// Approve the post
        /// - Similar comments as request_review() above are relevant here.
        /// - The supported transitions from State code is
        ///     Draft -> Draft (no transition, a no-op)
        ///     PendingReview -> Published
        ///     Published -> Published (no transition, a no-op)
        pub fn approve(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.approve())
            }
        }
    }

    /// This defines the interface for state. This is currently a
    /// private one within this module (no `pub` modifier in front of
    /// `trait` keyword).
    /// - Good thing is that the users of the _Post_ API does not have
    /// to know about state and can completely rely on _Post_ interface
    /// we defined above.
    /// - The bad thing is that _State_ is hidden from the users of
    /// _Post_ API. One could argue that it is very important domain
    /// characteristic that impacts how the API behaves. Shouldn't
    /// the API user know that the a _Post_ is in _Draft_ state and
    /// hence has to be submitted for review?
    /// This also defines common interface to be implemented by all
    /// states, irrespective of whether this is relevant in that state.
    trait State {
        /// Represent a state action which _may_ a transition.
        fn request_review(self: Box<Self>) -> Box<dyn State>;
        fn approve(self: Box<Self>) -> Box<dyn State>;
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            ""
        }
    }

    /// Represents the Draft state
    struct Draft {}

    /// Here is the implementation of State interface for Draft.
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for Draft {
        /// Represents the transition of state
        ///     Draft -> PendingReview
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            Box::new(PendingReview {})
        }

        /// The approve is a no-op, just trivially return the current
        /// state.
        /// Shouldn't this be disallowed. Without going through the
        /// code, it difficult to understand the what we are doing here.
        /// Can someone approve a draft directly? Interface says
        /// 'Yes', but code says it ignores it.
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }
    }

    /// Represents PendingReview state
    struct PendingReview {}

    /// The implementation of State interface for PendingReview
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for PendingReview {
        /// This is actually a no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// Transition to Published state
        ///     PendingReview -> Published
        fn approve(self: Box<Self>) -> Box<dyn State> {
            Box::new(Published {})
        }
    }

    /// Represents Published state
    struct Published {}

    /// The implementation of State interface for Published.
    /// There are no transitions out of this state.
    impl State for Published {
        /// A no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// A no-op
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// This overrides the default implementation from State
        /// and returns the content of the associated post
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            &post.content
        }
    }
}
}

Now you can run the code from first block using our Post API and see that it is working as intended.

/*
 * Copyright 2022 Weavers @ Eternal Loom. All rights reserved.
 * Use of this software is governed by the license that can be
 * found in LICENSE file in the source repository.
 */
/// Original blog post example from [`Rust Book`] [`chapter`].
///
/// [`chapter`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// [`Rust Book`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
fn main() {
    use blog::Post;

    // Create new post
    let mut post = Post::new();

    // Add text to the post
    post.add_text("I ate a salad for lunch today");
    // Since this post is still in draft and not yet approved, you will
    // not see the content of the post yet.
    assert_eq!("", post.content());

    // Request review of the post
    post.request_review();
    // Content is still not visible
    assert_eq!("", post.content());

    // Approve the post
    post.approve();
    // Now you have access to the content of the post
    assert_eq!("I ate a salad for lunch today", post.content());
}

pub mod blog {
    /// Post has an internal state and string content
    pub struct Post {
        state: Option<Box<dyn State>>,
        content: String,
    }

    // This is the Post interface. This shows everything you can do with
    // Post
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state
        /// - It has empty content
        pub fn new() -> Post {
            Post {
                state: Some(Box::new(Draft {})),
                content: String::new(),
            }
        }

        /// Add text to the post
        /// - It appends the new text to existing content
        /// - This is not a stateful operation. The way it is defined
        /// here implies that you can add text to posts that are
        /// pending review or approval.
        /// - This is not very difficult to alleviate. We just have to
        /// delegate this to state as we do with other stateful
        /// operations
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Returns the content
        /// - It delegates to the current state so that the state
        /// can decide the right thing to do (either return an content
        /// or the actual content)
        pub fn content(&self) -> &str {
            self.state.as_ref().unwrap().content(self)
        }

        /// Request review of the post
        /// - This may cause a state transition. Here we don't have to
        /// worry about the whole associated state transition, but the
        /// State implementation will take care of it.
        /// - But we have to check the code for each state to see what
        /// happens when we request a review.
        /// - If you look at the code, you will see the following valid
        /// transitions between state when we invoke request_review
        ///     Draft -> PendingReview
        ///     PendingReview -> PendingReview (no transition, a no-op)
        ///     Published -> Published (no transition, a no-op)
        /// - See comments on state implementation
        pub fn request_review(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.request_review())
            }
        }

        /// Approve the post
        /// - Similar comments as request_review() above are relevant here.
        /// - The supported transitions from State code is
        ///     Draft -> Draft (no transition, a no-op)
        ///     PendingReview -> Published
        ///     Published -> Published (no transition, a no-op)
        pub fn approve(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.approve())
            }
        }
    }

    /// This defines the interface for state. This is currently a
    /// private one within this module (no `pub` modifier in front of
    /// `trait` keyword).
    /// - Good thing is that the users of the _Post_ API does not have
    /// to know about state and can completely rely on _Post_ interface
    /// we defined above.
    /// - The bad thing is that _State_ is hidden from the users of
    /// _Post_ API. One could argue that it is very important domain
    /// characteristic that impacts how the API behaves. Shouldn't
    /// the API user know that the a _Post_ is in _Draft_ state and
    /// hence has to be submitted for review?
    /// This also defines common interface to be implemented by all
    /// states, irrespective of whether this is relevant in that state.
    trait State {
        /// Represent a state action which _may_ a transition.
        fn request_review(self: Box<Self>) -> Box<dyn State>;
        fn approve(self: Box<Self>) -> Box<dyn State>;
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            ""
        }
    }

    /// Represents the Draft state
    struct Draft {}

    /// Here is the implementation of State interface for Draft.
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for Draft {
        /// Represents the transition of state
        ///     Draft -> PendingReview
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            Box::new(PendingReview {})
        }

        /// The approve is a no-op, just trivially return the current
        /// state.
        /// Shouldn't this be disallowed. Without going through the
        /// code, it difficult to understand the what we are doing here.
        /// Can someone approve a draft directly? Interface says
        /// 'Yes', but code says it ignores it.
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }
    }

    /// Represents PendingReview state
    struct PendingReview {}

    /// The implementation of State interface for PendingReview
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for PendingReview {
        /// This is actually a no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// Transition to Published state
        ///     PendingReview -> Published
        fn approve(self: Box<Self>) -> Box<dyn State> {
            Box::new(Published {})
        }
    }

    /// Represents Published state
    struct Published {}

    /// The implementation of State interface for Published.
    /// There are no transitions out of this state.
    impl State for Published {
        /// A no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// A no-op
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// This overrides the default implementation from State
        /// and returns the content of the associated post
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            &post.content
        }
    }
}
// --snip-- the rest of the code from above two blocks

Analysis of object-oriented Post implementation

The original book chapter goes through a lot of details on the design and implementation choices that were made as well as their pros and cons. We will discuss some of them here and then add a few of our own observations. We highly encourage you to read the original book chapter. We hope
the discussion shall help everyone, at the very minimum, appreciate the gamut of design thinking and choices involved in even simplest of API's.

Modeling considerations that we have been discussing in the context of metals were not the focus of the authors of the Rust book in the chapter. So, the points we are brining out here are not those of omission or inadequate design considerations, but more a matter of content choice on a specific subject.

From a model perspective (please keep our comments from previous paragraph in mind), there are a lot of technical details that can be intimidating even for technical folks without fair understanding of Rust. What the hell is Option<Box<dyn State>?
While enum variant is a good choice to represent mutually exclusive states, the choice (of trait object) made in the code provides better ergonomics at the API call site (by doing away with the need for match expressions).
There is clear separation of concerns. Post knows nothing about various behaviors. It also promotes modularity and clarity. There is one place to look at how the system behaves in a given state.
The Rust book chapter has some suggested extensions and hints to implement them. It is seems easy to extend the system with new behaviors. Well, to an extend!
There is strong coupling between states as they implement the transition to next state.
Using the example from the original book chapter, if we add another state between PendingReview and Published, such as Scheduled, we would have to change the code in PendingReview to transition to Scheduled instead.
The transitions are pretty much embedded in code for different states. We have to go through the code for each State to understand the real state transition model of the system.
There is duplication of code where we have to implement all state actions on every state even when they are no relevant transitions.
We also have to duplicate state actions such as request_review and approve on Post which simply delegates to their counterparts in State.
We definitely don't make illegal states unrepresentable. We are returning empty string for content in states where content access is illegal.
Externally extending this API can be tricky. In technical jargon, this implementation suffers from an expression problem.
Adding a new behavior would need us to reach into Post implementation code to add new things. Imagine adding reject action or reset_text functionality.
As it is implemented here, we would need to reach into Post implementation code to add them.
Referring back to our discussions on modeling data, this model represents an application programming view.
The primary purpose of Posts is to manage its content. We can think of many ways in which we want to allow the post content to be manipulated - reset the content, change parts of the content, maintain change history, add review comments, so on and so forth.
We will end up adding a lot of code all across states!
This pattern does not address many data modeling concerns. For example, the request_review cause a transition to PendingReview only if the content is nonempty.
To be fair, that is not what this pattern was designed for.
All states operate on the same globally shared data instead of them operating on just the parts that they are interested in.
As in most design patterns from the past decade, this is emblematic of top down, system breakdown approach to building modular systems as opposed to compositional approach to building larger system from smaller ones.

Doing it the Rusty way

The original book chapter also shows how to implement this pattern differently that is more idiomatic of Rust. This is based on the idea of implementing states and behaviors as types.

Here is the revised Post code.

#![allow(unused)]
fn main() {
pub mod rusty_blog {
    /// _Post_ has String content
    /// - For starters, cleaner interface (no Box<dyn ...>> stuff)
    pub struct Post {
        content: String,
    }

    /// The interface for Post (encoding the behavior of Post)
    /// - We can create a new post, but pay attention to what it returns
    /// - We can get access to the content
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state, captured by the fact that it
        /// returns a DraftPost type
        /// - It has empty content
        pub fn new() -> DraftPost {
            DraftPost {
                content: String::new(),
            }
        }

        /// Returns the content
        pub fn content(&self) -> &str {
            &self.content
        }
    }

    /// DraftPost also has String content
    /// - Note that it is just another type
    /// - We are effectively trying to encode the _State, DraftState_ as a type
    pub struct DraftPost {
        content: String,
    }

    /// The interface or the encoded behaviors for DraftPost
    /// - We can add text to the post when you are in Draft state
    /// - We can request review of the post at which point the state would change
    impl DraftPost {
        /// Add text to the post.
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Request review of the post
        pub fn request_review(self) -> PendingReviewPost {
            PendingReviewPost {
                content: self.content,
            }
        }
    }

    /// PendingReviewPost also has String content
    pub struct PendingReviewPost {
        content: String,
    }

    /// The interface or the encoded behaviors for PendingReviewPost
    /// - One can approve the Post in this state
    impl PendingReviewPost {
        /// Approve the post
        /// - Here is when the actual Post is getting constructed
        /// - Since _Post_ has content() method on its interface, the content
        /// is available to the callers
        pub fn approve(self) -> Post {
            Post {
                content: self.content,
            }
        }
    }
}
}

We have to make small adjustments to the call site (the main function), but will retain most of it.

/*
 * Copyright 2022 Weavers @ Eternal Loom. All rights reserved.
 * Use of this software is governed by the license that can be
 * found in LICENSE file in the source repository.
 */
/// Original blog post example from [`Rust Book`] [`chapter`].
///
/// [`chapter`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// [`Rust Book`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// This is the re-implementation of State pattern in idiomatic rust
///  based on the idea of representing states and behaviors as types
fn main() {
    use rusty_blog::Post;
    // Create new post. Here we are getting a DraftPost
    let mut post = Post::new();

    // We can add the text in Draft state
    post.add_text("I ate a salad for lunch today");
    // We can't call content() here. It is a compilation error as
    // DraftPost does not support this behavior (nice!)
    //assert_eq!("", post.content());

    // Request review of the post
    // Since it returns a new type (PendingReviewPost), we have
    // to capture it in a variable (See let post = ... added below)
    let post = post.request_review();
    // Content is still not visible
    // It is a compilation error to try and access content
    // assert_eq!("", post.content());

    // Approve the post. We get the real Post
    let post = post.approve();

    // Now we have access to the content of the post.
    assert_eq!("I ate a salad for lunch today", post.content());
}

pub mod rusty_blog {
    /// _Post_ has String content
    /// - For starters, cleaner interface (no Box<dyn ...>> stuff)
    pub struct Post {
        content: String,
    }

    /// The interface for Post (encoding the behavior of Post)
    /// - We can create a new post, but pay attention to what it returns
    /// - We can get access to the content
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state, captured by the fact that it
        /// returns a DraftPost type
        /// - It has empty content
        pub fn new() -> DraftPost {
            DraftPost {
                content: String::new(),
            }
        }

        /// Returns the content
        pub fn content(&self) -> &str {
            &self.content
        }
    }

    /// DraftPost also has String content
    /// - Note that it is just another type
    /// - We are effectively trying to encode the _State, DraftState_ as a type
    pub struct DraftPost {
        content: String,
    }

    /// The interface or the encoded behaviors for DraftPost
    /// - We can add text to the post when you are in Draft state
    /// - We can request review of the post at which point the state would change
    impl DraftPost {
        /// Add text to the post.
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Request review of the post
        pub fn request_review(self) -> PendingReviewPost {
            PendingReviewPost {
                content: self.content,
            }
        }
    }

    /// PendingReviewPost also has String content
    pub struct PendingReviewPost {
        content: String,
    }

    /// The interface or the encoded behaviors for PendingReviewPost
    /// - One can approve the Post in this state
    impl PendingReviewPost {
        /// Approve the post
        /// - Here is when the actual Post is getting constructed
        /// - Since _Post_ has content() method on its interface, the content
        /// is available to the callers
        pub fn approve(self) -> Post {
            Post {
                content: self.content,
            }
        }
    }
}
// --snip-- the rusty_blog module code from previous snippet

Analysis of Rusty Post implementation

Let us take a closer look at the this implementation.

It makes illegal states unrepresentable. We can only change Post content while it is a Draft and can only access the content once it is approved. This is a small step for API provider, big step of API consumer productivity.
We seems to have simplified the interface a lot. Lot of scary looking things like Box<dyn ...> have disappeared from the interface.
We have removed a lot of duplicated code and unnecessary implementations.

A lot of good things have happened and looks like this is moving in the right direction. Still there are several things we have not addressed yet.

The transitions remain embedded in the code.
There is still very strong coupling between the states. Since these are types, not the interfaces (those dyn State thingies), we probably have made the problem worse. There is tremendous power in our ability to adhere to an interface contract while changing implementations. This is why build APIs in the first place.
This also impact the extensibility of the API both from inside as well as outside. Ideally, we would like to be able extend the interface of a state, say PendingReview, and provide an implementation for new interface without having to modify the existing code. We need some kind of dependency injection to reduce coupling of states and retain the power of interfaces we had in our original object-oriented implementation.
We really have not addressed any of the data modeling concerns we discussed in our previous analysis section.
On the other hand, we have completely localized the states and lost the complete idea of a shared data. May be we swung too far. We still want the concept of shared data model and data access so that we can deal with those aspects, say syncing with a datastore more efficiently and effectively.
Just our behaviors are intricately connected to states, these state we use to encode different modes of the API are linked to the underlying data. This means the state actions can fail or transition to different states depending on the data and the rules governing them. We are looking at a non-deterministic automaton as opposed to a deterministic one.
As far as state design patterns go, there are established approaches such as immutable states and fine-grained action interfaces to make it deterministic and keep it simple. All the other concerns are packed up as side effects and handled through a set of other patterns.
At the very least, we should be able to see that the complex business rules are moving away from the neat interfaces we built. The point we are making here is that in an API design, the state/interface dependency is not at the top layer, but they go all the way down. We need to be able to package them and compose them both vertically and horizontally. That sentence may not mean much now, we will return to it when discuss metals.

On Handling States

Here in this chapter, we are specifically looking at the design pattern as discussed in the Rust book. It is important to note that there are several approaches to handling states besides this single pattern, especially if you have familiarity with frontend frameworks. If you have worked in that area, you would have come across Redux, XState, MobX and many more and their integrations to frontend frameworks like React. Redux, arguably the most popular of them all, has been ported to many languages including a Rust (redux-rs).

Redux uses a more functional pattern to state management. In order to make the usage simpler and reduce the level of broiler plate code, there are also several libraries and tools available including Redux Toolkit.

There is also a rich field of formal techniques for modeling states, deterministic and nondeterministic finite automatons (DFA and NFA), Moore and Mealy machines, Harel statecharts and more (See State diagrams for a preview).

Another area that has profound insights and imputs to this world is coming from Dynamial systems theory. We derive our inspirations heavily from the work of David I. Spivak, especially his work on Polynomial functors.

Quote to Use. Here is a quote from the sipc book(The structure and interpretation of computer programs). One powerful design strategy, which is particularly appropriate to the construction of programs for modeling physical systems, is to base the structure of our programs on the structure of the system being modeled.

Metal plating the posts

THIS SECTION IS STILL WORK IN PROGRESS

Modeling with Metals

In this section we will cover the basics of metals modeling framework and provide a guided introduction to modeling with it.

THIS SECTION IS STILL WORK IN PROGRESS

Heavy Metals - Metals Deep Dive

In this section we will cover the technical and mathematical details behind the metals modeling framework.

We will start by discussing interfaces as a means to specify observable behaviors of a system. By combining some elegant mathematics and proven program modeling concepts, we will extend them to represent states, finally arriving at a unified modeling framework for composing complex systems from simpler ones.

Metals turn interfaces into models that can talk and run at the same time, talks as in its ability to communicate clearly about the systems they represent and run as in they are executable.

Modeling Composable Systems

We are on a quest to compose complex systems from simple ones. The character of a system is determined by its observable behaviors, internal states and the dynamics that generates the behaviors from states.

Systems: Interfaces, States and Dynamics

We will build our core ideas behind metals in this section.

Systems

A system in our world is anything that we are trying to model and build. In its most simplistic form, it can be just a function. But it can be a type that implements an interface, a process, a user interface (UI), a service, an entire application programming interface (API), an application with all its layers (frontend, backend, data etc.) of functionality or a collection of subsystems that work together. Any unit of software regardless of its size or form, really!

See section on dynamics below for a formal characterization of systems.

Interfaces

We characterize the behaviors of systems through their interfaces. An interface is the atomic building block in metals. The time honored way to specify an interface is as a map from an input to an output.

f: (A) → B         // interface f as a map from A to B

Here we have the specification of an interface f that transforms A inputs into B outputs. This conforms to our concept of functions in programming. This describes the observable behavior of a system as an obligation to engage in an interaction specified by the interface.

It is even more important to understand what an interface specification does not say.

It does not say anything about the actual dynamics of this transformation of A's into B's.
It does not say anything about the internal states. By looking at this interface, we can't tell if this is a stateful or stateless system. We don't know if this represents a pure computation or one with side effects. There is no referential transparency guarantee that the same inputs shall produce same output.
The interface says nothing about the space, whether this computation is local or remote. It works with A and B types as per the representations supported by the runtime used by the consumers of this interface.
It hardly says anything about time either. is it synchronous or asynchronous? Interface is silent on the subject. This interface only guarantees to produce the output A eventually, if the consuming system is willing to wait for it.

We need to build all these notions on top of this starting model while preserving the simplicity of interface specification. Our goal is to create a model framework to provide such extensions. We will do this systematically from ground up using, you guessed right, composition.

States and Dynamics

While interfaces specify the observed behavior, we need states to model the dynamics of the system. A complete specification of the system is achieved by mapping a state to an interface.

A system in our discussions refers to this mapping from a state to an interface.

system: (S) → I  // A system as a map from state S to Interface I

An implementation of a system is the realization of this mapping. It represents a unit of computation.

Notational Conventions

Since we are building a meta language for such specifications, using Rust to build it and finally be used from other programming languages, we will need a few notational conventions.

A generic abstract notation that is independent of any programming language.
Their concrete specifications in Rust.
The mapping of the abstract notation to other programming languages. We will not be covering this in this document.

Let us look at some of the basic conventions. We will refine them as we go along.

Interfaces, Interface Types and States

We write <Input, Output> to represent an interface from Input to Output.

<Input, Output>     // An interface from Input to Output

We use the capital letters A and B with or without numbers for input output types respectively. For example, A, A₁ or A1 for inputs and B, B₁ or B1 for outputs.

An interface type I<A, B> represents any interface from A to B. It is the family of all interfaces from A to B. We will also use I, I₁ or I1 or capitalized names to denote to interface types.

I<A, B>: Type of interfaces from input type A to output type B
Handlers<HttpRequest, HttpResponse>: Interfaces from HttpRequest to HttpResponse

Some of the key ideas we use in characterizing the systems and their behaviors come from the area of Polynomial Functors, especially from the groundbreaking work of David I. Spivak and his collaborators. We highly encourage readers to read the books on Polynomial functors and Dynamical systems theory. There are several educational videos and even a complete Polynomial Functors course on YouTube.

From the point of view of polynomials, an interface <A, B> can be thought of as a monomial of the form By^A. We will use this notation sparingly, but the fact that dynamic systems and their interfaces can be represented as polynomials and their interactions can be understood in terms of polynomial arithmetic is both fascinating and critical to understanding our approach. Whenever necessary, we will use the polynomial representation to clarify such ideas. We will write a monomial as,

By^A.

We also write it as,

I = By[A]: A monomial representing interface from input type A to output type B

When we deal with multiple interfaces, we will use labels to distinguish individual interfaces. A labelled interface is written as below.

I<A, B>: I is the interface type representing any interface from A to B
i<A, B>: i is a labelled interface, a specific instance of type I<A, B>

// This means i has type I
i: I<A, B>

I<(i32, i32), i32>      //Interface type specifying any interface from 2-tuple (pair) of i32's to i32
add<(i32, i32), i32>    // A specific interface labelled add of type pair of i32's to i32
sub<(i32, i32), i32>    // Another interface labelled subtract of type pair of i32's to i32

add, sub: I<(i32, i32), i32> // Both add and sub have type I

Handlers<HttpRequest, HttpResponse> // An interface type for HttpHandlers
handler<HttpRequest, HttpResponse>  // A specific handler of type Handler

handler: Handlers<HttpRequest, HttpResponse> // handler has type Handlers

We use lowercase letters to represent labels of interfaces and uppercase letters for interface types.

A system with a state s and an interface i is written as,

[s, i<A, B>]    // Interface labelled i from type A to type B with state s
s: S            // State s is of type S

We use capitalized names, S for example, to denote types of states. They are usually encoded as some type implementing the interface and refer to them as implementing type. We will redefine them in terms of interfaces soon.

We have to combine an interface with a state to get an executable or runnable system.

We also use _ when we are specifying generic types. For example,

[_, i<A, B>]: Refers to any system implementing interface from A to B
[S, i<_, B>]: Refers to an interface from any type to B implemented by S
[S, i<A, _>]: Refers to an interface from A to any type implemented by S

Multiple Interfaces

Systems have multiple interfaces. A calculator, for example, has interface for addition, subtraction, multiplication, division, etc. Let us consider two interfaces,

add<(i32, i32), i32>
sub<(i32, i32), i32>
// Both have the same interface type
add, sub: I<(i32, i32), i32>

An interface that supports both add and sub behaviors can be constructed by combining individual interfaces,

Calc: add<(i32, i32), i32> | sub<(i32, i32), i32>
// Or we may write this in short as,
Calc<add | sub>

We just added two monomials to get a polynomial.

Calc = (i32) y^{(i32, i32)} + (i32) y^{(i32, i32)}

This is an algebraic sum or disjoint union of two interfaces. This is defining a sum type for those of you familiar with _Algebraic Data Types (ADT).

Generalizing to a system with multiple interfaces i1<A1, B1>, i2<A2, B2>, ..., in<An, Bn>, we write,

// In our notation
I: i1<A1, B1> | i2<A2, B2> | ... | in<An, Bn>   
// In polynomial notation
I = B1 y[A1] + B2 y[A2] + ... + Bn y[An]

Polynomial notation in full glory,

I = (B1) y^A1 + (B2) y^A2 + ... + (Bn) y^An.

A calculator (the system) implementing this interface can be written as,

Calculator[(), add<(i32, i32), i32> | sub<(i32, i32), i32>]
// () indicates unit type for state
// Or shorter
Calculator[(), add | sub]
// Or even shorter
Calculator[(), Calc]
// Above, Calc is the sum interface defined as before Calc<add | sub>
// We may even write this by combining state and system together
[Calculator, Calc]

Please note that Calc is the interface type, () is the unit state and Calculator is an implementing system.

A system with state S implementing the generalized interface i can be written as,

[S, i1<A1, B1> | i2<A2, B2> | ... | in<An, Bn>]
// Or shorter
[S, i1 | i2 | ... | in]
// Or even shorter
[S, I]

S is the type of state and I is the interface type in the above.

At this point, we have recovered the standard definition of an interface that we are all familiar with as programmers - a group of named (labels) functions (interfaces) with input/output signatures (interface type).

Let us translate this to Rust.

#![allow(unused)]
fn main() {
// The trait is what is equivalent to an interface type in Rust
// This defines Calc as the sum interface, add + sub.
trait Calc {
    fn add(i: (i32, i32)) -> i32; //labelled interface with type (i32, i32) → i32
    fn sub(i: (i32, i32)) -> i32; //labelled interface with type (i32, i32) → i32
}

// The system implementing the interface
// In this case, it has no internal state
struct Calculator;
// The actual implementation details
impl Calc for Calculator {
    // x-- snip the implementation details
  fn add((m, n): (i32, i32)) -> i32 { m + n }
  fn sub((m, n): (i32, i32)) -> i32 { m - n }
}
}

A general interface or sum type maps to a trait in Rust. An implementing type with unit state is defined using a unit struct.

A general interface in metals is equivalent to a trait in Rust. But we don't use the term trait outside of Rust implementation discussions.

We refer to a general interface simply as an Interface or a Frame.

It is because we use the concept of interface to model and compose systems and their states, not just interfaces.

It is for the same reasons we use | to denote our disjoint union instead of + as in Rust. In our composition of systems, we go into products and concurrent products in addition to sums.

Understanding Sum of Interfaces in `metals`

It may appear that we have just managed to duplicate a basic programming language feature using a new set of notations.

In metals, we are composing systems. In addition to composing two interfaces using sum, I = I1 | I2, to define a new composite interface, we can compose two systems implementing those interfaces,

[S, I] = [S1, I1] | [S2, I2].

The composite system on the left has the interface type I = I1 | I2. Moreover, we are constructing a system that can operate in two modes, either as a system with interface I1 or as a system with interface I2, each of which may have a different underlying system implementing it with different state (S1 and S2 above). We are incorporating the context of systems implementing the interfaces into the composition.

For example, consider two interfaces,

I1: <A1, B1> // Interface type A1 to B1
I2: <A2, B2> // Interface type A2 to B2
I: I1 | I2   // Sum of interfaces

Consider the systems implementing these interfaces,

Y1: [S1, I1] // Y1 is the system with state type S1 implementing interface type I1
Y2: [S2, I2] // Y2 is the system with state type S2 implementing interface type I2
Y: [S, I]    // Y  is the system with state type S  implementing combined interface

Here there is no relationship between the three systems, even though there is a relationship between their interface types. The systems are independent of each other. Interface I is composed of I1 and I2, but Y is a completely independent implementation of this combined interface. You can picture the situation as follows,

y1: (s1, a1) → b1  // When invoked with input a1 (type A1), produces b1 (type B1) using its state s1
y2: (s2, a2) → b2  // When invoked with input a2 (type A2), produces b2 (type B2) using its state s2
y:  (s, a1)  → b1  // When invoked with input a1 (type A1), produces b1 (type B1) using its state s
 Or (s, a2)  → b2  // When invoked with input a2 (type A2), produces b2 (type B2) using its state s

// s1 != s2 != s (!= means not equal)

The composition is only skin deep! You are just defining a new interface type.

Now, consider different kind of composition of systems,

Y1 = [S1, I1] // System Y1 implementing interface I1
Y2 = [S2, I2] // System Y2 implementing interface I2
Y = Y1 | Y2   // Y is a system composed of Y1 and Y2

Y has the interface type I: I1 | I2 as before. But it is composed of two subsystems where it operates S1 with inputs of type A1 and S2 with inputs of type A2. In terms of state, the situation now is,

y1: (s1, a1) → b1  // When invoked with input a1 (type A1), produces b1 (type B1) using its state s1
y2: (s2, a2) → b2  // When invoked with input a2 (type A2), produces b2 (type B2) using its state s2
y:  (s1, a1) → b1  // When invoked with input a1 (type A1), produces b1 (type B1) using s1
 Or (s2, a2) → b2  // When invoked with input a2 (type A2), produces b2 (type B2) using s2

It is true composition of two systems! We are constructing a new system.

We want both, an ability to combine systems or create a new system that implements a combined interface.

A sum or polynomial interface represents a composed system that can act as any one of its subsystems, one at a time.

It is as though we have a box with a mode selector button. Anywhere we need to make B_i's out of A_i's, we jut have to set the mode to i position first and then run it with A_i.

The system represented by a sum interface has acquired a state, even if the underlying system implementing the behavior represented by the interfaces has state or not. We need to pick the right interface before we can call it with the right input. In our calculator example, we have to select add or subtract operation first and then pass the pair of numbers.

Interface Maps and Representation of General Interfaces and Systems

Let us summarize our discussion so far.

Simplest form of Interface is <A, B> where A is the input type and B is the output type.
An Interface Type I<A, B> defines a type of any interface with input A and output B.
A System implementing an interface, Y: [S, I] associates a State type S with an Interface Type I.
The general form of an interface is a disjoint sum of single interfaces, I = I1 | I2 | ... | In (∑_iI_i<A_i,B_i> using a shorter notation and ∑_iB_i y^A_i in polynomial notation)
In metals, we can not only compose interfaces to form a combined interface, but also compose systems implementing those interfaces to form a new system.
A general interface introduces modality to the system's interface where by introducing an additional step in the interaction with the system.

From this point onwards, when we refer to an interface, we mean a generalized interface of the form I1 | I2 | ... | In or ∑_iI_i<A_i,B_i>. It is a polynomial ∑_iB_i y^A_i

Introducing Interface Maps

As we saw above, an interaction with a system with a general interface is no longer a simple call with an input. We have to pick a specific component system with one of the component interfaces (one of the I_i's) first and then call it with the input. We have a mode selector interface of the form <Mode, I>.

I = I1 | I2 | ... | In
InterfaceSelector: <Mode, I> //Interface selection interface type

// Interface selector accepts some input of type Mode and returns a value
// of the _Interface Type_ I, which is one of I1, I2, ...or In.

It appears that we need a way to translate this interaction pattern in terms of our language of interfaces. The key lies in looking at the interface definition with our understanding that interfaces themselves are types.

Let us revisit our original definitions

<A, B>    // Interface from type A to type B
I: <A, B> // Interface Type from type A to type B

Since interfaces are types, let us replace A and B with some other interface types. We have 3 cases.

Case 1: A is an interface type, I1<A1, B1>

I1: <A1, B1>   // Interface Type from type A1 to type B1
A = I1         // A in I is an interface of type I1
I: <I1, B>     // I takes an interface type I1, returns a value of type B

This is the interface description for evaluating or running an interface I1 with input A1, then converting B1 values to B values. That means the interface describes a system converting A1's to B's.

We can capture this idea of turning an interface <A1, B1> into a <A1, B> as map of interfaces, an IMAP as follows,

I:= <A1, B>     
IMAP: <(<A1, A1>, <B1, B>), <A1, B>>

IMAP takes two interfaces, one from A1 to A1 (an identity map) and one that converts B1 into B and returns an interface from A1 to B.

Case 2: B is an interface type, I1<A1, B1>

I1: <A1, B1>   // Interface Type from type A1 to type B1
B = I1         // B in I is an interface of type I1
I: <A, I1>     // I takes an input of type A and returns an interface type I1

This system turns an interface from A1 to B1 into an interface from A to B1.

This can be expressed as an IMAP as follows,

I:= <A, B1>
IMAP: <(<A, A1>, <B1, B1>), <A, B1>>

IMAP takes a map from A to A1 and a second one from B1 to B1 (an identity map) and returns an interface from A, B1.

Case 3: Both A and B are interface types, I1<A1, B1> and I2<A2, B2>

I1: <A1, B1>   // Interface Type from type A1 to type B1
I2: <A2, B2>   // Interface Type from type A2 to type B2
A: I1          // A in I is an interface of type I1
B: I2          // B in I is an interface of type I2
I: <I1, I2>    // I takes an interface type I1 and returns an interface type I2

This is a more general form of mapping between interfaces. This converts an interface from A1 to B1 into an interface from A2 to B2.

As an IMAP, this is,

I:= <A2, B2>
IMAP: <(<A2, A1>, <B1, B2>), <A2, B2>>

Please note that the two maps that IMAP uses goes in opposite directions. On inputs, it goes from A2 to A1 while on outputs it goes from B1 to B2.

We can think of <A1, B1> as the inner interface type and <A2, B2> as the outer interface type.

Readers familiar with functional programming will recognize the relation to contravariant and covariant functors as well as profunctors here.

We will generalize the IMAP we defined above to the general interface case, those with multiple interfaces.

Before we do that, let us talk about the types and states that appear in our definition of interfaces, those A's and B's on interfaces and S's in the definition of systems.

Data Types and States as Interfaces

We already introduced Interface Types as types representing interfaces. All data types can be represented as interfaces as well. There is a duality between interfaces and data types.

This ideas of sum and product types must be very familiar to those who have some background in Algebraic Data Types (ADT). Here are some references that provide simple introduction to algebraic data types for programmers in [Javascript][adt-js], [Rust][adt-rust] and in [Swift][adt-swift].

There are two special data types we will start with, a zero type and a unit type. In Rust, they translate to the types never ! and unit () respectively. They represent types that can take no value and exactly one value. They are represented by constant polynomials 0 and 1.

We can translate them to interface types Zero and Unit as follows.

Zero: <0, 0>    // Interface Type from never type to never type
Unit: <1, 1>    // Interface Type from unit type to unit type

The data types 0 and 1 are translated to an appropriate data type in the target language. In Rust, they translates to,

Zero: <!, !>
Unit: <(), ()>

As one would expect, adding Zero interface type to an interface type I returns I and adding Unit interface type to a type I + Unit.

I + Zero = I
I + Unit = I | Unit

A Unit interface represents a pure side effect, a system behavior with no observable input or output.

Note: To fully understand the statements we are making about a system behavior, we need the machinery of interface maps and the concept of a system as a map from state to an interface type, S → I as well as the idea that a state has an interface type <S, S>. It will become clearer by the end of next section

Any data type A is a constant polynomial of the form Ay⁰. Translating it to an interface type,

A: <0, A>

A data type A represented by interface <0, A> describes a behavior of read only access to value of of type A.

We can extend the above definitions to other combinations of A, 0 and 1.

A data type A with a power polynomial of the form 0y^A has interface,

Panic: <A, 0>   // Interface Type from type A to never type

An interface <A, 0> represents a pure panic behavior of the system.

A data type A with a power polynomial of the form 1y^A or simply y^A has interface,

UpdateInternalState: <A, 1>  // Interface Type from A to Unit

An interface <A, 1> represents a side effect or unobserved internal state changing behavior of the system based on the values of type A.

A data type A with a linear polynomial of the form Ay¹ or simply Ay has interface,

UpdateRead: <1, A>   // Interface Type from Unit to A

An interface <1, A> represents an update followed by read of values of A. The update is completely internal to the system.

A state S is represented by a monomial of the form Sy^S with interface,

State: <S, S>   // Interface Type from S to S

An interface <S, S> represents a state transition system with states from values of S.

A state transition system or state system for short, takes the current state and returns a new state.

The idea that states are interfaces and interfaces can be represented as data types is a central concept in metals.

Putting It All Together

In last three sections, we introduced the idea of general interfaces and systems, discussed the concept of maps between interface types and established that there is a duality between data types and interfaces.

We described a system as a map from states to a set of observable behaviors encoded as an interface, (S) → I. We used a special notation [S, I], to represent a system with state S and interface I where I = ∑_iI_i<A_i,B_i>.

Since a state system S, has interface type <S, S> (Sy^S), the above system definition translates to an interface type with interface types as input and output,

System: [S, I]
System: <<S, S>, I>    // System has an interface type from <S, S> to I

If I is an interface <A, B> then, the system has interface type,

System: <<S, S>, <A, B>>

The behavior of the system can be described as follows,

Uninitialized: An uninitialized system shall accept initial state s₀ ∈ S. System is not ready to accept inputs A yet.
Initialized: Once initialized, the system shall turn into an <A, B> system and ready to accept input A.
On Input: The system shall return the output of type B.

Someone interacting with an initialized system, can't distinguish it from another system implementing interface <A, B> without a state.

There are many ways to implement the above interface and the behavior. One way to do this is to use an IMAP we defined earlier. Our system as an IMAP,

System: IMAP<(<A, S>, <S, B>), <A, B>>

The system has inner state S with interface <S, S> representing the state transitions and outer interface <A, B>, which is the observable behavior of the system. The maps <A, S> and <S, B> are referred to as the update and read maps respectively.

If we implement the system using an IMAP, it will simulate the behavior we described earlier as follows,

Uninitialized: An uninitialized system shall accept initial state s₀ ∈ S. System is not ready to accept inputs A yet.
Initialized: Once initialized, the system shall turn into an <A, B> system and ready to accept input A.
On Input: The system shall
1. Use the update map to updates its state,
2. Use the read map to generate output and return it.

Again, an initialized system is indistinguishable from any other system with interface <A, B>.

The IMAP implementation is reminiscent of most state management systems out there with update and read functions,

update: S x A → S

read: S → B

In Redux, one of the most popular state management frameworks used by user interface (UI) developers, the update signature is exactly that of a reducer.

But the subtle difference is that our interface definition is a prescription for turning a state system S into a <A, B> system,

(s) → ((a) → b).

The state management system is a prescription for the conversion

(s, a) → (s, b).

It is converting a (state, input) tuple into a (state, output) tuple. It is a different behavior. It is possible for us to translate this to our interface language. We will have a lot more to say later about systems with states especially regarding mutability, locality and synchronization of state.

Let us focus our attention to a system with state S and general interface I = ∑_iI_i<A_i,B_i>.

THIS SECTION IS STILL WORK IN PROGRESS

Contributors

Here a list of contributors to this book.

Weavers @ Eternal Loom

Things Left Todo

This is a work in progress document. Here is a list of things we are working on.

Add WIP

Metals - Meta Programming Language System