Posting with Metals

In the Rust book, there is a chapter on Implementing an Object-Oriented Design Pattern which describes a stateful Posts API at some length. We will revisit this example in the context of metals.

Note: The primary objective of the original example and code used in the Rust book is educational, not building production ready code. This article should be read in the same spirit and not as a comparison of design decisions or superiority of technical choices.

Original Post Example from Rust Book

The book discusses the State pattern and how to go about implementing it in Rust. The crux of the pattern is that the behavior of a value can change based on a set of states which we track internally.

The book first implements this pattern closely following the object-oriented way first and then refines it using the Rust language capabilities. We will look at both approaches here.

Overview of functionality

Here is a summary of the functionality from the book for each reference:

A blog post starts as an empty draft.
When the draft is done, a review of the post is requested.
When the post is approved, it gets published.
Only published blog posts return content to print, so unapproved posts can’t accidentally be published.

Object oriented implementation

The code below shows the intended use of this API (reproduced from original article). This will not run yet.

fn main() {
    use blog::Post;

    // Create new post
    let mut post = Post::new();

    // Add text to the post
    post.add_text("I ate a salad for lunch today");
    // Since this post is still in draft and not yet approved, you will
    // not see the content of the post yet.
    assert_eq!("", post.content());

    // Request review of the post
    post.request_review();
    // Content is still not visible
    assert_eq!("", post.content());

    // Approve the post
    post.approve();
    // Now you have access to the content of the post
    assert_eq!("I ate a salad for lunch today", post.content());
}

Here is the module that implements the Post functionality as expected by the text and code above. Again, this is the same code from original book chapter, with a lot of added comments to make it easier to follow for those less familar with Rust syntax and also to bring out some of our observations.

#![allow(unused)]
fn main() {
pub mod blog {
    /// Post has an internal state and string content
    pub struct Post {
        state: Option<Box<dyn State>>,
        content: String,
    }

    // This is the Post interface. This shows everything you can do with
    // Post
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state
        /// - It has empty content
        pub fn new() -> Post {
            Post {
                state: Some(Box::new(Draft {})),
                content: String::new(),
            }
        }

        /// Add text to the post
        /// - It appends the new text to existing content
        /// - This is not a stateful operation. The way it is defined
        /// here implies that you can add text to posts that are
        /// pending review or approval.
        /// - This is not very difficult to alleviate. We just have to
        /// delegate this to state as we do with other stateful
        /// operations
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Returns the content
        /// - It delegates to the current state so that the state
        /// can decide the right thing to do (either return an content
        /// or the actual content)
        pub fn content(&self) -> &str {
            self.state.as_ref().unwrap().content(self)
        }

        /// Request review of the post
        /// - This may cause a state transition. Here we don't have to
        /// worry about the whole associated state transition, but the
        /// State implementation will take care of it.
        /// - But we have to check the code for each state to see what
        /// happens when we request a review.
        /// - If you look at the code, you will see the following valid
        /// transitions between state when we invoke request_review
        ///     Draft -> PendingReview
        ///     PendingReview -> PendingReview (no transition, a no-op)
        ///     Published -> Published (no transition, a no-op)
        /// - See comments on state implementation
        pub fn request_review(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.request_review())
            }
        }

        /// Approve the post
        /// - Similar comments as request_review() above are relevant here.
        /// - The supported transitions from State code is
        ///     Draft -> Draft (no transition, a no-op)
        ///     PendingReview -> Published
        ///     Published -> Published (no transition, a no-op)
        pub fn approve(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.approve())
            }
        }
    }

    /// This defines the interface for state. This is currently a
    /// private one within this module (no `pub` modifier in front of
    /// `trait` keyword).
    /// - Good thing is that the users of the _Post_ API does not have
    /// to know about state and can completely rely on _Post_ interface
    /// we defined above.
    /// - The bad thing is that _State_ is hidden from the users of
    /// _Post_ API. One could argue that it is very important domain
    /// characteristic that impacts how the API behaves. Shouldn't
    /// the API user know that the a _Post_ is in _Draft_ state and
    /// hence has to be submitted for review?
    /// This also defines common interface to be implemented by all
    /// states, irrespective of whether this is relevant in that state.
    trait State {
        /// Represent a state action which _may_ a transition.
        fn request_review(self: Box<Self>) -> Box<dyn State>;
        fn approve(self: Box<Self>) -> Box<dyn State>;
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            ""
        }
    }

    /// Represents the Draft state
    struct Draft {}

    /// Here is the implementation of State interface for Draft.
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for Draft {
        /// Represents the transition of state
        ///     Draft -> PendingReview
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            Box::new(PendingReview {})
        }

        /// The approve is a no-op, just trivially return the current
        /// state.
        /// Shouldn't this be disallowed. Without going through the
        /// code, it difficult to understand the what we are doing here.
        /// Can someone approve a draft directly? Interface says
        /// 'Yes', but code says it ignores it.
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }
    }

    /// Represents PendingReview state
    struct PendingReview {}

    /// The implementation of State interface for PendingReview
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for PendingReview {
        /// This is actually a no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// Transition to Published state
        ///     PendingReview -> Published
        fn approve(self: Box<Self>) -> Box<dyn State> {
            Box::new(Published {})
        }
    }

    /// Represents Published state
    struct Published {}

    /// The implementation of State interface for Published.
    /// There are no transitions out of this state.
    impl State for Published {
        /// A no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// A no-op
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// This overrides the default implementation from State
        /// and returns the content of the associated post
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            &post.content
        }
    }
}
}

Now you can run the code from first block using our Post API and see that it is working as intended.

/*
 * Copyright 2022 Weavers @ Eternal Loom. All rights reserved.
 * Use of this software is governed by the license that can be
 * found in LICENSE file in the source repository.
 */
/// Original blog post example from [`Rust Book`] [`chapter`].
///
/// [`chapter`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// [`Rust Book`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
fn main() {
    use blog::Post;

    // Create new post
    let mut post = Post::new();

    // Add text to the post
    post.add_text("I ate a salad for lunch today");
    // Since this post is still in draft and not yet approved, you will
    // not see the content of the post yet.
    assert_eq!("", post.content());

    // Request review of the post
    post.request_review();
    // Content is still not visible
    assert_eq!("", post.content());

    // Approve the post
    post.approve();
    // Now you have access to the content of the post
    assert_eq!("I ate a salad for lunch today", post.content());
}

pub mod blog {
    /// Post has an internal state and string content
    pub struct Post {
        state: Option<Box<dyn State>>,
        content: String,
    }

    // This is the Post interface. This shows everything you can do with
    // Post
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state
        /// - It has empty content
        pub fn new() -> Post {
            Post {
                state: Some(Box::new(Draft {})),
                content: String::new(),
            }
        }

        /// Add text to the post
        /// - It appends the new text to existing content
        /// - This is not a stateful operation. The way it is defined
        /// here implies that you can add text to posts that are
        /// pending review or approval.
        /// - This is not very difficult to alleviate. We just have to
        /// delegate this to state as we do with other stateful
        /// operations
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Returns the content
        /// - It delegates to the current state so that the state
        /// can decide the right thing to do (either return an content
        /// or the actual content)
        pub fn content(&self) -> &str {
            self.state.as_ref().unwrap().content(self)
        }

        /// Request review of the post
        /// - This may cause a state transition. Here we don't have to
        /// worry about the whole associated state transition, but the
        /// State implementation will take care of it.
        /// - But we have to check the code for each state to see what
        /// happens when we request a review.
        /// - If you look at the code, you will see the following valid
        /// transitions between state when we invoke request_review
        ///     Draft -> PendingReview
        ///     PendingReview -> PendingReview (no transition, a no-op)
        ///     Published -> Published (no transition, a no-op)
        /// - See comments on state implementation
        pub fn request_review(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.request_review())
            }
        }

        /// Approve the post
        /// - Similar comments as request_review() above are relevant here.
        /// - The supported transitions from State code is
        ///     Draft -> Draft (no transition, a no-op)
        ///     PendingReview -> Published
        ///     Published -> Published (no transition, a no-op)
        pub fn approve(&mut self) {
            if let Some(s) = self.state.take() {
                self.state = Some(s.approve())
            }
        }
    }

    /// This defines the interface for state. This is currently a
    /// private one within this module (no `pub` modifier in front of
    /// `trait` keyword).
    /// - Good thing is that the users of the _Post_ API does not have
    /// to know about state and can completely rely on _Post_ interface
    /// we defined above.
    /// - The bad thing is that _State_ is hidden from the users of
    /// _Post_ API. One could argue that it is very important domain
    /// characteristic that impacts how the API behaves. Shouldn't
    /// the API user know that the a _Post_ is in _Draft_ state and
    /// hence has to be submitted for review?
    /// This also defines common interface to be implemented by all
    /// states, irrespective of whether this is relevant in that state.
    trait State {
        /// Represent a state action which _may_ a transition.
        fn request_review(self: Box<Self>) -> Box<dyn State>;
        fn approve(self: Box<Self>) -> Box<dyn State>;
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            ""
        }
    }

    /// Represents the Draft state
    struct Draft {}

    /// Here is the implementation of State interface for Draft.
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for Draft {
        /// Represents the transition of state
        ///     Draft -> PendingReview
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            Box::new(PendingReview {})
        }

        /// The approve is a no-op, just trivially return the current
        /// state.
        /// Shouldn't this be disallowed. Without going through the
        /// code, it difficult to understand the what we are doing here.
        /// Can someone approve a draft directly? Interface says
        /// 'Yes', but code says it ignores it.
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }
    }

    /// Represents PendingReview state
    struct PendingReview {}

    /// The implementation of State interface for PendingReview
    /// Note: This inherits the default content() implementation from
    /// State which returns an empty string.
    impl State for PendingReview {
        /// This is actually a no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// Transition to Published state
        ///     PendingReview -> Published
        fn approve(self: Box<Self>) -> Box<dyn State> {
            Box::new(Published {})
        }
    }

    /// Represents Published state
    struct Published {}

    /// The implementation of State interface for Published.
    /// There are no transitions out of this state.
    impl State for Published {
        /// A no-op
        fn request_review(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// A no-op
        fn approve(self: Box<Self>) -> Box<dyn State> {
            self
        }

        /// This overrides the default implementation from State
        /// and returns the content of the associated post
        fn content<'a>(&self, post: &'a Post) -> &'a str {
            &post.content
        }
    }
}
// --snip-- the rest of the code from above two blocks

Analysis of object-oriented Post implementation

The original book chapter goes through a lot of details on the design and implementation choices that were made as well as their pros and cons. We will discuss some of them here and then add a few of our own observations. We highly encourage you to read the original book chapter. We hope
the discussion shall help everyone, at the very minimum, appreciate the gamut of design thinking and choices involved in even simplest of API's.

Modeling considerations that we have been discussing in the context of metals were not the focus of the authors of the Rust book in the chapter. So, the points we are brining out here are not those of omission or inadequate design considerations, but more a matter of content choice on a specific subject.

From a model perspective (please keep our comments from previous paragraph in mind), there are a lot of technical details that can be intimidating even for technical folks without fair understanding of Rust. What the hell is Option<Box<dyn State>?
While enum variant is a good choice to represent mutually exclusive states, the choice (of trait object) made in the code provides better ergonomics at the API call site (by doing away with the need for match expressions).
There is clear separation of concerns. Post knows nothing about various behaviors. It also promotes modularity and clarity. There is one place to look at how the system behaves in a given state.
The Rust book chapter has some suggested extensions and hints to implement them. It is seems easy to extend the system with new behaviors. Well, to an extend!
There is strong coupling between states as they implement the transition to next state.
Using the example from the original book chapter, if we add another state between PendingReview and Published, such as Scheduled, we would have to change the code in PendingReview to transition to Scheduled instead.
The transitions are pretty much embedded in code for different states. We have to go through the code for each State to understand the real state transition model of the system.
There is duplication of code where we have to implement all state actions on every state even when they are no relevant transitions.
We also have to duplicate state actions such as request_review and approve on Post which simply delegates to their counterparts in State.
We definitely don't make illegal states unrepresentable. We are returning empty string for content in states where content access is illegal.
Externally extending this API can be tricky. In technical jargon, this implementation suffers from an expression problem.
Adding a new behavior would need us to reach into Post implementation code to add new things. Imagine adding reject action or reset_text functionality.
As it is implemented here, we would need to reach into Post implementation code to add them.
Referring back to our discussions on modeling data, this model represents an application programming view.
The primary purpose of Posts is to manage its content. We can think of many ways in which we want to allow the post content to be manipulated - reset the content, change parts of the content, maintain change history, add review comments, so on and so forth.
We will end up adding a lot of code all across states!
This pattern does not address many data modeling concerns. For example, the request_review cause a transition to PendingReview only if the content is nonempty.
To be fair, that is not what this pattern was designed for.
All states operate on the same globally shared data instead of them operating on just the parts that they are interested in.
As in most design patterns from the past decade, this is emblematic of top down, system breakdown approach to building modular systems as opposed to compositional approach to building larger system from smaller ones.

Doing it the Rusty way

The original book chapter also shows how to implement this pattern differently that is more idiomatic of Rust. This is based on the idea of implementing states and behaviors as types.

Here is the revised Post code.

#![allow(unused)]
fn main() {
pub mod rusty_blog {
    /// _Post_ has String content
    /// - For starters, cleaner interface (no Box<dyn ...>> stuff)
    pub struct Post {
        content: String,
    }

    /// The interface for Post (encoding the behavior of Post)
    /// - We can create a new post, but pay attention to what it returns
    /// - We can get access to the content
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state, captured by the fact that it
        /// returns a DraftPost type
        /// - It has empty content
        pub fn new() -> DraftPost {
            DraftPost {
                content: String::new(),
            }
        }

        /// Returns the content
        pub fn content(&self) -> &str {
            &self.content
        }
    }

    /// DraftPost also has String content
    /// - Note that it is just another type
    /// - We are effectively trying to encode the _State, DraftState_ as a type
    pub struct DraftPost {
        content: String,
    }

    /// The interface or the encoded behaviors for DraftPost
    /// - We can add text to the post when you are in Draft state
    /// - We can request review of the post at which point the state would change
    impl DraftPost {
        /// Add text to the post.
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Request review of the post
        pub fn request_review(self) -> PendingReviewPost {
            PendingReviewPost {
                content: self.content,
            }
        }
    }

    /// PendingReviewPost also has String content
    pub struct PendingReviewPost {
        content: String,
    }

    /// The interface or the encoded behaviors for PendingReviewPost
    /// - One can approve the Post in this state
    impl PendingReviewPost {
        /// Approve the post
        /// - Here is when the actual Post is getting constructed
        /// - Since _Post_ has content() method on its interface, the content
        /// is available to the callers
        pub fn approve(self) -> Post {
            Post {
                content: self.content,
            }
        }
    }
}
}

We have to make small adjustments to the call site (the main function), but will retain most of it.

/*
 * Copyright 2022 Weavers @ Eternal Loom. All rights reserved.
 * Use of this software is governed by the license that can be
 * found in LICENSE file in the source repository.
 */
/// Original blog post example from [`Rust Book`] [`chapter`].
///
/// [`chapter`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// [`Rust Book`]: https://doc.rust-lang.org/book/ch17-03-oo-design-patterns.html
/// This is the re-implementation of State pattern in idiomatic rust
///  based on the idea of representing states and behaviors as types
fn main() {
    use rusty_blog::Post;
    // Create new post. Here we are getting a DraftPost
    let mut post = Post::new();

    // We can add the text in Draft state
    post.add_text("I ate a salad for lunch today");
    // We can't call content() here. It is a compilation error as
    // DraftPost does not support this behavior (nice!)
    //assert_eq!("", post.content());

    // Request review of the post
    // Since it returns a new type (PendingReviewPost), we have
    // to capture it in a variable (See let post = ... added below)
    let post = post.request_review();
    // Content is still not visible
    // It is a compilation error to try and access content
    // assert_eq!("", post.content());

    // Approve the post. We get the real Post
    let post = post.approve();

    // Now we have access to the content of the post.
    assert_eq!("I ate a salad for lunch today", post.content());
}

pub mod rusty_blog {
    /// _Post_ has String content
    /// - For starters, cleaner interface (no Box<dyn ...>> stuff)
    pub struct Post {
        content: String,
    }

    /// The interface for Post (encoding the behavior of Post)
    /// - We can create a new post, but pay attention to what it returns
    /// - We can get access to the content
    impl Post {
        /// Create a new empty post.
        /// - It starts out in Draft state, captured by the fact that it
        /// returns a DraftPost type
        /// - It has empty content
        pub fn new() -> DraftPost {
            DraftPost {
                content: String::new(),
            }
        }

        /// Returns the content
        pub fn content(&self) -> &str {
            &self.content
        }
    }

    /// DraftPost also has String content
    /// - Note that it is just another type
    /// - We are effectively trying to encode the _State, DraftState_ as a type
    pub struct DraftPost {
        content: String,
    }

    /// The interface or the encoded behaviors for DraftPost
    /// - We can add text to the post when you are in Draft state
    /// - We can request review of the post at which point the state would change
    impl DraftPost {
        /// Add text to the post.
        pub fn add_text(&mut self, text: &str) {
            self.content.push_str(text);
        }

        /// Request review of the post
        pub fn request_review(self) -> PendingReviewPost {
            PendingReviewPost {
                content: self.content,
            }
        }
    }

    /// PendingReviewPost also has String content
    pub struct PendingReviewPost {
        content: String,
    }

    /// The interface or the encoded behaviors for PendingReviewPost
    /// - One can approve the Post in this state
    impl PendingReviewPost {
        /// Approve the post
        /// - Here is when the actual Post is getting constructed
        /// - Since _Post_ has content() method on its interface, the content
        /// is available to the callers
        pub fn approve(self) -> Post {
            Post {
                content: self.content,
            }
        }
    }
}
// --snip-- the rusty_blog module code from previous snippet

Analysis of Rusty Post implementation

Let us take a closer look at the this implementation.

It makes illegal states unrepresentable. We can only change Post content while it is a Draft and can only access the content once it is approved. This is a small step for API provider, big step of API consumer productivity.
We seems to have simplified the interface a lot. Lot of scary looking things like Box<dyn ...> have disappeared from the interface.
We have removed a lot of duplicated code and unnecessary implementations.

A lot of good things have happened and looks like this is moving in the right direction. Still there are several things we have not addressed yet.

The transitions remain embedded in the code.
There is still very strong coupling between the states. Since these are types, not the interfaces (those dyn State thingies), we probably have made the problem worse. There is tremendous power in our ability to adhere to an interface contract while changing implementations. This is why build APIs in the first place.
This also impact the extensibility of the API both from inside as well as outside. Ideally, we would like to be able extend the interface of a state, say PendingReview, and provide an implementation for new interface without having to modify the existing code. We need some kind of dependency injection to reduce coupling of states and retain the power of interfaces we had in our original object-oriented implementation.
We really have not addressed any of the data modeling concerns we discussed in our previous analysis section.
On the other hand, we have completely localized the states and lost the complete idea of a shared data. May be we swung too far. We still want the concept of shared data model and data access so that we can deal with those aspects, say syncing with a datastore more efficiently and effectively.
Just our behaviors are intricately connected to states, these state we use to encode different modes of the API are linked to the underlying data. This means the state actions can fail or transition to different states depending on the data and the rules governing them. We are looking at a non-deterministic automaton as opposed to a deterministic one.
As far as state design patterns go, there are established approaches such as immutable states and fine-grained action interfaces to make it deterministic and keep it simple. All the other concerns are packed up as side effects and handled through a set of other patterns.
At the very least, we should be able to see that the complex business rules are moving away from the neat interfaces we built. The point we are making here is that in an API design, the state/interface dependency is not at the top layer, but they go all the way down. We need to be able to package them and compose them both vertically and horizontally. That sentence may not mean much now, we will return to it when discuss metals.

On Handling States

Here in this chapter, we are specifically looking at the design pattern as discussed in the Rust book. It is important to note that there are several approaches to handling states besides this single pattern, especially if you have familiarity with frontend frameworks. If you have worked in that area, you would have come across Redux, XState, MobX and many more and their integrations to frontend frameworks like React. Redux, arguably the most popular of them all, has been ported to many languages including a Rust (redux-rs).

Redux uses a more functional pattern to state management. In order to make the usage simpler and reduce the level of broiler plate code, there are also several libraries and tools available including Redux Toolkit.

There is also a rich field of formal techniques for modeling states, deterministic and nondeterministic finite automatons (DFA and NFA), Moore and Mealy machines, Harel statecharts and more (See State diagrams for a preview).

Another area that has profound insights and imputs to this world is coming from Dynamial systems theory. We derive our inspirations heavily from the work of David I. Spivak, especially his work on Polynomial functors.

Quote to Use. Here is a quote from the sipc book(The structure and interpretation of computer programs). One powerful design strategy, which is particularly appropriate to the construction of programs for modeling physical systems, is to base the structure of our programs on the structure of the system being modeled.

Metal plating the posts

THIS SECTION IS STILL WORK IN PROGRESS

Metals - Meta Programming Language System