An organic approach to state management

tbd

The Problem

As so often, my journey starts with something I’m deeply dissatisfied. There are actually a lot of things I’m unhappy with; the light in the kitchen in one corner is too dark, so I add an Arduino with bright LEDs and a lightness sensor to light it up automatically. The shed is a mess, so I drive nails into the wall and hang up all the tools there. I fix things, it’s what I’m best at.

This time, what unsettled me, was the current state of managing state in web applications. What am I’m talking about? If you open a web application, and this applications has a lot of text-boxes, then what you fill into those text-boxes has to be remembered by the application. More often than not, the application will also try to help you, marking errors if you for instance filled in the email in an incorrect format, or that the city you provided does not exist and probably has a spelling error.

Managing this state has a long history. When the internet was young, a lot of this happened on the server. You filled out all the fields, sent them by clicking on submit, the server would process the information, and you would get the result with all the errors you made. Nowadays, to be more user friendly, those errors are shown immediately, which often means that this is done directly in the browser.

With this move to the browser however, an old separation that was common in web development, vanished. The strong separation between displaying the information, and processing it.

Why is that relevant?

A typical mid sized software project can easily have hundreds of thousand lines of codes. As a software developer you will be required to navigate the code, and find the right line of code to fix a bug or add a feature. And you should not break existing functionality by doing so. That’s usually verified by a lot of tests.

Separating the ui and the logic can help you to find the right line of code easier, and have tests that verify either that the thing looks correctly, or that it works correctly. All of that does not hold true in reality, as of course the way things are displayed are often strongly coupled to how they behave, but still, it can help structure your code.

As the all the code moved to the browser, we entangled logic and ui. While modern libraries try to help you decouple the ui from the logic (ie by using libraries like redux), but more often than not some business logic gets in your ui and makes the code hard to understand, refactor and test.

Frustrated by that, I tried to come up with a solution that would allow (or even enforce) a stronger decoupling, while trying to avoid implicitness and accidental complexity that often comes for free when you try to separate your concerns.

The goal

My goal was to achieve the following:

• A separation between business logic and the ui if it makes sense.
• Independent of framework, possibly even with communication between multiple frameworks.
• Explicit and easy to understand.
• Easy to test.

The parable

Compared to life, computers are fascinatingly brittle. A little bug and you end up in a system failure, whereas there are organisms like the Tardigrade that are almost indestructible, and complex and complicated organisms like humans operate their organs in realtime. We walk, a very complex movement, while taking in a fascinating amount of data, computing it on the fly and take it into consideration for our next move, without need of a central processing. Recreating that this is what I aim to build, and achieving that goal has been eluding me of the years. Something, which always leaves me astounded on the wonders of life. Life, that operates which such seemingly simple methods such terribly complicated beings.

For this very exercise, I choose the parable of hormones. Hormones are signaling molecules, used in our bodies to modify and control how we function, and how we act. They are received by special receptors on the cells, which lead to follow up actions in the cell.

Hormones can be created in different areas in the body, and are released upon stimulation. Also, there are releasing hormones, that exist to trigger the release of other hormones. This is controlled in different parts of the body, most importantly the hypothalamus.

Following this setup, I mapped the language for this framework in the following way:

Now all I had to do was to implement it.

The result

You can find the result on the github page MatthiasKainer/Organismus, and when I started working with it, I was pretty happy with the result. A typical example project in the software development realms of web applications is a todo list, and building that was simple enough.

const todoAdd = defineHormone<string>("todo/add");
const todoList = defineHormone<string[]>("todo/list", { defaultValue: [] });

hypothalamus.on(todoAdd, (todo) =>
  releaseHormone(todoList, (todos) => [...todos, todo])
);

pureLit("nice-list", (element) => {
  const list = useState(element, [])
  useReceptor(element, element.receptor, async value => list.publish(value));
  return html`<ul>
    ${list.map((item) => html`<li>${item}</li>`)}
  </ul>`;
});

export default pureLit("todo-application", () => {
  return html`
    <text-input @onChange=${(value) => releaseHormone(todoAdd, value)}>
    </text-input>
    <nice-list .receptor=${todoList}></nice-list>
  `;
});

As we can see, it all starts with two hormones. One, todoAdd, is released when a todo is added to the list of todos, and acts solely as a releasing hormone for todoList via the hypothalamus, a second hormone that transports the state of the todo list.

The ui, created by pure-lit, a functional extension on top of polymers LitElements, needs to contain no knowledge on the business logic, that is handled entirely by the hypothalamus.

The receptor on the nice-list is passed in from the outside, so it can easily be used in other applications, as long as the hormone that is released contains a list of strings.

Using a strongly typed language like typescript can make this very explicit and easy, as the typed hormones help you understanding the contract for them, without having to look at the code.

Doing the same in react was easy enough as well.

const todoAdd = defineHormone<string>("todo/add");
const todoList = defineHormone<string[]>("todo/list", { defaultValue: [] });

hypothalamus.on(todoAdd, (todo) =>
  releaseHormone(todoList, (todos) => [...todos, todo])
);

export const List = ({receptor}) => {
  const list = useState([])
  useReceptor(this, receptor, async value => list.publish(value));
  return <ul>
    {list.map((item) => <li>${item}</li>)}
  </ul>;
});

export const Todo = () => {
  return <>
    <TextInput onChange={(value) => releaseHormone(todoAdd, value)} />
    <List receptor={todoList} />
  </>;
});

On the high level, it’s actually very similar to libraries like redux. They main difference is however that there is no central store, but one in every hormone; and that hormones can be released everywhere, not just components. That allows you release them outside of your element tree, and it allows you to receive it anywhere, not just your current framework.

I needed those two requirements, because mostly this library should not be the perfect fit for a todo list, but for far more complicated environments. Environments like the ones I have at work. And those can be broken down to the following three types of applications:

Whenever I try something in the frontend, these are my tests for the piece of code before me.

In addition, I’m working with more and more multi-framework projects. Where Angular has to talk to React has to talk to WebComponents. Managing a global state can become difficulty in such environments, and this framework should show me if that issue can be resolved.

Testing the result

You can see examples for the code (without backend) here: matthiaskainer.github.io/Organismus

Search

Search is not so different from our todo list, and similarly we can start with two hormones. One that is released when the search term is set, another one to fill the result list.

const query = defineHormone("search/query", {
  defaultValue: "",
});
const filtered = defineHormone<ListReceptor>("search/filtered", {
  defaultValue: [],
});

As before, the query hormone is a releasing hormone for the filtered hormone, but unlike before it will be triggered only after asynchronously we received the result from the server.

hypothalamus.on(query, (value) => {
  fetch("/api/search?q=" + encodeURIComponent(value))
    .then((response) => response.json())
    .then((results) =>
      releaseHormone(filtered, results)
    );
});

The html can look the same as in the todo list, so I’ll skip it. But the fact that it so easy to skip that part shows that splitting the logic and the display can come with benefit.

Testing this is also very easy. All you have to do is mocking the fetch, and then you release the query and listen in a receptor that the expected result is returned.

Now let’s take a look at the additional requirement: If the result list changes on the server, it should be updated on the client as well. One way I had implemented that in the past was with web-sockets. It keeps a connection with the server open, the server can emit events to trigger changes, and the client (browser) can send messages. The nice thing about how we set this up is that it works nicely with this setup. Rather than triggering the hormone directly however, it will send messages from the server and receive the replies. The code for the hypothalamus changes to:

const searchSocket = new WebSocket("wss://www.example.com/api/search")
hypothalamus.on(query, searchSocket.send)
searchSocket.onmessage = (results) => releaseHormone(filtered, results.data);

There. That looks like a lovely start, not like a failure. Post a medium article about it and people might actually feel I solved all their problems and start using it.

But as we will see, complexity grows with the use case, and this might not be the library you were hoping for after all.

Forms

Spreadsheets

Spreadsheets are basically a collection of rows and cells, that can be referenced by other cells, and every developer that had to build something like it may have experienced how difficult it can be to do this in a performant and yet controlled way.

It’s one of my favorite examples, because it involves a lot of complexity in the ui to avoid re-rendering the whole screen for every change, and by the mere fact that there can be a lot of references to compute and update. Also you have to find a good way to become aware if something has changed, to recompute your data.

Tackle this incorrectly, and your application will not be able to handle more then a few hundred cells before slowing down miserably.

Let’s start by looking at the hormones we might need. The first two should be pretty obvious - to set a cell, and to notify others that it was changed.

Why not use the same hormone for both? Because the value that others might be interested in, is not necessarily the one it was set to. Let’s take a sum example: When changing the cell, I set it to =5+5. A cell, that has a reference to this field is however not interested in the formula, but the result, which is 10. So a cell in our spreadsheet receives =5+5, calculates the value, and notifies the others that the value has changed.

Also we will need a third - when a cell contains a formula (and thus a reference to other fields), it has to request the value of those fields to be able to compute the calculation.

The code for this would be roughly like this:

const onCellSet = defineHormone<Cell>("cell/set");
const onCellChanged = defineHormone<Cell>("cell/changed");
const onCellRequested = defineHormone<Cell>("cell/request");

Next we create the cell. It should display the value or formula if focused, otherwise the calculated value. Also, it releases the cellChanged hormone when changed, and uses a receptor to receive cellSet hormones.

pureLit(
  "cell-element",
  (element: LitElementWithProps<Cell>) => {
    const { row, column } = element;
    // the value stored in the cell. Can be either a formula (start with =) or a value
    const { getState: getValue, publish: setValue } = useState(element, "");
    // is the cell currently focused or not?
    const { getState: isFocused, publish: setFocused } = useState(element, false);
    // holds the referenced cells if this a formula, so we don't have to parse the
    //  value over and over.
    const references = useState<Cell[]>(element, []);

    // decide what should be shown if the cell is not focused or a value
    const displayValue = () => isFormula(getValue())
        ? calculatedField(getValue(), references.getState())
        : getValue();

    const cellChanged = () => releaseHormone(onCellChanged, {
      value: displayValue(),
      row,
      column,
    });

    // if the field was set externally (ie loading a file)
    //  notify everybody about the changes by using a receptor
    //  to the isSet function
    useReceptor(element, onCellSet,
      cell => equal(cell, element),
      async cell => setValue(cell.value) && cellChanged()
    )

    // if this cell was requested, notify any subscribers
    //  on the value of this field by releasing the
    //  onCellChanged hormone
    useReceptor(element, onCellRequested,
      cell => equal(cell, element),
      async () => cellChanged()
    )

    // this receptor adds receptors for all the other
    //  fields. If one of the other fields changes, it will receive
    //  the value and update
    useReceptor(element, onCellChanged,
      cell => references.some(ref => equal(ref, cell)),
      async cell => references.publish(
        ...references.getState().filter(ref => !equal(ref, cell)),
        cell
      ) && cellChanged()
    )

    return html`<input
      type="text"
      class="${isFormula(getValue()) ? "formula" : ""}"
      @focus=${() => setFocused(true)}
      @blur=${() => setFocused(false)}
      @change=${async (e: InputEvent) => {
        const value = inputValue(e);
        if (isFormula(value)) {
          // if it is a formula, request all the other
          //  fields to collect their values.
          const fields = parseField(value).filter(
            (field: any) => !isString(field)
          );
          for (const field of fields) {
            await releaseHormone(onCellRequested, field);
          }
        }
        cellChanged()
      }}
      .value=${!isFocused() ? displayValue() : getValue()}
    />`;
  }
);

As you can see, receptors can take in filters. That makes it easy to ensure we only receive receptors relevant for us. The onCellChanged for instance looks only at fields:

Only if the current field is holding a formula, and has a reference to the cell changed, the receptor is triggered when the hormone is released, and the cell is updated with the new value and re-rendered.

That’s not too much code for a cell, let’s add the app that holds a few thousand of them:

export default pureLit(
  "spreadsheet-app",
  () => {
    const columns = "ABCDEFGHIJKLMNOPQRSTUVXYZ";
    const rows = 99;
    return html`
      <table>
        <tr>
          <th>&nbsp;</th>
          ${[...columns].map((column) => html`<th>${column}</th>`)}
        </tr>
        ${[...new Array(rows)].map(
          (_, row) =>
            html`<tr>
              <th>${row}</th>
              ${[...columns].map(
                (column) =>
                  html`<td>
                    <cell-element .row=${row} .column="${column}"></cell-element>
                  </td>`
              )}
            </tr>`
        )}
      </table>
    `;
  });

Okay. While all of that’s neat, we kinda lost our main promise already. In the cells, our business logic is tightly coupled to the logic. To test that a value can be set, we have to set the cell.

In addition, it does not work. When we add 1 into field A1 and =A1+A1 in the field B1, there is no result. Reason is that the field A1 has released it’s hormone before B1 knew it would be important and added a receptor. We have to add another hormone to request the hormone, and add request the value whenever the text for the current cell changes.

+ const cellRequest = defineHormone<Cell>("cell/request");

+   // Also release the value if our receptor to receive the current
+   //   cell is triggered
   if (
      isSet(element) ||
      referencesWithChanges(element, getValue, references) ||
+      isValueRequested(element)
    ) {
      releaseHormone(cellChanged, {
        value: displayValue(),
        row,
        column,
      });
    }

    return html`<input
      type="text"
      class="${isFormula(getValue()) ? "formula" : ""}"
      @focus=${() => setFocused(true)}
      @blur=${() => setFocused(false)}
+      @change=${async (e: InputEvent) => {
+        const value = inputValue(e);
+        if (isFormula(value)) {
+          const fields = parseField(value).filter(
+            (field: any) => !isString(field)
+          );
+          for (const field of fields) {
+            await releaseHormone(cellRequest, field);
+          }
+          releaseHormone(cellChanged, {
+            value: displayValue(),
+            row,
+            column,
+          });
+        }
+      }}

So whenever a specific field with a formula changes, it requests the values for the fields it references.

But the below the hood issues start to pile up.

We added a receptor if the value is requested. This receptor will only trigger a re-render for this component if and the hormone is released. However, the hormone also holds state. The state of the cell requested. The last cell will remain there, and whenever this cell is called it will assume it was requested and trigger a change.

The same goes for the references with changes. The last reference that triggered a change will stay as body in the hormone, whenever one of the other components that have a dependency get the cellChanged hormone, it will be triggered.

This is an unexpected behavior for our current usage of the hormones, can lead to unexpected releases of hormones, and thus lead to unwanted bugs and issues.

The hormones don’t work as they are, we will have to extend them.

One time hormones

We identified that the way we have set up the hormones is not sufficient. It works well enough for scenarios like todo lists and search, where we expect the state to be returned every time we call the receptor. As soon as we use it as something that is closer to an event system it fails. To make this possible we add a functionality that returns a value from the receptor only the first time it is read.

- const cellChanged = defineHormone<Cell>("cell/changed");
+ const cellChanged = defineSingleHormone<Cell>("cell/changed");

The change is subtle, but kinda intuitive. Let’s take another look at our receptor for referencesWithChanges.

const result = useReceptor(
    element,
    cellChanged,
    ({ row, column }) =>
      isFormula(getValue()) && hasReference(getValue(), { row, column })
  );

This function is called every time the cell is re-rendered. Let’s assume the cell we are interested in has the formula =A0. When field B0 changes, the useReceptor returns undefined. When A0 changes, it returns the value for A0. Before our change however, if our current cell is re-rendered (ie because an isSet is called), the receptor returns the value of A0 again, as if it would have changed. After the change, it will return undefined as the value has vanished, just as it would if the field wasn’t changed.

On the other hand, the behavior of the receptors to return undefined is not really that nice. It matches the organism parable, as when there’s no hormone released, then the receptor is not activated. On the other hand, from a code perspective that feels weird. Having a function that may or may not return something is not something that helps you understand the system,

At most once delivery

Cross-Framework communication