Frontend software design research

Current situation

Our frontend code is overly complicated. We have built this over time without writing unit tests much. It is not easy to debug, and not easy to add features.

We have been trying to solve this problem. Here's a couple of attempts:

Try using a modern library (Vue.js)
Decaffeinating

However, they have not been working really well.

Vue.js
- It was an ok approach, however, we did not educate developers properly for using this. Only a few developers are familiar with it.
- We haven't had any change that requires brand-new UI components. The library works well for making a complicated UI, but doesn't help us making existing complicated codebase easier.
- We didn't know what features we needed to cover. Because there's no tests, renewing a product was impossible.
Decaffeinating
- CoffeeScript is not supported by good ecosystem (IDE, test coverage).
- We have a tool that decaffeinates, however, it's not safe using it because we don't have much unit tests.

I got feedback in late 2020 about decaffeinating: "Other teams don't care whether we're using CoffeeScript." Developers care about this, but Sales doesn't sell our product because it's written in better language such as JavaScript/TypeScript.

Deep dive - decaffeinating

The feedback is 100% true. I consulted with other developer about this. The conversation went like this:

How long would decaffeinating take? -> It takes long, because we don't have unit tests. -> Why? -> Our components are not really testable. -> Does JavaScript/TypeScript help the problem? -> No. -> "Code is not testable" a strong smell! We should refactor for making them testable.

The feedback and this conversation made me think about the original motivation of Decaffeinating - Improve our code quality for making reliable products. I started researching how to refactor, what is good software design, what is good code pattern.

The 'untestable' components

We're manipulating DOM objects in so many places. Testing DOM objects in unit test is not easy because the unit test is not running in a browser. If you're using jQuery, you're likely manipulating DOMs really freely - that's a yellow light for writing unit tests.

We do have tests against DOM, however, testing with jsdom is limited as jQuery code can try to access to an element out of jsdom scope.

Refactoring

To learn about refactoring, I started reading Martin Fowler's "Refactoring".

With this book, I learned "Agile" is formed by the three factors:

CI
Self-testing code
Refactoring

We have our own CI system that's working great. But we're lacking the other two.

Refactoring book's "The First Step in Refactoring" says that we need solid unit tests. ... We don't have tests :(

To refactor, we need tests. To write tests, we need testable modules. To write testable modules, we need to refactor.

The immediate goal for us would be making testable modules without breakage by refactoring.

I started making this type of refactoring. Here's some examples:

When should we refactor

The section When Should We Refactor of the book gives you a good idea when to refactor.

Here's some questions I ask myself when I refactor:

Do I understand? Is this testable? Can I understand easily? How can I make this part testable?

Also, please read When should I not reafactor. Refactoring is great, but we don't always have time / budget. I recommend you to set the scope when you refactor.

Frontend software design

When you refactor, you'd like to make small module files instead of having a big file. What is a good file structure?

This question led me to study about Domain Driven Design (DDD), Clean Architecture, Functional Programming and React.js application structure.

DDD & Clean Architecture

What is DDD? Clean Architecture? They are architecture pattern designed for backend. Although, I believe frontend can take some principles.

Quoted from Comparison of Domain-Driven Design and Clean Architecture Concepts - Khalil Stemmler:

Domain-Driven Design, initially written in 2003 by Eric Evans, introduced new approaches towards designing software by using a layered architecture with a rich domain model in the center.

Uncle Bob wrote Clean Architecture in 2017 and summarized his research on what constitutes a clean architecture, also using a layered architecture with a domain layer in the center.

DDD

I learned an interesting concept: "Value Object" - ValueObject by Martin Fowler

I have a simple example of Value Object here. I think roles in our Perl land is similar (e.g., hasLocation). The object encapsulates domain knowledge in one file.

Value object
Entity: Data model.
Domain Service: Service that manages an entity.
Repository: When you change data, the change is not reflected to data itself but this.
Application Service: Service that forms application. Use case.

Clean Architecture

Let's take a look at the clean architecture diagram.

The image is from this webpage written by Uncle Bob.

It explains about how it splits software by layers, and each layer is depending on inside of its layer.

Web UI (blue circle) presents data.
The presenter creates web UI that shows data for the use case (green circle).
Use case knows how to present data (red circle). In DDD, it is split into Domain Service and Application Service.
Entity is the data we want to present (yellow circle).

Each layer is depending on its inside. Never depend on layers outside.

For better understanding, please read the original post.

Applying principles to frontend

It seems like we can split frontend codebase into:

Entity (Models. We do have data models (data transfer object, DTO) such as Page, User, etc.)
Use cases (Business logic - the part we create the application by using entities. e.g., authentication logic. Value Objects live here.)
Presenters and Controllers (HTML rendering, event handler.)

This is just a hypothesis, I don't want to conclude here just yet :) Let's see if this is valid, with a tiny React.js app later.

Programming patterns

Command-Query Separation (CQS)

CQS: Every function/method should be either a "Command" that takes an action, or a "Query" that returns data.

I think this is the same level as refactoring. When you refactor, try creating a function that is either "Command" or "Query". Then write tests against each function. There may be some that you can't test, such as using Web APIs, but mostly you should be able to.

We learned how the clean architecture is organizing layers. Let's take a step back and think what frontend application is.

I think frontend application can be simplified as the following flow:

Execute "Commands" to invoke data fetches.
Execute "Queries" to fetch data through web API.
Execute "Commands" to display UI based on the data we want to show.
Execute "Commands" to attach event handlers that creates "Commands" in the UI.
"Command" is invoked and execute "Commands" to change UI, "Queries" to fetch more data, and execute "Commands" to update data via Web API.

Fetch data, read fetched data, render UI, create event handlers, create forms to POST/PUT. One web application has hundreds of this flow. It should be this much simple.

Functional Programming

Functional programming (often abbreviated FP) is the process of building software by composing pure functions, avoiding shared state, mutable data, and side-effects.

From Master the JavaScript Interview: What is Functional Programming?

What are those phrases? Here's a quick summary.

Pure functions: Given the same inputs, always returns the same output, and no side-effects. Which is that you can reuse, run multiple time, and returns the same result for the input all the time.
Shared state: Anything shared between scopes. Global objects, values in cookies, for example.
Mutable: An immutable object is that it can't be modified after created.
Side-effect: Console.log, rendering HTML, writing to the network, changing property of other object.

React.js application and the principles

React.js - the most popular modern library is using functional programming principles. It let developers build the "Fetch data, read fetched data, render UI, create event handlers, create forms to POST/PUT." flow in functional programming way.

The following pseudo code snippets are for rendering a user list. Each list item has an edit form to edit their name. When a user name is updated,UserList and UserEditForm get updated automatically.

(I'm not advocating we want to use React.js.)

// UserList.jsx
import { useContext } from 'react';
import { getUserByArea } from '../User/UserService.js'
import UserListItem from './UserListItem.jsx'
export default async function UserList(props) {
  const area = useContext(AreaContext); // context is a read only variable defined in a higher level component.
  const users = await getUserByArea(area);
  return (
    <ul>
      {users.map((user) => <UserListItem user={user}/>)}
    <ul>
  );
}

// UserListItem.jsx
import UserEditFormContainer from './UserEditFormContainer.js`
export default function UserListItem(prop) {
  const {user} = props
  return (<li>{user.name()} <UserEditFormContainer user={user}/></li>)
}

// UserEditFormContainer.js
import UserEditForm from './UserEditForm.jsx'
import { updateUser, createUserName } from '../User/UserService.js'
export default function UserEditFormContainer(props) {
  return UserEditForm(props, updateUser, createUserName)
}

// UserEditForm.jsx
export default function UserEditForm(props, updateUser, createUserName) {
  const { user } = props.user
  const [userName, setUserName] = useState(user.name()) // state machine. `useState` takes an initial value, and it returns a state variable and a function that renews new state. It then refresh this function and assign new `userName`.
  const handleSubmit = (e) => {
    e.preventDefault();
    try {
      const userNameValueObject = createUserName(userName);
    } catch () {
      alert('Invalid user name!');
    }
    try {
      updateUser(user, userNameValueObject);
    } catch () {
      alert('Failed updating user!')
    }
  }
  const handleChange = (e) => setUserName(e.target.value);

  return (
    <form onSubmit={handleSubmit}>
      <label>Name</label>
      <input type="text" value={userName} onChange={handleChange} />
      <button type="submit">Update</button>
    </form>
  )
}

// UserService.js

// Repository. In true clean architecture, Service (Red circle) SHOULD NOT depend on repository (Green circle). But we can't DI in JavaScript. Wish we used TypeScript.
import UserRepository from '../UserRepository.js'

// Value object: This case, this value object knows the rule of "What is user name, what is NOT user name"
class UserName {
  constructor(name) {
    if (typeof name !== 'string') { throw new Error('It has to be a string!') }
    if (name.length <= 1) { throw new Error('It has to be more than 1 letter!') }
    if (name.indexOf(' ') > -1) { throw new Error('No space is allowed!') }
    this.name = name;
    // Value Object should be an immutable object.
    Object.freeze(this);
  }
  value() {
    return this.name;
  }
}

// Factory function
export function createUserName(name) {
  return new UserName(name)
}

export function getUsersByArea(area) {
  // Possibly HTTP GET request
  return UserRepository.getUserByArea(area)
}

export function updateUser(user, userName) {
  if (userName instanceOf UserName) {
    // Possibly HTTP PUT request
    // Possibly, repository is doing something like this before the HTTP request
    // const newUser = {...user.json(), name: userName.value() }
    return UserRepository.putUser(user, userName)
  } else {
    throw new Error('User name has to be a UserName instance.')
  }
}

Let's assume UserService is a service for User entity.

React.js app and the clean architecture diagram

Let's map modules into the clean architecture diagram.

User is a data object. It is Entity in frontend. (Yellow circle)
UserService is a domain service, which is part of Use case. (Red circle)
UI Components: The event handlers are controller. (Green circle) + UI (Blue circle)

UI components are depending on UserService. UserService is doing it on User. We can also see the dependency hierarchy here.

It looks like the application design of React.js is aligning to the diagram.

React.js app and CQS

getUsersOfArea is a query, updateUser is a command. Event handlers are all commands. Frontend can adopt CQS.

Previously, I simplified a web app with the "Fetch data, read fetched data, render UI, create event handlers, create forms to POST/PUT" flow. I can see flow in the React.js snippets.

React.js app and Functional Programing

Pure functions: A UI component has this criteria: "Given the same inputs, always returns the same output", however, there's a side effect (in UserService).
Shared state: Within the scope (this case, a use case), there's no shared state (which is good).
Mutable: Nothing is re-assigned. You may want to argue userName gets updated, but it's actually not. Read: Introducing Hooks
Side-effect: No side effects within a UI component. (warning: alert is a side-effect)

React components are following the functional programming manner. When updateUser, the user object stays as a read-only object - immutable.

Side-effects in web application

We have to PUT via a HTTP request, we have to render jsx to virtual DOM, we would need to read a cookie value... They are all side effects. The best practice is to place them in one place. This way, we know what's making side-effect. It makes components that have no side-effects really easy to unit-test.

Let me list a few examples:

Rendering
HTTP Requests
Read/Write Cookies

1. Rendering

DOM manipulation has to be done in UI components. If we use do in Domain Service or Domain Model, we're doing it wrong.

For manipulating DOM, we use jQuery. We have some other place that we're using jQuery, however, we really shouldn't. Stuff like $.map(), $.grep() are for supporting really old IEs, we should use standard JS instead.

There's an only exception I can think of - Promise.all isn't supported by IE. We're using jQuery to make an equivalant logic.

2. HTTP Requests

HTTP requests should be in one place for the package:

// UserRepository.js
import Request from 'some-http-request-library'  // Only Repository is importing this.

export class UserRepository {
  constructor(context) {
    this.context;
  }
  getUsersByArea(areaId = null) {
    return Request.get(['area', areaId || this.context.areaId, 'users'].join('/'))
  }
  getUserById(userId) {
    return Request.get(['area', this.context.areaId, 'user', userId].join('/'))
  }
  putUser(userId, data) {
    return Request.put(['area', this.context.areaId, 'user', userId].join('/'), {data})
  }
}

3. Read/Write Cookies

Create a component and only that object should access to cookie. Then let Domain Service / Application Service use it. That way, you can mock it easily when you test. Same for reading the navigation bar (URL), sessionStorage, or localStorage.

How we could adopt the principles

We went through the following topics with React.js app.

DDD, Clean Architecture
CQS
Functional Programming

Our application's core framework is jQuery + template-toolkit. You can do ANYTHING with jQuery, but it means you can violate a boundary between components easily, you can build a feature that "works" with the worst practices.

The reason why our components are complicated is not because of jQuery. The developers who use jQuery to build an application really need to know about design patterns and good software architecture for building a large size application.

While with React.js, even though you don't know any of those high level knowledge, you can build a scalable app. Because it enforces developers to write in a certain way.

We learned the idea of what React.js enforces. Let's see if we can write it with jQuery.

First, we want to update a user name in a li element when the user is updated. We're using Observable pattern here.

// Subject.js
export class Subject {
  constructor() {
    this.observers = []
  }
  addObserver(observer) {
    this.observers.push(observer)
  }
  notify(value) {
    this.observers.forEach(function (observer) {
      observer(value)
    })
  }
}

Then making the user list component. One component should only render one component. Child component takes a parent element, and append itself to the parent whenever the child component wants.

// UserList.js
import UserListItem from './UserListItem.js'
import { getUserByArea } from '../User/UserService.js'
export default async function UserList(props) {
  const $elm = $('<ul></ul>')

  const area = SomePageStore.getArea()
  const users = await getUserByArea(area)

  // This function is called once in this example. Just making it for consistency with other UI components.
  function render(users) {
    $elm.empty()
    users.forEach(function (user) {
      $elm.append(UserListItem({ user }))
    })
  }

  // Initial rendering - defining `ul`.
  render(users)
  return $elm
}

List item component wants to know user state when it's children update a user. Let's define userState which is a simple observable subject. When a user gets updated, this component wants to re-render. That's what render function is for.

// UserListItem.js
import { Subject } from './Subject.js'
import UserEditFormContainer from './UserEditFormContainer.js'
export default function UserListItem(props, userState) {
  const $elm = $('<li></li>')

  const { user } = prop
  // This component wants to know 'user state' when a user gets updated.
  const userState = new Subject()

  // We want to re-render when a user gets updated.
  function render(user) {
    /**
     * [% user.name() %] <span class='editor'></span>
     */
    const $content = $(Template('user-list-item-content.tt2', { user }))
    // `render()` is called multiple time. empty out the `$elm` is a must!
    $elm.empty().append($content)
    $elm.find('.editor').append(UserEditFormContainer({ user }, userState))
  }

  // Initial rendering - defining `li`.
  render(user)

  userState.addObserver(function (user) {
    // This is called when `userState.notify()` is called.
    console.log('User has been updated! The list item is going to be updated.')
    render(user)
  })

  return $elm
}

The container is for making UserEditForm dependent. When you want to test UserEditForm, you can mock parameters such as updateUser and createUserName.

// UserEditFormContainer.js
import UserEditForm from './UserEditForm.js'
import { updateUser, createUserName } from '../User/UserService.js'
export default function UserEditFormContainer(props, userState) {
  const { user } = props
  return UserEditForm({ user }, updateUser, createUserName, userState)
}

We finally use userState in this component. When user is updated successfully, we notify it to the component that defined userState, in this case, UserListItem. Then UserListItem runs render() to refresh the UI.

// UserEditForm.js
export default function UserEditForm(props, updateUser, createUserName, userState) {
  const $elm = $('<form></form>')

  const {user} = props;
  // In React.js, this is immutable. It's complicated to do it without React.js.
  let mutable = { userName: user.name() }
  const handleSubmit = (e) => {
    e.preventDefault();
    try {
      const userNameValueObject = createUserName(mutable.userName);
    } catch () {
      alert('Invalid user name!');
    }
    try {
      updateUser(user, userNameValueObject).then(function (newUser) { userState.notify(newUser) });
    } catch () {
      alert('Failed updating user!')
    }
  }
  const handleChange = (e) => mutable.userName = e.target.value;

  function render(user) {
    /**
    * <label>Name</label>
    * <input type="text" value="[% user.name() %]"/>
    * <button type="submit">Update</button>
    */
    const $content = $(Template('edit-form-content.tt2', { user }));
    $elm.empty().append($content)

    // Attach event handlers
    $elm.find('button[type="submit"]').on('click', handleSubmit);
    $elm.find('input').on('change', handleChange);
  }

  // Initial rendering - defining `$li`.
  render(user)

  return $elm
}

UI components are just focus on rendering. Other functionalities such as updateUser are separated from UI.

Side note: Template that we're using is not giving us much value with this structure. Because we want to keep reference of the top level element (e.g., $('<form></form>') in UserEditForm.js).

Structure inspired by DDD & Clean Architecture

In my mind, this file structure below is making sense. Making a package creates context boundaries. For instance, other packages should not touch models of a package directly, but services.

- users (package name)
  - components (UI components. Green circle and blue circle)
    - UserList.js
    - UserListItem.js
    - UserEditFormContainer.js
    - UserEditForm.js
  - services (Red circle)
    - UserModelService.js (Functions of HTTP request for user entity, treat multiple user objects such as "aggregating users".)
    - UserApplicationService.js (Use cases. Depending on cookies or other entities, and glue them together with user entity.)
  - models (Yellow circle)
    - User.js (One single user data model.)
    - UserName.js (A value object. Rules related to "user name" is written in this.)

CQS

With this jQuery app example, you can see most of functions are doing one thing: "Command".

Functional Programing

Pure functions: Unlike React, a UI component is making side-effect - updating an element. But we are able to keep props (user) immutable.
Shared state: Same as the React.js app example.
Mutable: We still have mutable objects. But they are within a scope.
Side-effect: Same as the React.js app example.

We should encourage ourselves to freeze objects built based on data from backend. With this example, user is the one. Making code immutable reduces confusion a lot.

Conclusion

Open Closed Principle - one of SOLID principles is pretty much "Developing software that is robust and flexible to add feature". And this is what I want to achieve. To Achieve this, we should refactor. To refactor, we should have tests. To build testable components, we want to know our direction and best practice. I hope this practice leads to better developer experience.

The Clean Architecture does not give you strict rules how to organize your applications, it only provides you with recommendations.

From The Clean Architecture using React and TypeScript. Part 1: Basics

Why not React.js ?

You may realized that we can build a jQuery app with React.js style. - Then why not just writing it in React.js? If I were to oppose...

Not all developers like to learn React.js from beginning. I felt this after we started using Vue.js.
React.js does not support jQuery UI, and it really shouldn't. If we can replace all jQuery UIs with React.js components, that's a different story.
Need to maintain build system.
Importing React.js into our app makes the bundle size even bigger, while we've already added Vue.js components.
A single React.js component has own CSS scope. While our classic app has global CSS scope. I don't know how they get along. Maybe not a concern.
When we are ready to replace this style with React.js in the future, it's gonna be easy anyways.

Of course there's some trade-offs with not using React.js.

You need to know high level (software architecture, design pattern).
You need to pay attention what variable is immutable, and with exception, what variable is mutable.
You need to understand observer pattern. It's powerful only if you use appropriately.

"if all you have is a hammer, everything looks like a nail."

References

Resources I studied with.

DDD

Clean architecture

The Clean Architecture by Robert C. Martin (Uncle Bob)
Japanese: Clean Architecture for React.js
Clean Architecture for frontend in Action
Youtube: Functional Programming and Domain Driven Design - a match in Heaven! - Marco Emrich - KanDDDinsky
The Clean Architecture using React and TypeScript. Part 1: Basics

wataruoguchi/FrontendSoftwareDesignResearch.md Secret