Case Study in Modular Design

We have covered quite a bit so far on how our application is architected. If you missed out on following it, check out our other blog post in this series:

At the start of any software development project, it’s impossible to predict how it will grow. A sudden jump in the scale of your user base, a change in requirements, or even a new feature release from your cloud provider can have a huge impact.

Change will happen. That’s just a reality. The environment around your application might evolve in a way that invalidates a decision you made months earlier.

In this blog post, we’ll look at what it took to keep our application adaptable.

Form Field Configuration: ricocheting between an API and a module

If you remember from the first post, our application collects data from new employees around the world and what data was required was different from country to country.

So, how can we control that? It’s hard to know what the right decision should be. But we had one guiding principle: adding field requirements for new countries needed to be as easy as possible.

Easy for whom? An employee who is not strong technically? A junior developer? An outsourcing team? We didn’t know at the time, but sticking to a philosophy that adding new countries needs to be easy, whatever that meant, ultimately helped us stay flexible.

v1 Configuration File

In the beginning, it was just a JSON file containing all the data necessary to render fields on the form. You had a field name, title, and field type. What more do you need?

JSON is not a programming language

A lot, it turns out. Is the field required? How about a regex to validate data? Does that regex apply worldwide? The system of record for this data has a complicated interface that changes without warning, so the complexity of parsing out data was pushed into the config file. Sometimes the way you get data from the system of record is different from the way you save it back.

It didn’t take long to realize this was not a workable solution…

v2 Node Module

Having outgrown the might of JSON, we started moving towards turning this into a node module. We built a directory structure that looked something like this:

├── /src
│   ├── /models
│   └── /data
│   └── index.js

What was once the JSON configuration file was moved into the /data directory. Complex functionality, like the aforementioned data parsing, was moved to a /models directory.

This worked great! It was easy for a developer to add new fields, and there was a place for additional code to live for anything more complicated.

Uh oh…

We got a new requirement: there needs to be a UI for users to add or remove fields, and launch new countries. Our “flat file” database isn’t going to work anymore. This, coupled with other requirements, drove the decision to turn this application into an API.

v3 RESTful API

This turned out to be easy. We added Koa, and a /routes directory to expose the functionality that was written in /models. So now our directory structure looks like this:

├── /src
│   ├── /routes
│   ├── /models
│   └── /data
│   └── index.js

Once that was done, we moved the flat files into DynamoDB, deployed the whole thing into our ECS cluster, and called it good. It only took a day or so.

Man, you talk a lot

This had a devastating effect on performance. We had two APIs: this one and one that actually communicated with our people management software. This made them incredibly chatty and slow as a result.

Remind me again why you split this responsibility out. We knew we needed to make launching new countries and changing fields around easy. And while we weren’t sure who would take that on, our product owners were adamant it would not be the development team. This needed to be something encapsulated that could be handed off to someone else. Whoever that was might not have the legal right to have access to our people management software API. This drove the decision split this into its own project.

It was also becoming clear that having a UI that an unskilled person could use to manipulate the form was not realistic. Mostly because our people management software’s API was so inconsistent and complex, but also because the onboarding form itself was getting more and more complex. Having fields that are only required depending on how other fields are filled out, dynamically showing sections of the form…this would only be possible by inventing a simple programming language for people to use in our UI, which is a step down the road of The Inner Platform Effect.

Never mind the fact there were better incremental steps that could’ve been taken, such as keeping it as a node module, but putting the data in DynamoDB, with a UI in front of the database. Hind sight is 20/20.

v4 Back to a Node Module

We started the journey back to an node module by ripping out Koa, the /routes directory, and the data in DynamoDB. Now that we’re free from the constraint of building a UI, we could start to organize the project in a way that makes sense for a developer. And chances are, it’ll stay that way. The power user UI was cut by our product team.

Lessons learned

Focusing on the right priorities helped us to keep our application flexible so we were free to experiment. Ultimately, we landed on a paradigm that works best for us.

It’s easy for a technical lead to myopically focus on architectural patterns when what matters is the software is easily understood, maintained, and performant. Hypothetically, if we had decided in the beginning that because we’re aiming for a microservices environment everything needs to be a RESTful API, it would have been easy to create something architecturally pure, but severely broken in more significant ways.

Instead, we kept in mind the goal of keeping this functionality as easy to change as possible, whatever that meant.

By keeping our application code properly encapsulated (primarily through dependency injection and relying on AWS Managed Services), we were able to try new things. Each of our services are small enough to rebuild in under one sprint.

It’s impossible to predict how your application will need to change in the future, but by planning for change and making your code reusable, you’ll be able to avoid the technical debt plaguing teams as software projects drag on.