When the Yelp Android App was first built it was a monolith, with almost everything inside a single module. Single modules are easier to deal with; there are no complex dependency graphs to account for and some details of how the app is managed, such as how testing is implemented, are simpler. Eventually the app and company structure grew to a size where the monolith made code changes and testing harder, where modularization would help the app scale. After some discussion we decided to break the app into modules, thinking carefully about how we structured the app. We’ll discuss our experiences and the lessons we learned modularizing our applications and how they can help inform similar decisions for other Android applications.

Why modularize?

Modularization helps the app scale - builds are faster, tests are faster, and it’s easier to understand how the app works. If the app is modularized in a logical way, then individual modules will be the responsibility of a particular team, and when working on a feature, only a few modules would be affected, maybe even only one. This impacts how effectively developers can work on the app. For example, developers no longer have to build the entire app when they make a change. We’ve recently introduced other optimizations, such as only running tests for a specific module (discussed more later). Furthermore, it helps conceptually group related code in the app, making the app easier to understand. In essence, instead of needing to deal with problems at the scale of the entire app, you can now spend most of your time only considering a small amount of the codebase, which will become increasingly valuable as the app scales.

We’ve seen some pretty substantial performance benefits from modularization. Currently, a typical build from a minor change in one module takes about 12 seconds to build. A similar change in a previous version of the app which was partially modularized - no feature modules - took 47 seconds to build. While these numbers are relatively small, we have well over a thousand builds a day and those time savings add up.

We’ve also seen performance benefits for running tests. The time savings are not as dramatic in relative terms, because our tests run in parallel so test times are determined by the longest-running test. In general, though, running tests selectively will tend to decrease the longest running test since it’s likely that a small subset of tests won’t happen to have the longest running test in the larger subset.

We have a job that, among other things, runs all tests, that developers frequently run when working on a feature, and which runs when code is merged in. Previously, this would take about 16-20 minutes to run. Now, a test in just one module takes about 13 minutes to run.

App structure

The main consideration when dividing code into modules is how to structure dependencies. Module dependencies must form a directed acyclic graph; there can be no cycles in the dependency tree. Apps as they are written do not necessarily naturally fall into this structure. Thus, when modularizing, it’s a question of how to untangle code into modules with the desired dependency structure, while managing to divide code into conceptually related chunks.

There are two approaches that we explored for dividing up the app.

“Horizontal” modules

The first is to divide the app into horizontal modules. Broad categories of functionality are placed into an individual module, then stacked on top of each other in dependency relationships. You have all the UI elements near the top, and utility functions near the bottom, with the dependency relationship going from the top to the bottom. In the middle you have modules such as the data repository (a module through which the network and the cache are accessed), networking and analytics, and models. The precise modules you have would depend on how your app is designed - in practice, our app has tens of modules, multiple modules at the same layer, and not all modules are dependent on modules at lower layers.

Feature modules

The second is to have vertical feature modules. In these modules, for a given feature, the UI code, data repository, models, and everything else, are all in one module. An example of a feature module we created is one for onboarding. A major advantage of this approach is that developers will often work on specific features, and having a corresponding module means that the code they touch is restricted to one (or a few) modules. It’s also possible that a developer will focus on a horizontal module — revamping how analytics are done, for instance – but this is less common at Yelp and likely at most companies. Furthermore, modules are more independent of each other: modules affect any module upstream, so having many independent modules at the same level with no connections between them gives more of the benefits of isolation between modules. It should also be possible to have feature modules be at the top (or almost at the top) of the module hierarchy, meaning that many changes will only affect one module.

The big advantage of the horizontal modules is that well-structured code falls into these horizontal slices with clear layering one above the other: for instance, UI code depends on network code, but network code probably shouldn’t depend on UI code. We thus started by dividing the app into horizontal modules, untangling our code base and separating out related functionality. This puts us in a good position to make feature modules later, and in the meantime, there are still savings in build speed if we can separate out less commonly modified parts of the code into their own modules. There are also some cases where we will probably keep code in the horizontal modules. For instance, it will probably be easier to keep the Data Repository code in its own module. It includes some relatively expensive annotation processing we want to restrict to one module, and with the way we manage caches, it’s convenient to have them be in the same module.

Circular dependencies

If an app hasn’t been substantially modularized yet, there are likely circular dependencies that will need to be addressed. A circular dependency is where class A references class B, and class B references class A, and it makes sense to put the classes in different modules. There can also be more complex circular dependencies (e.g. A->B->C->A). In this case, there is a problem: dependencies between modules must form a directed acyclic graph.

Ideally, these dependencies could be broken. Perhaps some classes can be moved around so that they’re in the same module. Perhaps it makes sense to break apart a class, or pass a different value into a method.

Ultimately, though, there will be circular dependencies that can’t be removed. One way to resolve this is to make these indirect dependencies using an interface (as shown in the figure below). We found a few common cases where this would happen in many parts of the app, and created a Base module to address them. This module sits at the base of the dependency tree. The class that is higher in the dependency tree then is made to implement an interface that sits in the Base module. Classes can then refer to that interface, which at runtime resolves to the higher class.

You don’t want every single class to have a corresponding interface, and we generally try and keep these interfaces to a minimum, finding ways to refactor to avoid needing to make these references where possible.

What should a feature module look like?

When modularizing the app, one team (Core Android) took the lead in the initial modularization work, in particular setting up the horizontal modules and some example feature modules. The goal is for feature teams (e.g. the team focused on Search) to be able to build on this groundwork and make the modules that make sense for them. A good feature module should be whatever structure the feature teams find to be beneficial.

We don’t want to set strict guidelines on the size of modules. Modules will hold features or logical pieces of features, and these are likely to vary greatly in size (although a one class module is probably too small). However, we want to have modules be the smallest self-contained unit of functionality that a person is likely to work on at a given time. That way, a developer sees the highest benefit: they see the smallest possible build time and test time for the amount of code they’re working on.

Another consideration is that you want to avoid having dependencies between your feature modules. Dependencies mean that if you change one feature module, other modules have to be built or tested. While dependencies can be eliminated using the base class trick above, it’s something you want to keep to a minimum - it somewhat obscures how the code works and adds an invisible dependency that might come back to haunt you later (for instance, if you run tests based on what modules appear to have changed).

Where should you start with feature modules? A challenge to keep in mind when modularizing is that moving code to a module that’s being very actively developed results in many conflicts when you merge your modularized branch back in. Of course, putting the most important code in modules gives you the greatest benefits, so it has to be done at some point. However, when you’re creating your first feature modules, you may need to experiment with different approaches and it’s best to do so with a less frequently used part of the codebase. On the other hand, make sure your first modules touch most of the important systems in your app.

When deciding what to put in a feature module and what to keep in a horizontal module, it makes sense to think about what’s compiled and modified often. It might make sense to leave code that could fit in a feature module in a horizontal module if it’s almost never changed. One example of this are our strings. Furthermore, we have specialized tools for handling them (for internationalization) and keeping them all in one module made the most sense with the way our tools work.

There’s also the question of how much to modularize. When pulling code out into feature modules, there were a lot of places where the horizontal modules could be broken down further to make the separation of code in the app more elegant. In particular, our UI module had a lot of utility code that could probably have been pulled out into different modules. In the end, though, it didn’t make sense to do so: aside from being more elegant, the actual benefits were fairly small. When modularizing, it’s important to have a goal for what you want to achieve. Faster build times and test times are a good goal, as is empowering feature teams to pull their code out into modules. There are advantages to creating very fine-grained modules, but modularization is a time-consuming process, and there’s a tradeoff between that and the benefits of modularization.

What next?

After seeing the initial benefits of modularization, there is a lot of enthusiasm among feature developers to modularize their features. Older parts of the code will probably take longer to be modularized, but new features will most likely be built in new modules. By setting the groundwork for modules to be created, we make it possible for any Android developer to easily modularize their code.

One question we’re exploring is whether we can use modules to limit the number of tests we run. Modules structure your app in a way that you know what code depends on what other code. In doing so, you can greatly reduce the number of tests you need to run: you can run only the tests that are actually affected by a change. If you’ve made a change to how Reviews are displayed, for instance, it’s unlikely you need to run tests on the ordering platform (if the ordering platform doesn’t use Reviews). If they are both in modules, it’s easy to determine, by looking at the module dependency graph, that these two features are unrelated. Currently, we run tests on every merge, and developers frequently run the full set of tests as they work on a feature.

This is a big win if you do much UI testing. UI testing can be expensive, in terms of both time and resources, and if developers have to wait twenty minutes for all the UI tests to run, they get frustrated. As the app gets bigger, this becomes more and more of a problem. Modularization can allow rigorous testing to scale as your application grows.

One concern, though, is the indirect dependencies above. At runtime, modules can depend on code from other modules in a way that isn’t reflected in the dependency graph. In our particular app, we looked through these dependencies to understand the potential impact of running tests selectively, and concluded that, with the way the app is structured and the types of indirect dependencies we have, it’s unlikely to be a problem. Fortunately, we have a limited number of base interfaces, making it possible to reason about the impact of these indirect dependencies. Furthermore, we run the full set of tests on the master branch after every commit (in addition to before every merge), so we expect to catch any false negatives (see our previous blog post here about how we run tests). However, as we deploy the selective running of tests, we’ll carefully monitor to see if these assumptions are correct.

Why not modularize?

Modularization has a lot of advantages, but it comes at a cost and may not be for everyone. In particular, it’s not a magical fix for all your problems. For instance, reduced build times were a major motivator for modularizing our app. As described above, we did see some pretty substantial benefits, but after profiling our build we found other approaches that had an even more dramatic effect (for instance, our virus scanner made file i/o slow). Don’t just assume that because modularization helped someone who wrote a blog, that it suits your use case!

We found modularizing took a substantial amount of engineering effort. Resolving circular dependencies is time-consuming, and changes that affect large portions of the app run into problems with extensive merge conflicts. When the largest module was broken out of our app, the engineer involved merged in his code on a weekend. Otherwise it would have been impossible to deal with the merge conflicts. Completely modularizing the app could easily take months. Furthermore, our testing, merging and release infrastructure was based around code being a single module and there was large overhead in getting it to work with multiple modules, particularly our testing infrastructure.

There are also corner cases you have to deal with, especially around tests. We had a lot of code shared between tests, but code can’t depend on code in the test folder of another module. This made splitting test code into modules more difficult. Furthermore, many of the tools (particularly those related to tests) assumed that the main code was in one module, and so they had to be updated.

There is probably a point at which the app grows too big to be one module. The bigger the app, the more important that it be modularized. On the other hand, the bigger the app and the more developers, the larger the engineering effort to modularize (due to the number of circular dependencies to resolve and the number of merge conflicts respectively). There’s no easy answer as to the optimum point in time to modularize the app.

If the app isn’t growing much, though, and you’re fine without modules as it is, it’s probably best to leave it as it is. Some apps may never reach the point where modules make sense, and you want to avoid prematurely wasting effort that could better be spent elsewhere.

Whether to start out with modules when creating a new app is an interesting question. Since our experience was to first create a monolith and then break it up, it’s hard to authoritatively claim that starting with modules is a good idea. The argument has been made that usually apps are more successful when you start with a monolith - there is an interesting discussion of the topic here.

Overall, there isn’t an easy answer to that question. The goal of this blog post isn’t to convince people that modularization is necessarily the right decision, but to bring forth the considerations when deciding whether and how to modularize.

Become an Android Infrastructure Engineer at Yelp

Interested in these kinds of problems? The Core Android team is hiring!