Past, present and future of BK2 - an Engineering Perspective

I thought I would share some thoughts about where we are, how we got here, and how we can face what is ahead of us. We have some big challenges ahead, and I thought it was a good idea to write about how we got here in the first place. And especially about what makes BK2 unique and how we can move forward.

The other day David (our Founder) asked me about what having a separate DB per customer would look like. And I mentioned that it would have many benefits, but also would make for a complex solution that involves dynamically managing DBs and connections. And that, in my opinion, the cons outweighed the benefits significantly.

And I mentioned that we worked hard on making BK2 simple and maintainable. So, I wanted to first write a little more about that:

How did we keep the code simple and Maintainable?

BK1 (Our old Rails application) was a big application that was difficult to maintain. We were spending more time fixing bugs and performing maintenance tasks than writing features and moving forward. And the bug list was soooooo long, and every time we fixed one bug there were already three new ones, like a hydra.

At some point it became clear to us that something had to change. And we decided to rewrite the application from scratch in a different language. But this time we would do it differently. We didn’t want to make the same mistakes we did in BK1.

We would make sure from the start that the application was always (now and in the long term) easy to maintain by a team of 3 engineers. It’s something we have actually written in the first lines of Brandkit’s README.md file:

BK2 - Brandkit rewrite in Elixir

Goal: To be easy to maintain for a small team.

We put a lot of effort to make the application as simple as we could in all areas. HTML, CSS, Javascript, Application code and Infrastructure.

How did we keep HTML and CSS simple?

In BK1 we were using Bootstrap. In practice, this made HTML difficult to read and write, and JS difficult to follow and to edit. And we were using an old version and every time we wanted to update it was a pain.

In BK2 we decided to not use any CSS framework, and instead use semantic HTML and write CSS from scratch. All our CSS code is in well-named directories and everything is pretty intuitive.

This made the whole HTML and CSS code much smaller and easier to maintain.

It’s easy to understand BK2 HTML code just by looking at it. Everything uses semantic tags and class names, there is very little “wrapping”, and very little nesting (tags inside tags inside tags) in comparison.

The same HTML page in BK2 has ~5 times less code than the BK1 equivalent.

And because of this we also send much less data to the client on every request. Which shows. We always get pretty decent results in Lighthouse.

How did we keep Javascript code simple?

In BK1 we were using Angular 1. And it was a pain. Code was difficult to follow and to change. It was an old version of Angular, and changing it would have been a huge task, and we never got into it.

In BK2 we decided to rely less on Javascript, and more on server-side. And for those pages that needed lots of interaction we used LiveView. This made the whole JS situation easier to work with.

In BK1 we were using YARN as a Node package manager, and we had the “Dependency hell” problem, which caused us a lot of headaches in the past.

In BK2 we don’t use node at all. If we need to use a third party, we explicitly load the code. All our JS is kept in simple modules with good names. We use esbuild, but just as a bundler: It puts all the JS in a single file with a random code at the end.

How did we keep database simple?

BK1 uses rails, which has the convention over configuration philosophy. Rails encourages you to use their “sane defaults” and not to think a lot about your database. You use an ORM, and trust it to do the hard work for you.

This sounds nice, but in practice if you are not careful, you end up with unoptimized queries, bad indexes and orphaned rows, which is exactly what happened to us. All the protections were in the Rails code, and it didn’t provide real data integrity in the database.

In BK2 we have a more data-centric approach. Elixir and Ecto makes it your responsibility to add good constraints and indexes to the database, because everything is explicit. Working with validations and constraints is super intuitive. And we pay much more attention to how we keep data organized in the database. We add specific rules for what should happen to relations when deleting a record in the database. This is great. We can trust the database for integrity.

In BK1 we used Redis for background jobs, and Elasticsearch for fast search. In BK2 we decided to rely 100% on Postgres for both things. This made the application even simpler and more maintainable. And still very fast.

How did we keep code simple?

In BK1 we had code everywhere. Sometimes in the models and sometimes in controllers. Sometimes it was not easy to change code because it was all over the place. It was difficult to change a line of code, because you didn’t know how it would impact other parts of the code.

In BK2 we separate the web application (brandkit_web) from the core of our application code (brandkit). And code is organized in contexts. This separation of concerns allows us to have simpler code in general.

In BK1 we didn’t have any automated test at all. If we ever made a change, we had to test all the cases manually and cross our fingers that we didn’t make any mistake.

In BK2 we have a very decent test coverage. We have 1400 tests! And they run in my computer in less than 4 seconds. This gives us a lot of confidence when we make changes.

Time well spent

But also, the most important thing we have done for keeping the code simple is refactor, refactor and refactor. We allow us to spend the time to remove that unused function, rewrite that overly nested HTML, simplify those CSS rules, rename that CSS class, clean up that large function. And also stop to think about what we are doing, and try to find better and simpler solutions all the time.

But sometimes we have to sacrifice simplicity for something else

Infrastructure

In BK1 we were using Heroku, and (while somewhat expensive) it was pretty simple.

In BK2 we decided not to use Heroku, because the Dynos architecture didn’t adapt well to Erlang processes and connections. We decided to use Fly. Because it was simple to use, and also it provided easy distribution across regions. In combination with the distributed nature of Erlang, this felt like a great decision.

This is one of the few cases where we decided to sacrifice simplicity for better features.

By using Fly we made the infrastructure a little more complex: Instead of a managed single Postgres instance, as we had in BK1, we had an unmanaged Postgres Cluster.

I think this was the right call. But also it was a source of some headaches because of connection errors that were difficult to understand.

Thumbnail generation and file management

The other case where we decided to sacrifice simplicity for an important feature was AWS Lambdas. We were having problems and challenges with creation of Thumbnails, and at some point it became obvious that we needed to start using Lambdas.

I was worried that this would make everything much more complex. But Fede (fellow Brandkit Engineer), did a great job in automating everything using CDK, and in the end we benefited a lot by Lambda and started using it in other places.

The new challenge

Now it looks like we have to make another difficult decision.

Data sovereignty has become more important than ever. And the best solution seems to be duplicating the application across regions.

This makes BK2 more complex.

I don’t like that, but it is what it is. We are in the process of understanding everything we will need to make this happen. It’s not just about duplicating applications (we already have duplicated applications. One is called “staging” and another one is called “production”.).

This is more than that. We have to not only duplicate (or multiply) the application across regions. We also have to create the infrastructure to support every new application. And it has to be an automated process, so that we can add new applications easily in the future.

And there is also a migration process, where we have to clone the database, and change a lot of DNS rules to point to the new locations. And put the application in “maintenance” mode (at least for the accounts that are going to be migrated). And once the migration is done, delete their data from the old database.

And we have to reconfigure the whole continuous integration process, so that we can deploy to multiple places at the same time. And we have to understand what to do when something fails (e.g, a deploy to IAD and STD succeeds, but LHR fails).

The good news is that this new infrastructure doesn’t impact directly those parts of the application that are already simple. HTML, CSS, Javascript, database tables, and application code will keep being simple and maintainable.

We will have to put more effort into the infrastructure and monitoring side of things.

But we will be fine

BK2 is pretty big. As of today, it has 200,484 lines of code in total. I think that’s impressive, for such a small team. The application is very stable. Just some bugs here and there that we end up closing pretty fast. And I think we have good velocity for developing. Most of the time we are working on new features.

I think we are in a good position to make this type of change.

The only thing we need in order to keep succeeding is to remember our first goal when rewriting this application: It has to be maintainable by a small team. So, let’s keep going! Let’s continue testing, and refactoring, and removing old code, and finding better names for things. Let’s keep removing features that don’t add much value.

We’ve got this!.

(And we will have more challenges in the future. We got them as well.)

~ Nica