Beyond 10,000 Lines

Lessons Learned from a Large Phoenix Project

Daniel Berkompas
Red Shift

--

We recently finished up a large Phoenix project at Infinite Red, and we learned some tips and tricks along the way that I want to share. Overall, we’ve been impressed at how well organized and maintainable a Phoenix project can be beyond 10,000 lines of code.

Some of the tips in this post are specific to Phoenix, others could be equally applied to Rails or other web frameworks. None of them are meant to be applied woodenly though, so keep your brain on.

1. No Repo in Ecto Schemas

Don’t use your Repo in your Ecto Schemas. Don’t alias it, and don’t use it. This is a very easy way to prevent your schemas from becoming large and bloated like ActiveRecord models tend to be in Rails. Without access to Repo, your schemas can’t be the center of your business logic anymore, because they can’t execute queries. This actually feels natural, because Phoenix has much better ways to share business logic, such as the following:

Views. If the business logic is view-related, put it in helper functions in your view modules. You can easily share these functions between your views with import calls.

import MyApp.PhoneView, only: [format_phone: 1]

Module Plugs. Often you’ll have somewhat complicated logic for fetching data out of the database. We find that the best place to put logic like this is in module plugs that add assigns to the conn. You can then use these plugs in any controller, or even whole Router scopes. We added a web/plugs folder to our project to promote this.

# In controllers
plug MyApp.CurrentUser

# In Router Scopes
pipeline :authorization do
plug MyApp.CurrentUser
end

scope "/", MyApp do
pipe_through [:authorization]
end

Services. As I wrote in “Replacing Callbacks with Ecto.Multi,” side effects are best done in a service layer built on Ecto.Multi. See that post for more. We added a web/services folder for these.

2. Use OTP to Your Advantage

Because Heroku was our deployment target, we were limited in how we could use OTP. (Heroku isolates your dynos from each other, so you can’t really set up an OTP cluster without paying a lot extra for a private Heroku space). However, we still came up with a number of clever ways to use supervised processes to our advantage.

  • Automatically starting Elasticsearch and Kibana in development mode. See more in this blog post.
  • Running mix tasks regularly on a custom schedule, (like cron jobs) not supported by Heroku scheduler. We did this with a very simple scheduler GenServer, which you can see in this gist.

3. Write Fewer, Valuable Tests

We focused our automated tests on our controller actions and plugs rather than going for 100% test coverage. Since these are the main ways that the Phoenix application interfaces with the outside world, they’re the critical points of failure.

Controller tests also exercise a lot of the code paths in your application, making it less necessary to unit test every single module. As a result, you end up with fewer tests, which makes it easier to do refactoring, provided your changes preserve the behavior of the controllers and plugs.

4. Avoid DSLs

We avoided adding any new DSLs beyond those which come with Ecto and Phoenix, and so far this has been a big win for us. There are at least three areas where we could have used a DSL, but didn’t.

  1. Integration with 3rd-party APIs
  2. Test Data
  3. Admin Panel

Let’s talk in more detail about each one of these.

4.1 Integration with 3rd-party APIs

We used Elasticsearch to power a search feature, but instead of using a library with a DSL, we talk to the REST API directly with a tiny HTTPoison module.

This means that our code looks very close to the official Elasticsearch documentation. We don’t have to figure out how to make a given JSON request with some DSL, we just make the request! If the documentation shows a query like this:

GET /_search
{
"query": {
"bool": {
"must": [
{ "match": { "title": "Search" }},
{ "match": { "content": "Elasticsearch" }}
],
"filter": [
{ "term": { "status": "published" }},
{ "range": { "publish_date": { "gte": "2015-01-01" }}}
]
}
}
}

We can execute that query from Elixir like this, only having to convert the notation from JSON to Elixir.

Elasticsearch.get! "/_search", %{
"query" => {
"bool" => {
"must" => [
{ "match" => { "title" => "Search" }},
{ "match" => { "content" => "Elasticsearch" }}
],
"filter" => [
{ "term" => { "status" => "published" }},
{ "range" => { "publish_date" => { "gte" => "2015-01-01" }}}
]
}
}
}

Simplicity like this is very valuable and allowed us to iterate much quicker. It makes the site much easier to maintain since new developers can follow the official Elasticsearch documentation closely to learn how it works and implement new features.

4.2 Test Data

We followed my advice in “Fixtures for Ecto” for test data, and it has worked out great. Even for a project of this size with 37 models and many associations, there really is no need for a DSL. Simple functions work just fine. In fact, we only have one file for all of our fixtures, yet it’s been quite manageable.

4.3 Admin Panel

The project has an extensive admin panel which will continue to expand in the future. In the Rails world, it’s common to generate these kinds of panels using ActiveAdmin or other libraries that rely on a DSL.

Rather than use this approach, we built the admin using generators which create standard Phoenix code. We get all the benefits of ActiveAdmin, such as quick setup, search filtering, and even some new things like customizable styles, but without the downsides of a DSL.

If we want to add anything to the Admin, we can do that. It’s just regular Phoenix code, with controllers, views, templates and the rest. There’s no need to learn anything new or wrangle with a DSL to maintain it. We’ll be open-sourcing these admin panel generators soon. Follow this publication if you want to be the first to know!

5. Avoid Non-RESTful Routes

This tip holds true for any REST-based web framework, including Rails. Whenever you find yourself adding a function to your controller that is not “index,” “new,” “create,” “show,” “edit,” “update,” or “delete,” treat it as a code smell and investigate. Why are you adding this? There might be a good reason, but usually there isn’t.

Throughout this project, I’ve observed that whenever we added a non-RESTful route:

  • It was the result of muddled thinking about the requirements, and often didn’t work at all.
  • It combined multiple responsibilities into one controller which logically belong to two or more separate controllers. This made the controller harder to work with and maintain because it was doing too much.
  • Because of the above, each instance had to be refactored later, causing extra effort.

Far better to just keep all your routes and actions RESTful in the first place. Creating new controllers is cheap; there’s no reason to resist it. We currently have 81 controllers, and it still doesn’t feel like too much.

Conclusion

We learned a lot from this project, but it has validated for us that Phoenix is a stable platform for large and small projects alike. Despite the fact that we’re well over 20,000 lines of Elixir code, the codebase does not feel bloated or disorganized, and refactoring remains easy.

These practices helped a ton. But we also found that Elixir itself is pretty forgiving of mistakes. Since nearly all the code is made up of stateless modules and functions, it isn’t hard to move things around and refactor. Even if you do make mistakes, you’re less likely to end up with code you can’t maintain.

So, if you’ve been waiting for Phoenix to “mature” before jumping in, your wait is over. It’s mature enough now. Let’s build things!

--

--