GenStage Under Umbrella?

GenStage Under Umbrella — Part 1

Testing The GenStages Under Umbrella

Implementing GenStages Under Umbrella

Subscribe GenStages Under Umbrella

At Fitzdares, we have already quite a few production projects implemented in Elixir. We worked with Umbrella apps and with GenStage but never combined them. Until now! One advantage of the 10% personal investment time is that we can use it to experiment new ideas that can eventually end up integrated into the live projects.

I must say it from the start. This is not an article about GenStage. It is not about back-pressure or optimizing maximum demand. We will use the most basic GenStage configuration.

But we are discussing something very specific. Using GenStage inside an Elixir Umbrella as a way of communication between applications.

Context

For the purpose of this demo, we will imagine a stock market information application. We call it Stockr. The application is based in the UK, but we have 2 counterparts in US and Germany. They provide us with the country specific stock market info. In return, we do the same for them. We are sending back real time UK stock market data. Each of the 2 info providers is sending info in their country currency. Also they must receive back information in the same currency.

eg:
- the US data provider sends info in USD
- our app must receive it in GBP
- our app will send back info in GBP
- the US provider must receive it in USD

At this point we assume that the project is doing much more than this and using many apps inside an umbrella makes sense. In other words, we are not questioning the umbrella vs monolith decision.

The example, which we will develop in Part 2 of the article will have no UI. We will use end to end testing to check if our concept works.

The Umbrella Apps

From the context above we can think about 4 applications in the umbrella:

• UsaMarket — handle info from and to the US counterpart
• GerMarket — handle info from and to the German counterpart
• Converter — converts the prices between the currencies
• MyUkApp — our actual stock market info app

Let’s Talk!

It’s time to think how the apps will talk to each other. Umbrella apps may have other apps as dependencies. Add {:my_dependency_app, in_umbrella: true} in the mix.exs deps. The first approach would be to UsaMarket and GerMarket to have Converter as dependency. Converter will have MyUkApp as dependency. Very easy, and it works! Unfortunately just up to some point.

We can receive info from US and Germany, convert it in GBP and display it to our users. Yet, sending info back from MyUkApp may fail. Circular dependencies are not possible between the Umbrella apps. You cannot include the “parent” app as “child’s” dependency. You will be able call, for example, a function from the Converter module in MyUkApp. But is certainly not the right way to do it. You may end up with an error like: module Converter.MyModule is not available if Converter is not compiled. More than this, your MyUkApp tests will fail if you try to run them from inside the App and not at the Umbrella level.

To implement the reverse information flow we need to create more apps. Each of them will fulfil a specific role, like a producer or a consumer of data. The diagram below shows such a scenario:

Again, this will work. But we already created 7 applications and probably lots of code duplication. This is not going in the right direction. Any new possible scenario or edge case may need a new set of applications. All this just to ensure the correct data flow by managing the dependencies.

But observe the big picture! Without even realizing we designed a GenStage information flow. If you didn’t until now, it would be a good time to check at least the first chapters of the GenStage documentation.

The “Flat Umbrella” and the GenStage

all the apps in the Umbrella are created equal

First, we would want to get rid of the app dependencies in the Umbrella. Calling a function from a child is not the only way of communication between Umbrella apps. There can be various ways: direct processes messages, PubSub messages, message queues, etc. But we will pick the GenStage.

We will keep the four initial applications, without any dependency between them. Each of them will “host” two GenStages:

• one for the receive information flow, for the data we get from the external sources. The two external counterparts will play the role of provider. Our app, in this case, will be the GenStage final consumer.
• the other for the send information flow, for the data we send back. MyUkApp will become the producer in this case. The external sources will be theconsumers.

In both cases the Converter will be a producer_consumer who’s responsibility is to handle the currency exchange.

This is not hard to implement and is easily extensible. Let’s say we will add another provider from China. We will create its receive producer and send consumer and “plug” it to the Converter. That’s it. MyUkApp will not even know about it.

In the next article, we will write the tests for the future GenStages implementation.