Not so long ago, I’ve eventually decided to dive into the world of microservices.
I did look for an opportunity to make use of this architectural pattern for quite some time and finally was able to do so.
After 3 months of trying out the new things and learning stuff mostly on my own (the hard way) I believe it’s a good time to share some of my experience. I have no doubts that at some point in the future when I look back at this post I might be like – “oh God, what was I thinking back then, it’s so wrong”, but well, let me show you what did I learn so far and maybe you won’t repeat some of my mistakes.
Before we begin, let me just quickly “warn” you that what you’re about to read was not written by someone having tons of experience in designing the distributed systems. Although I’m not a total newbie in this area, I’m far from being an experienced system architect either, so just keep that in mind. Finally, let’s start with the core approach to the system design, which is defined as following:
Usually, you would create HTTP API that should act as a gateway, with no business logic defined whatsoever. Its most important job is to dispatch queries & commands via service bus without having any idea about the other services except the storage one which is being used for retrieving the data (which eventually will be returned to the end-user).
Of course, you might need to include some authentication mechanism here that will differentiate whether the requests is being sent by an anonymous user or not (usually via some token authentication included in HTTP header).
As the name states, it’s a kind of service responsible storing the data (like a read-only database). It acts as a sort of bridge between the API and the other microservices. Since the API has no idea what the heck is going on behind the service bus, in order to retrieve any data it sends the requests to the storage service. The storage service, on the other hand, has no idea about the API, yet does know about the microservices in case there’s a need to fetch the data which is not available in its internal database.
I do realize that you might wonder – why would you create a separate service acting as a data storage? Isn’t having a separate database that the API can talk to directly a good enough solution? Well, it really depends on what are you trying to achieve. It certainly might be a good and sufficient solution, but think about the following scenario – in order to store objects in a read-only database (CQRS stuff here) you most likely need to subscribe to all sorts of the events like UserCreated, InvoicePaid etc. Then you need to talk to a specific (micro)service, fetch the data and save it into the database. In the former scenario your API would be responsible for subscribing the events, mapping the data and saving it into the database – is that wrong? Most, certainly not, however, I prefer the latter solution which is a full separation of the API from the microservices.
In that case, there’s a so-called storage service that subscribes to the events, fetches the data from the (micro)services, knows how to flatten the objects etc. The API only needs to perform the HTTP requests to the Storage service in order to get the data – and it doesn’t really care whether it comes from some internal database or cache or some service that is located at the end of the world. Like I said – both solution are viable, both of them have their pros and cons, but personally I really like to keep the API clean and remove any unnecessary logic from it, which in that scenario would be subscribing to the events and storing the data on top of dispatching the commands.
The core of the system where each service is a totally separate “being” from each other service, period. They (services) have no idea that there’s someone else in the world listening to the commands and events dispatched by themselves. Each (micro)service does possess its own domain models, repositories, business logic and so on. The only thing that is common for the whole infrastructure are the service bus and a set of commands & events.
Now, that we have a rather (at least I hope so) clear picture about the system, here are some of my remarks related to the idea per se and some implementation details.
Tips & tricks
Keep your services small.
It’s better to have many small microservices that focus on a single domain than a few bloated ones that perform totally different tasks & manage unrelated responsibilities under the same scope. The most usual examples would be: creating/validation the user accounts, sending the messages, managing products, handling the payments etc. and each domain would fit into a separate service with its unique entities.
Follow the CQRS pattern.
Send the commands that have no return values, and perform queries that are idempotent – that’s all you need to follow the CQRS. If you stick to this pattern, you will quickly find out that it’s much easier to scale your applications simply by separating read & write operations.
Share commands, events and DTOs.
Create a shared project where you will keep commands, events and DTO (data transfer objects) used within the whole system. You may also include here some helper methods, extensions etc. – just make sure you don’t put there any specific domain models or business logic.
Organize your workflows by using events & command.
Thanks to the commands (that you can think of requests being sent by the end-users) which may produce the events (and so on), you can easily create very complex processes if you need to. The services shouldn’t talk to each other at all, as it does introduce tight coupling. Instead, publish additional commands and events.
Choose a database that suits service needs.
Each service (not a single instance, cause you may have many instances of the same service running on different nodes) should have its own database. Not only you will eliminate a single point of failure (a single, huge database for the whole system) but most importantly you will have a freedom to pick the best database for the particular tasks. You might want to use SQL for some financial operations which rely heavily on transactions or NoSQL database for storing billions of JSON documents.
Track the information about requests being processed.
Whenever you dispatch a command to the service bus, assign to it some meaningful information like id, name etc. and store it somewhere. Thanks to this, you will be quickly able to determine whether it has succeeded or not. Personally, I create a separate (micro)service that subscribes to all of the important commands & events and stores the information about performed requests within its internal database. For example, there’s a CreateUser command, it gets some unique ID and state equals to “created” and once the user was created there’s a UserCreated event being published which contains ID of its parent command, therefore I’m able to update the requests that e.g. to “succeeded”.
Adjust your end-client apps to the asynchronous processing.
Once you send a request to the API you will not receive an instant response, at least not in a synchronous way that you might be accustomed to.
For example, I return a 202 Accepted HTTP status code with some custom header containing the URL of the unique operation (described in the previous paragraph). Then, the end-user can fetch operation state and once it’s completed the resource will become available to use.
Ensure you can easily deploy a new instance of the service.
The beauty of a good (micro)services architecture is having an ability to quickly scale required services in order to handle the increased load.
If you can’t simply deploy a service to the new server without interfering the other parts of your system it means that something ain’t right.
Write end-to-end tests.
I believe that having end-to-end tests is a crucial part of the system being built with (micro)services. There’s hardly any another way to ensure that everything works as expected until you can run some E2E tests (e.g by using HttpClient that will send raw HTTP Requests to the API and validate the received response).
Include failover, service discovery and other useful mechanisms.
Whenever something goes wrong, you want to make sure that your whole system doesn’t crash, or at least part of it. Make sure you have some retry policies included (e.g. Polly), service discovery tools such as Consul and also keep your credentials in some centralized place e.g. using Vault, Azure Key Vault or my open source project Lockbox.