When you get an opportunity to design a system from scratch and try to decide on an architecture; you would have stumbled upon various decision crossroads. Microservices, services bus, why not do monoliths so on and so forth. But what none of these take into consideration, seems to be the real-world human element to it.
Most of these don't take into account the market skillset, real-world human emotions, traits, and cultural advantages\baggage.
The IT processes and design patterns are documented and evangelized as if it's a homogenous IT crowd with similar skillset at varying degrees based on experience.
So, before we go into the pattern I am trying to explain here; let's have a look at the various kind of software engineers we come across in the market
The superstar difficult employee
You come across these frequently where they are naturally exceptionally intelligent and can crank out code like crazy. Not accepting their mistakes or not having the ability to work with others easily, are just a few challenges you have with them.
They need to be hired as contractors once you finalize the architecture; they can get it up and running very quickly and take it very far. Then without flinching and getting very comfortable with having so much done in such a short time, you need to end their contracts. They have accomplished the part you hired them for.
They are expensive and are worth every bit of their time. But you also need to know how to box their work and use them just for that need. I hope you never have to use them in the first place.
The Analyst (or as I call the real Developer)
You would be lucky to have these on your team. They are so far few in the current millennial crowd. They not only know how to write decent code but are thorough and can do very good unit testing both within and outside the given parameters; and also try to understand the business and think like a technical BA.
You can give an entire module or project and be rest assured it will be delivered with all scenarios thought through and that the right questions were asked.
They take ownership, can see the big picture, easy to work with, and go beyond their core responsibility and look to make the project successful as their responsibility instead. They need to be judged for their skill and need to be compensated fairly for what they are worth for your team than clubbing them into the job bands and throwing an HR playbook.
The heads-down coders
The bread and butter of the current IT industry. You get them from varying experiences, some who can write better code than others, know better coding concepts, patterns, practices, etc. Depending on the expertise they are either junior, mid or senior.
But what is common to all of them is, that they just want to write code.
They don't like analysis, requirements, or front running the QA and completing more user stories if they get time. All they want to do is come to work, take the requirements and write code for it and go home.
It's not that they are bad, it's just the kind of attitude the millennial generation has. 'I have a life' and 'That's not my job' are usually some of the responses you hear. Which is valid on paper and theoretically, but not ideal in the real-world IT delivery environment.
Now, I am not going through the employment market and going through all the kinds of candidates we come across; like the ones who cant code or won't even contribute enough to be worth their seat. I am just talking about folks who are worth filling a spot.
With the above mix, how do we build and maintain a system from scratch? While it's always enticing to use the latest and greatest, implementing what works is smarter than what's good in technology.
Monoliths
Monoliths are great! Ok, now hear me out. Any proven pattern has a place and use and if you cannot implement a monolith right, you cannot successfully implement any latest and greatest pattern too.
A well-written monolith application is fast and gets the work done needed and changes can be made easily.
If you have a need for a small application and a good Dev Analyst or 2 is all you think are needed to implement it, going with a monolith is your best way than overcomplicate it and break it down and all.
Now as we scale, you would hope you have these Sr Analysts who can hold the fort down, do very good code reviews checks and balances and make sure they utilize additional devs to write code while they maintain the integrity of the system.
Then on a larger scale, they have their shortcomings which have been told tirelessly by many already. But the point I am trying to make here is, that any pattern has its benefits and uses. If you cant implement one right, there isn't another pattern that is going to work for you to implement right.
Microservices with Service bus
Everyone advocates for Microservices, rightfully on their own merit. They are modular, distributed, and scalable.
But what are the core tenets of a system with microservices and how does it work
1. You have smaller broken-down services that are completely self-contained within the defined confines of the business responsibility.
2. They absolutely are not dependent on each other and are completely stateless. It gets a request, does what it should for the request, and sends a response with the data defined in its own database.
Now, how do all these come together for orchestration? There are multiple ways, but the most popular is having a service bus act as a load sink and coordinator across services while abstracting them.
Each one of these services doesn't care what others do, nor is dependent on them. They continuously look for messages which pertain to their trigger criteria to execute their methods and at the end of it publish a message of their process completion.
This is all good and will work, but the business requirement for this would for example go like this
Agreed that a well-defined abstract messaging structure will help it, but you cant break it down to give requirements to a bunch of developers without them knowing the larger picture. It's easier said than done to do that with a bunch of heads-down coders who can implement the business responsibility within their microservice and be done with it.
Microservices, Contextual services, and Orchestrator Bus
Assuming you have a typical mix of talent where you have a majority of the dev team of heads-down coders who write good code based on requirements, and then a few coding analysts whom you can depend on to write good analytical code and see the big picture, this could be a way to have a stable architecture which can run smoothly and also scale as much as you need.
What is an Orchestrator Bus?
It's a term I came up with where I felt a service bus instead of being dumb while each of the services is smarter to understand the business process. We tweak the roles a bit and make the bus have workflows that are defined at a centralized location with the services being dumbed down to just doing units of work.
The advantage to this is; coming to my initial point you have one good application that needs more Dev Analysts who typically are few in any IT team and then use all the coders to code your microservices.
Is the Orchestrator bus in this scenario a big monolith, with a single point of failure? Sure it is, but isn't the Service bus the same? And we did also talk about how limited responsibility and well-written monoliths are actually good.
The Orchestrator bus needs to be a very well-written monolith where managing the workflows needs to be configurable and easy on the fly changes than a code change to accommodate every time. And this will have to be maintained strictly by analysts who can see the bigger picture and maintain this well-oiled application.
This might be similar to what you might call a "Workflow manager" in the traditional sense, but it's not a generic workflow but custom defined for that context which is determined for the whole stack.
Contextual services?
The microservices will have to be made to adhere to have a standard set of input and output responses for the Orchestrator to be configurable and workflows defined on the fly. And hence, I call them contextual services. Because we work based on a context. The context should be defined during the initial setup with basic context (think of it as a session in a UI app) and then the same is passed along by the workflow manager.
The advantage to this is if a developer is working on Service 2 in the app, he just needs to be onboarded to understand the context and the limited Business responsibility that the service should accomplish. He need not worry about the big picture or anything. Just create a bunch of business functions in the service which use the context to gather data and execute it.
This structure also helps in the fact that if at any point there is a failure in the services, the restart or resubmission would be managed at the workflow manager level, while the service just does its unit of work when invoked.