Learn Building Modern Go applications

Whatever you think of AI coding, the way we build software is changing. Everyone wonders, “What will stay relevant?” All we have is opinions, so here’s mine: most ideas behind Domain-Driven Design are now more relevant than ever, as DDD has never been strictly about code. As we do more coding with agents, we still need to understand the domain, model it well, and work as a team. Let’s see what DDD can still teach us and what AI can’t replace. The usual disclaimer: I write about complex projects maintained over months or years, where teams tend to struggle and where DDD shines. There’s no reason to use advanced patterns in pet projects or CRUDs. What hasn’t changed since 2003 Eric Evans published Domain-Driven Design more than two decades ago, but it’s still surprisingly fresh. The book covers many topics, but the main message is that the hard part of software engineering is understanding the problem domain and modeling it well in code. It calls on engineers not to get lost in the technical details and to focus on what they’re solving, because it’s more difficult to get that part right. Instead, the technical talent goes to work on elaborate frameworks, trying to solve domain problems with technology. Learning about and modeling the domain is left to others. Complexity in the heart of software has to be tackled head-on. To do otherwise is to risk irrelevance. Eric Evans Domain-Driven Design We now see that the implementation details become less relevant thanks to AI tools. To a degree, you can be productive without a deep knowledge of a programming language or a framework, and it’s easier to move to another tech stack. If you’ve tried using the top coding agents, you’ve probably seen them quickly generate decent code. But knowing high-level software concepts is still essential to guide agents and to decide whether the output makes sense. Most of all, the domain complexity (the business problem you’re solving) is still the hard part. The good news is that coding is easier than ever, so we can focus on these complex problems instead of getting lost in the implementation details. Except, instead of obsessing over building the frameworks (or adding more microservices, or whatever else is a cool trend), we now follow the model benchmarks and optimize our agentic setup to generate better code with less effort. As Evans noted back in 2003, we still try to solve the domain problems with technology. Funnily enough, this time the CEOs also believe it and push us toward it. There’s just one small thing left to figure out: how the thing should work. If someone tells us, we can throw tokens at the implementation and call it a day. And if no one knows, well, maybe an agent could write the plan for us? The domain model isn’t an artifact The foundation of DDD is model-driven design: distilling how the domain works into a model that’s expressed in code. The domain model is an abstract concept, and there’s no one way to represent it. You can use documents, diagrams, and code, but they are all simplifications of how the domain works. An accurate model comes from switching between design and implementation, and applying what you’ve learned over and over. DDD calls this knowledge crunching. You need domain experts (the people who know how the business works), and engineers who are experts at building software. They work together to understand what the software should do and how to implement it. (For example, in an Event Storming session, a workshop where developers and stakeholders map out how the business works with sticky notes.) Figuring out the domain is a team effort and hard work, so it’s tempting to have someone else do it. For example, AI agents are brilliant at research and analysis, and can access your documents, chats, and code. It seems they could create the domain model for you, and then also implement it. But this naive approach misses the point. The value of working on the model is that you (and your team) understand how the domain works. An AI-generated wall of text doesn’t help you figure out the business problem. The agent will just create an impressive artifact no one reads. A domain model isn't an artifact. The challenge has never been to write a long document on how something could possibly work. More often, software projects fail because of other issues: Engineers work on the wrong thing. No one knows what needs to be done in the first place. The team tries to build everything at once (scope creep). Here’s one trap I’ve fallen into many times. In engineering teams, it’s easy to feel like we’ve got it all figured out: we’re programming experts, know our patterns, and follow best practices. We just need someone (a business stakeholder or a product manager) to tell us what to do. But often, the domain experts don’t have a clear plan either. They know the problem they have, but not what the solution is yet. There’s no complete document with all requirements. A developer’s first instinct may be to think, “Let me know once you come up with something, I’ll be around.” In other words, we’re here for the fun part of coding. (Note this is the part AI agents automate.) At the other extreme, non-technical managers try to come up with the entire solution and hand it to the team to implement. Or, the most recent trend, someone uses AI to generate a full feature or product spec, doesn’t even read it, and passes it along. So the plan sort of exists, but it’s nowhere near something ready for production. (If someone claims otherwise, check how complex the thing they’re shipping really is.) In all of these scenarios, no one has a clear idea what they’re doing. No matter who writes the code or how fast, it has little value at this point. In contrast, when you work on the model together, your team understands the problem domain and agrees on a solution. Developers implement it, and domain experts can jump in once something unexpected comes up or another iteration starts. Even if you do most coding with AI, you need to get deeply familiar with the domain to guide the agent, just as you would a teammate. How you configure the agents and what LLM you use don’t matter as much as a clear mental model of the solution. If I had to choose, I’d rather never write any code myself than skip thinking about what needs to be done. Design before writing generating code Writing code has always been easier than reading it. It’s just more extreme now that an agent can single-shot a big feature for you. Even if it works, you have no idea what’s going on under the hood. We now get stuck on code review more often, but it’s also not a new problem. It even feels familiar, like working with a lone-wolf developer who prepares complete features and drops a massive PR on you. You know nothing about the design and there are often gaps that no one has considered. Reviewing the PR is painful. There’s a way to avoid situations like this, and I’ve seen it work in many projects. You design and discuss the solution as a team before sitting down to code. First, the domain parts: what you’re solving and how it’s supposed to work (see knowledge crunching above). Then, the technical side. After you kick off the implementation, there’s little chance of anything unexpected coming up. You don’t need to agree on every tiny detail, and don’t need a complete written spec. It’s enough to sketch a high-level plan that everyone is aware of. Some sticky notes and a whiteboard are good enough. The best moment to change the design is before anyone writes code. After sessions like this, reviewing PRs is smooth. It’s easy to split the work into small chunks if you don’t discuss the entire design in PR comments. And there are fewer gaps once the feature lands in production, because you’ve already agreed on the details and everyone understands what they’re building. Skipping the design may seem like saving time. No one likes more meetings. But if your team first learns about the feature from the PR, it takes much longer to review, discuss, and fix all the issues. Since reviews are usually asynchronous, you end up with many rounds of comments. Code review should be a double-check that the implementation is correct, not the start of a discussion about whether the approach makes sense at all. Starting with a brief plan saves you time on review later on. Working on design before code will also help you work better with AI agents. If you give the agent more high-quality context, its output will be more accurate and the coding part will be shorter. And the more your team knows about the solution, the easier the review, even if you generate all the code. Ubiquitous Language: Speaking the same language as your agents Recently, I’ve been trying to cut our cloud costs. It’s the perfect task for AI, since there’s an obvious metric to measure. I told the agent to find where our Go service uses the most memory and to cut it. After a couple of runs, I saved us many megabytes of RAM. But then I realized memory is pretty cheap, and the updates reduced our costs by $1/month. Not surprising, right? I didn’t explain to the agent that I was trying to cut costs. I just gave it the memory usage as the target. It should sound familiar, because it’s exactly what happens between humans in teams. Good practices that help you communicate in a team are also relevant when you work with AI. (Who would have thought soft skills may help you deal with computers?) DDD addresses this with a few ideas around the language you use. The first is Ubiquitous Language: speaking the same language across teams and in code, so everyone knows what you’re talking about. In many teams, messy names are the default. In code, engineers use technical terms that drift away from the names that the rest of the company uses. Even if you feed AI documents and decision records, the agent needs to make sense of whatever language you use there. During knowledge crunching, consider what you call the concepts in the domain. Initially, it’s common to have more than one name for each. Choose a name you already use, or come up with a more accurate one. Then, use it in the documents, spoken language, and code. An Event Storming session is a great place for choosing the names. You can’t choose the names just once, because the language will keep evolving with the project. It’s something you need to care about during the design sessions and discussions. Another idea directly connected to the Ubiquitous Language is that in bigger systems, the names you choose won’t be universal in all areas. DDD calls each of these areas a Bounded Context. You can think of your customer as a profile in the support context, and as a user in the e-commerce context. You want to use consistent names within each context, not force them across the entire project. This is especially important with bigger products, when you need to split the software between teams. If you focus too much on creating one model for all kinds of tasks, it becomes difficult to draw boundaries. With bounded contexts in mind, it’s easier to end up with a decoupled system. Agents have access to your whole repository, and they’ll look for the names you give them. They may naively try to unify similar entities, so make it clear they are separate for a reason. The same domain concept can have different names depending on the context. If you use well-known, precise concepts, it’s easier to express what you mean. It’ll work both with your teammates and in your prompts. A vague prompt can make the agent spend a lot of effort on the wrong solution: Add user to CRM and support after it’s created You’ll make the goal more obvious by sticking to the precise names: Once the user signs up on the website, asynchronously create: 1) a customer entry in the CRM, 2) a profile in the support system You want the details about your domain language in your agents’ context. It’s yet another reason to keep the documentation up to date. Markdown files close to the code are a good starting point, since you don’t need a custom integration to fetch the documents. Developers become domain experts The models and agents keep getting better, but you can’t fully trust them. They still make mistakes, and you need someone accountable in the end. How can you know if the outcome is correct? Working in the domain is how you build trust that you know what you’re doing. People who design the system and understand how it works can tell you whether something is likely to go wrong. Developers become experts in the subject they’re working on. They learn skills beyond writing code, and for a curious engineer, even a complex business domain becomes intuitive with time. If you need help in that area, you know who to ask. You save yourself massive headaches by working with someone you can trust to get the job done without extremely detailed instructions. There are two recent AI narratives. One is that anyone can now generate a clone of a well-known app. This was shocking at first, but isn’t that interesting in the long run. We already have too many half-baked apps out there even without AI. The more interesting story is how experienced engineers can build more complex applications now. For example, a team can replace expensive third-party software with an in-house solution. But how can they trust their version works well? Because of the knowledge they already have thanks to working in that specific domain. They combine what they know with AI and build their own version so fast, it’s cheaper than paying for the license. On top of that, they can also add any custom features they need. For employers, having product-minded engineers around who know the domain well is huge. They can figure out the obscure details and guide the discussions. It’s always been naive to treat developers as people who code what you tell them, and now it’s even more so. For engineers, learning to work with an unknown domain pays off more than focusing on technical skills alone. It’s easier to work in a profit center if you’re good at solving business problems, rather than being an expert in one framework. Don’t delegate thinking DDD shows how there’s much more to software than writing code. If you’ve worked with complex systems, I’m sure you’ve seen it many times. This topic brings back memories from my first software job. I once worked on a tough problem, and spent hours doodling with pen and paper. I finally figured it out, even though I barely touched the keyboard. I had only recently started, so I was speechless: “Wow, today I got paid to think.” This felt wild at the time, but today it feels even wilder. Back then, my manager wouldn’t blink an eye if I said I spent a day thinking about a problem. Today, we focus more on the output. Why not have the AI do the thinking instead? Thinking, delegated. For one, because the only reason I can judge whether the agent’s output makes sense is that I’ve spent long hours thinking about such problems in the past. There’s the argument that you no longer need this experience, since AI will handle everything for you anyway. Relying on AI could eventually become the new normal, but there’s one issue I can’t ignore. You need to exercise your brain to work on complex problems, and you can’t build new mental models without thinking. The best analogy I have is reading a book versus a summary of it. Reading forces you to think about the topic for hours. You slowly build a complex model in your mind, and you connect it to other things you know. You understand the topic, instead of reading a few sentences that you forget ten seconds later when you close the browser tab. No one knows the best practices yet, so it’s a good time to experiment and see what works. For now, I’ll mix old-school problem-solving with trying out new ways of working with code. In this post I focused more on the philosophy behind DDD than individual patterns. I plan to write more, including about the tactical patterns closer to the code. I still find them useful, even if I don’t type them by hand. Sign up below so you don’t miss the next posts.

This week I was working on one non-critical feature that seemed simple at first sight. 50k lines of code later, it seemed to work as expected. In a side project, or a service where “nobody will be upset if it’s dead and kills everything around”, it would be ready to ship. But for a production-grade project, the risk of an outage is too big to skip reading the code. Unfortunately, reviewing 50k lines of AI-generated code is much harder than reviewing 50k lines written by a human. We’re now obsessed with how much code we can create fast. But is it the right thing to optimize for? The biggest bottleneck of implementation quietly shifted from producing code to reading code. Fortunately, we don’t need to invent how to make code more readable: software engineering has had solutions for that for decades. The next bottleneck: creating code that we can be accountable for Many people no longer read the code that they ship. So why should we care about good coding practices and code quality if agents don’t? Keeping our code readable is still critical for one reason: it helps us overcome our biggest bottleneck, the human. When you write code by hand, you have the entire context of what you implemented. With agent-generated code, you have to rebuild that context, and it’s easy to get overwhelmed. You miss bugs that nobody would ever write by hand. You also lose the edge cases you would have found while writing the code yourself. Coding got faster. Review didn’t. Coding got faster. Review didn’t. We can review less code than before. It may be tempting to skip review entirely, but that’s throwing the baby out with the bathwater. The last block in the diagram is “successful delivery” for a good reason: our goal is not just to ship, but to ship working things that don’t break other things and won’t slow us down in the long term. Judging by how many outages big companies had recently, this goal is not obvious to everyone. It’s a well-known pattern: new techniques and tools make top-performing teams even faster, while low performers get slower. It happened with the cloud and with microservices. The pragmatic teams shipped faster, while low performers got stuck in never-ending migrations. The same will happen with AI. Teams with high friction between idea and implementation won’t see the same performance gain. And if you invest in AI in the wrong places, you may even end up slower. You can generate 100k lines of code and burn thousands of dollars on tokens in one day. That means nothing until you’re sure it works, doesn’t break other things, and solves the problem it should. In other words, it doesn’t matter how much code you can generate. What matters is how much of it you can take accountability for and whether it solves a real problem. You probably see PRs like this too often now You generated a new feature, and after a couple of iterations you believe it’s ready. What now? You have a couple of options for how to find out if it works as expected: Just create a PR without reading the code and let the reviewer check the feature… and wait for your team to hate you for submitting slop-PRs. Just read the tests and pray that AI agents didn’t forget any edge cases, that they covered all side effects, and that previous features are also protected against regressions. Read and understand the damn code. The third option is the safest, but unfortunately the hardest one. Maybe we saved a lot of time we would spend on writing code, but now we have a harder task: reviewing it. The volume of code generated by agents is much bigger than what people used to write by hand. What’s worse, reviewing code written by an agent is much harder than reviewing code that you have written by hand. After spending hours reading and fixing the code, you start thinking “Maybe writing it by hand would be faster…”. Writing by hand sounds like a comfortable solution, but even counting the hours you spend polishing the agent’s code, you would not be faster. What helps here is making the agents’ code more readable. The good news is that we don’t need to reinvent the wheel. Creating readable code is something that software engineers have been trying to figure out for several decades. Good practices exist to help us extend code faster. But we often miss the one part that matters most for review: readability. Not all software is the same We have always been big advocates of Domain-Driven Design and Clean Architecture in the Go community. We have seen how they helped us to rescue many projects. To our surprise, promoting them used to trigger pushback from many people saying “it’s overengineering and Go code should be simple”. And they have been right: Using DDD and Clean Architecture leads to overengineering… and so does skipping them. It all depends on the kind of project you apply them in and on how well you understand them. When we talked to skeptics who advocate for “Simple Go”, we quickly figured out that their opinion often came from a single poorly implemented project. Usually, the people who built that project had no idea how to use those tools. Or this project was just a simple CRUD that didn’t need any sophisticated techniques. Any tool, when wrongly used, does more harm than good. What works for prototypes, CRUDs, and side projects doesn’t work for production-grade complex projects. On the other hand, prototypes and simple code will suffer from many ceremonies that production-grade critical software needs. The difference matters more than ever with AI-assisted coding. Not reading generated code is also fine I don’t read every piece of AI-generated code. One type of code I often don’t review at all is the presentation layer (frontend, for example). As long as it looks good and works as expected, I don’t care about the implementation. There is one important precondition to make it happen: the frontend should be a thin presentation layer without overcomplicated logic, with a sane API behind it. When you have complex frontend state with GraphQL under the hood, such a strategy is much riskier. This is where any kind of layered architecture helps us: we separate the boring code from the critical code, so we don’t need to spend the same mental effort on all of it. Next time you hear someone say they never read generated code, check what kind of code they write. Most of the time it may be presentation-heavy non-critical software. It can be a sane strategy, as long as you apply it deliberately to one part of the project. The more things change, the more they stay the same Many software engineers now face an existential crisis: did everything we learned over the years become obsolete? If someone spent years learning languages, frameworks, and vim motions, then it’s true: those skills are much less relevant now. Even if we see more and more examples of agents doing impressive work, it’s mostly in scenarios with a clear feedback loop that agents can “brute-force”. The challenge starts when success criteria are unclear. That describes most of the business software we create. I also see arguments like “software is dead, my girlfriend just vibe coded in a weekend what I used to write in half a year”. Even if it may be true, we come back to the same point: not all software is the same. Shipping a PoC in a weekend is light years from building production-grade software with many users. If that PoC became popular, it would take a team of engineers to develop it and keep it working (and making money). An expert product engineer decides what to build and why, not only how. If that’s you, your knowledge of what code to write is still valid and in demand. Part of this job was always taking accountability for the changes we ship. It doesn’t matter if the code was written by hand or generated by an agent: you ship it, you own it. Not much has changed. Taking accountability was always the area where the best job offers were sitting. Good coding practices were never for the machines. Compilers never cared about them, and agents may work slightly better with clean code, but they are fast enough without it. Good practices were always for you, the person accountable for what you ship. The one thing that changed is that we, as software engineers, now have to figure out how to collaborate with agents to create code efficiently. In the next posts, we’ll show you how we do it.

Quick takeaways Start with a monolith - don’t create microservices from day one; the overhead isn’t worth it until you have real pain points Microservices solve human problems, not just technical ones - they help teams work independently, not just scale systems Tight coupling over HTTP is still tight coupling - separating services doesn’t automatically give you isolation Watch for signals - lots of calls between services or frequent changes across multiple services suggest wrong boundaries Sometimes joining services is the answer - merging tightly coupled microservices back together can be the right move Introduction We talk about splitting software: when to do it, how to avoid common pitfalls, and why microservices aren’t always the answer. We share stories from past projects where we jumped on the microservices hype train without thinking through the real problems we were trying to solve. The key insight is that splitting software is more about organizing teams and maintaining good boundaries than about technology. Notes The Domain Engineer training - includes a complex modular monolith example Modular monolith approach - starting with a well-structured monolith that can be split later if needed Event Storming - a technique for discovering system boundaries by mapping domain events Conway’s Law - the idea that systems mirror organizational structure Team Topologies book by Matthew Skelton and Manuel Pais - recommended for understanding how team structure affects software design Reverse Conway Maneuver - designing organizational structure first, then letting software follow Service mesh tooling (like Istio) - for observability between microservices Quotes This aspect of microservices I think can be also misleading. Because it seems like you have these separate things. You see the deployments, you see the different repositories. For an outsider, you can think they are separate beings that can be changed independently of each other. It can be a trap. Miłosz You can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem. Robert So if you consider splitting your project, I would start with this question: if you even need to do it. Because you often see people struggling or hear them asking, I’m starting a new project, kicking off some application, and what services do I need to create? So probably at this stage, you don’t need to consider this at all. Miłosz I think it’s also nicely visible if you take your service mesh tooling, and probably will see that between some microservices will have a lot of calls going back and forth. And it’s usually the sign that maybe those two services shouldn’t be separate. Robert The more developers you add to the project, the more complicated it gets. Because this is also important factor. If you are a single person working on software like this, it will be fine probably because you can keep the context in your head. You will refactor to whatever you want. But if you have like 20 people working on it, it starts getting more complicated. Miłosz But there’s one good thing about microservices is that it’s forcing the boundaries really hard. So it’s hard to go over network if it’s not possible. Robert Timestamps 00:00:00 - Introduction 00:01:37 - AI and splitting software 00:02:42 - When splitting matters 00:04:39 - Monolith first approach 00:06:08 - Microservices hype era 00:09:34 - Microservices force boundaries 00:13:16 - The isolation illusion 00:18:31 - Merging services back 00:23:09 - Event storming for boundaries 00:27:41 - Core vs generic domains 00:32:05 - Bounded contexts and shared kernel 00:35:30 - Conway's Law and Team Topologies Transcript Miłosz [00:00:00]: So, if you consider splitting your project, I would start with this question: if you even need to do it, because you often see people struggling or hear them say, ‘How do I start?’ I’m starting a new project, kicking off some application. What services do I need to create? This aspect of microservices I think can be also misleading. Because it seems like you have these separate things. You see the deployments, you see the different repositories. For an outsider, you can think they are separate. Entities separate beings that can be changed independently of each other. It can be a trap. You have to think: what problem do you want to solve? This way, rather than just ‘Oh yeah industrial best practice. Robert [00:00:41]: Probably, can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem. Probably, you’re mentioning the monolith first approach that we are big advocates. We’ve tried it multiple times for products that become bigger later. If done properly, it’s working. It’s accelerating you a lot at the beginning and allows you to change it later. Miłosz [00:01:05]: And what’s surprising to me, it’s not as popular. Miłosz [00:01:11]: So, welcome after break. Let’s talk about splitting software, different ways of splitting software, and why it’s important in the first place. Robert [00:01:20]: Yeah, I think it’s a good topic because, probably, it’s the topic that everybody writing software in this or another way needs to know how to do that. And if it’s not done properly, it may create a lot of issues. Miłosz [00:01:37]: Probably even if we end up doing coding with AI in the next couple of years, it will be an important topic because it’s like one of the key architectural decisions you can make in your project. Robert [00:01:50]: Especially, from our experience, even if people were doing that, usually they were missing that. And I would not expect that some cloud code or something like that. Miłosz [00:02:00]: Handle it better than probably the opposite, actually, because there are so many anti-patterns on the internet. You know, like people tend to make mistakes when splitting software or do it in a very naive way. So yeah, I would expect coding agents to probably make similar mistakes. Robert [00:02:20]: I think it’s you know not only about some inexperienced people—I think also sometimes very experienced people, without experience, may miss some signals from the systems. So when you’re building some system, it’s sometimes hard to miss some signal when something is going in the wrong direction. And it happens even for those experienced people. Miłosz [00:02:42]: So it’s often the issue with big codebases, right? When we work with a big project, over time tends to be more difficult to work with, especially with bigger teams or multiple teams. And then the topic of splitting comes up. And I was thinking about, you know, our past projects on different companies, how important the problem is. And one thing I realized is that when we were working on our own projects before our full-time jobs, the stopping wasn’t really that important. Important so it tells me something right, that’s even if those applications were sometimes fairly complex, wasn’t. If it’s just you and maybe another person working on some project, it might still be fine sticking to one repository or codebase. Robert [00:03:45]: Yeah, so probably can explain why, for example, people like Peter levels can have one PHP file that has 10,000 lines of code and he can work. Robert [00:03:55]: But if you are two, also probably. But if you have people of five, ten, twenty, fifty people, it no longer works. Robert [00:04:05]: When some application is done as proof of concept, later it evolves to some product that is making money. Sometimes it’s missing this. Transition phase from the application that was built as proof of concept by a team of five— even if okay, if you have team of five, it’s still not a big issue to handle, even very bad code, and be able to extend that. But when the team is growing, the product is growing, the number of features is growing, well, we start to have a problem. And the question is, what you do about that. Miłosz [00:04:39]: Yeah, so if you consider splitting your project, I would start with this question: if you even need to do it. Because you often see people struggling or hear them asking, ‘How do I start I’m starting a new project, kicking off some application, and what services do I need to create? So we are probably at this stage, you don’t need to consider this at all. Right, because it’s not not the pain point, and the overhead may be big enough that it will not pay you back. Robert [00:05:14]: So you probably are mentioning the monolith first approach that we are big advocates of. We tried it multiple times for products that become bigger later, and if done properly, it’s accelerating you a lot at the beginning and allows you to change it later. Miłosz [00:05:32]: And what’s surprising to me it’s not that popular, or maybe now it’s probably a bit more popular than it was. I i see it mentioned here and there, but back back when we first tried it, I think no one talked about it that much. Robert [00:05:45]: Yeah, but I think it’s probably still some small percent. And it’s maybe also a bit because of our bubbles. Yeah, maybe. But it’s still not. Miłosz [00:05:55]: It’s not like a crazy different idea from creating microservices. Robert [00:06:00]: But if you ask a random developer, what do you hear about? Kubernetes or modular monolith? I’m pretty sure I know the statistics. Miłosz [00:06:08]: Yeah, the hype is a funny factor here. So I remember in one project—when I joined— we had one huge codebase. Miłosz [00:06:20]: Typically, that’s the software no one wants to work with, and everyone would like to see refactored. And it was also around the time of the microservices hype era. So somehow we made a decision together as a team that we will split the monolith, as probably everyone back then did. Robert [00:06:40]: Yeah, I remember how everybody was optimistic because it was like, ‘whoa,’ microservices. Finally, a solution for all of our problems. Miłosz [00:06:49]: And when I think about it now, it’s funny how we had all these issues with this project and we assumed, ‘Okay, microservices will solve this for us I mean, the project was better in the end for sure, and I was responsible for the design of the framework and all the networking and stuff. So, as a programmer, it was very fun. It was great. Probably we followed the Robert [00:07:16]: ‘fun’ principle too often. Miłosz [00:07:18]: Yeah. But looking back at it, did it need all this complexity and distributed systems stuff? Probably not. Probably some parts could be simplified. Miłosz [00:07:31]: But I think we just jumped on the hype train back then. Robert [00:07:37]: But I think a good lesson from that was that you, as a team, saw some signal that ‘okay, there is something wrong with this project,’ probably it was like, ‘okay, we cannot maintain it, we cannot add new features easily.’ Robert [00:07:51]: So the problem was visible there. Miłosz [00:07:54]: Right. So the question is whether splitting software is a good approach to fix these kind of issues. Robert [00:08:02]: But do you think that maybe what would work better there was maybe a bit more balanced approach— so maybe a bit less microservices or staying as a monolith— was better. Miłosz [00:08:16]: Maybe less microservices or modular monoliths for the start. Because the issue for sure was that the boundaries were wrong in the initial software, so there was, you know, some function calls all over the place from one place to another with no control. The more developers you add to the project, the more complicated it gets. Because this is also important factor. If you are a single person working on software like this, it will be fine probably because you can keep the context in your head. You will refactor to whatever you want. But if you have like 20 people working on it, it starts getting more complicated. Robert [00:08:57]: But still, I think it’s still manageable with modular monoliths. Miłosz [00:09:01]: Yeah, for sure. Yeah, but the issue is if you don’t have these good boundaries, you don’t know how to progress. Everyone tries to do something and it ends up as this big ball of mud. Robert [00:09:14]: So in other words, you are modifying something in one place and everything around it goes up and you have no idea why. It should have zero dependency, but it seems that it does. Miłosz [00:09:27]: Yeah, it’s risky and you don’t like making changes there. Robert [00:09:30]: So the big bow of math or spaghetti code by the right, right. Miłosz [00:09:34]: So if you have these good boundaries, as microservices, promised everything will be fine because you make these changes in one small Please. Miłosz [00:09:45]: There’s low impact, you can deploy it independently from the other services, everyone’s happy, which makes sense, but you can’t do the same thing without microservices. Robert [00:09:56]: But there’s one good thing about microservices is that it’s forcing the boundaries really hard. So it’s hard to go over network if it’s not possible. I mean, when you have a big ball of mud, you can call everything if you don’t have proper encapsulation. Okay, you can do proper encapsulation, but it requires some kind of regime of hardware. Or tooling, at least. Yes, yes. And with microservices, as long as you will not get exposed public endpoint, it’s not. So it’s probably showing why sometimes microservices may help. And have the same outcome with more cost, obviously. But again, if you go too far, it’s much harder to undo that because, for example, if you were wrong with modules in modular monolith, you can copy from one catalog to another, join everything. We have one module with microservices. You need to change your Kubernetes configs. You need to change all dependencies in other team services. Robert [00:11:02]: It’s a mess, right? Miłosz [00:11:03]: So maybe it’s easier to see the microservices as well because you can show someone, ‘Yeah, this is our Kubernetes deployments. They can see, okay, yeah.’ So this is your, those are your modules or microservices. The monolith is more complex because you need some kind of structure. It’s still possible, and we did that, but maybe not as easy to see. Robert [00:11:28]: And I think there is also one thing that microservices may make easier out of the box. So this is all sort of observability, because you can put some Istio or whatever between those services and you have it kind of out of the box again, not for free. Yeah, with much less flexibility, because Again, it’s just put in the place by some cloud ops, DevOps team, whatever. But probably the flexibility is a lot limited. Right. From other side. You can achieve the same thing with your module monolith, but it requires some effort because you need to write it in code and you need to maintain that. other side, for example, I like this approach because it’s giving you a lot of flexibility. And it’s also removing some external dependency on the team that, in my opinion, should be more about running some transparent infrastructure rather than adding something like that. But from other side, it also depends because, for example, for some big enterprises, super big products, it kind of makes sense because you can have it out of the box and have it unified everywhere. Robert [00:12:36]: So again, it depends on what kind of company we’re talking about. Miłosz [00:12:40]: Right. So microservices are kind of more tangible. You have better observability. You can see them. And modules are more abstract. You have to more understand the project to see them. Yeah. Robert [00:12:54]: But yeah, trade-offs. We are mentioning that to show that there is not only one way, because I think now the way with doing it with Kubernetes, with some service mesh, it can be a way. Definitely for some bigger companies. But again, if you’re working on a startup, maybe not— maybe you can do it by hand and it will be much more flexible, much more expensive. Miłosz [00:13:16]: But this aspect of microservices, I think, can be also misleading because it seems like you have these separate things, right? You see the deployments, you see the— you know. Miłosz [00:13:30]: different repositories. From an outsider, you can think they are separate entities, separate beings that can be changed independently of each other, but it can be a trap. I remember one project where, when we joined, they were described as independent and isolated from each other. So you can change one thing, it doesn’t impact the others. But it turned out the kind of isolation layer was HTTP API between them. So, on the outside, it looks like nice, clean separation. But in reality, they are still very tightly coupled. They just communicate over the network instead of function calls. I remember this confusion we had and trying to explain it to the management as well. Why this nicely separated system is suddenly a problem? Miłosz [00:14:34]: Because from the outside, it looks like you have those tangible, separate microservices. The reality is much different. Miłosz [00:14:43]: That’s also one of the hard parts: how to know. If the isolation is good enough. Robert [00:14:51]: Yeah, and I think it’s often outcome of, let’s say, simplistic approach to splitting those services. So I would probably see two cases. One. We’re getting database tables and we have separate service for each database table. Hooray, microservices. Miłosz [00:15:07]: I often see this when someone asks us, on Discord or whatever, for some feedback on a system they did. And very often it’s something like, ‘So I have this application. And I have users microservice and orders microservice. And at this point, I know this won’t be easy.’ Miłosz [00:15:31]: For most e-commerce systems, you can just separate users out of the orders, or it will be, you know, there will be some tight coupling probably between them anyway, as long as you’re not building some extremely big yeah, exactly. So it depends, but you know, if you take a fairly simple system, and you have those three main entities, and you think each entity should have their own microservice. Then you will probably end up with many problems because just separating those services won’t give you isolation. It will just give you this illusion of isolation. Robert [00:16:13]: Yeah, and I think it’s also nicely visible if you take your service mesh tooling, and probably will see that between some microservices will have a lot of calls going back and forth. Robert [00:16:26]: And it’s usually the sign that maybe those two services shouldn’t be separate. For example, those users and orders, those orders. So you can see that. Robert [00:16:40]: To do an update to order or to do some shipment stuff or whatever, you need to call users many, many, many times. Robert [00:16:50]: Can see that if, for example, you would need to add some field to, probably it can be customers, maybe not users in this case. So let’s say orders and customers. Commerce. And let’s imagine that orders is handing the shipment or something like that. And you can see the Skype link nicely when you have some new. For example, you need to add to user something that needs to be added to shipment. And in worst case, if it’s handled by two teams, a project that could be done in one day with modular monolith is starting to take one month because you need to go back and forth with two teams and agree on contracts and on everything. Miłosz [00:17:30]: You have to wait for the other team to pick it up for the sprint or whatever they work with. So it takes twice as long at least. Robert [00:17:38]: Yeah, yeah. And, you know, It’s often killing these small features that could be developed within one day or even if I code it now. Sometimes those features are that simple that it can be easily vibecoded. But with microservices, you no longer can because it needs to be orchestrated on a higher level. Maybe, you know, era of vibecoding will be era of modular monoliths. Hopefully modular. For sure, easier to deploy. But. And I think it’s also one important technique to mention when we are talking about these patterns, when we see that. So I would say that dark two signals when you should consider if something is wrong with us. One signal is when there’s a lot of calls between those two services to do some scenario. And the second thing is when you are modifying some functionality. Robert [00:18:31]: It requires changes in two or many services multiple times. So you see this pattern over and over. And it may be a sign that it may be wrong separation of services. Robert [00:18:41]: To solve that, there’s one technique that I think it’s very rarely mentioned because maybe it’s, I don’t know, some heresy for many people, but it’s the microservice station. So basically you’re getting those two services and joining them. Robert [00:18:56]: And I know that for many people it sounds like a heresy. I hope it will change over the time. Miłosz [00:19:02]: And yeah, it’s not never not easy, depending on how they are built. Robert [00:19:06]: But Yes, but if you are getting into problem of changing those two services over and over and two teams that needs to do that, or even within one team, you need to always change two services and kind of synchronize how you’re deploying them. It may be worth considering if it’s worth investing in joining them. Because yeah, probably to introduce some costs, they need to do some kind of cost and income. Miłosz [00:19:32]: Yeah, it’s a good thing you mentioned Teams because it’s also something I think you can’t skip when designing. Miłosz [00:19:40]: Creating software, especially if you have teams working on the same project, maybe not the same code base, but microservices that work with each other. Miłosz [00:19:53]: And if you have many of the services. I think once we worked for a company that had probably more services than developers or it was something like that, which sounds funny now, but I think it’s the reality for many companies actually. Because when it’s easy to spin up a new service, maybe you have great tooling and some platform team that handles it for you. Robert [00:20:16]: It’s interesting risk, actually, of having great tooling for easily creating microservices. I mean, if it wasn’t easy, you would think twice before creating a new microservice. Miłosz [00:20:28]: If creating that takes one minute, you end up with teams who own 10 or more. Microservices for very simple tasks sometimes I don’t know— maybe it’s fine. Sometimes, for things like keeping avatars, for example, so it’s like—overhead— but I think the most problematic story I remember from from this was where there were some services that had no clear owner, and we both our teams and other teams’ responsibilities kind of crossed in one area. That’s not really a technical issue—it’s funny, that’s like a team organizational issue. Mm-hmm. Miłosz [00:21:19]: And it’s very difficult to figure out if you didn’t start with a good design of the system first, but you just, you know, added the microservices for this entity here. This entity here— this entity here— maybe this database table sounds like a separate service, and then you have to kind of— you know— create some teams and you assign them the services. Artificially. Yeah. So that’s not a good idea. Robert [00:21:47]: And two years later, if somebody asks why those services are like that, it’s: ‘It was always like that.’ Miłosz [00:21:54]: Why you ask? Yeah. So we finally figured it out somehow, but it was very painful. Miłosz [00:22:02]: Not only because of the technical challenges, but also all the discussions between teams, company politics, stuff like that. Robert [00:22:13]: What I remember was useful there. So in this scenario, we used event storming for that to discover that. But you don’t need to use event storming. You can use any technique to see how the flow of the system is going. And I remember that it was quite funny to see that entire critical flow of the application was. So, for example, the beginning of the flow was owned by one team, later it was some small element owned by another team, it was going over again to the previous team. And it was a lot of some small pieces. Crossing the boundary. Yes, yes. And those were usually the biggest problems because if something didn’t work, it was sometimes not clear who is responsible. And it was also harder to change this law because if team didn’t have entire ownership of this law, they were just not able to do it without some cross-team coordination. Robert [00:23:09]: Well, what we did there, the solution was quite simple. So it was looking on high level on this and trying to understand how we can change ownership. Robert [00:23:19]: Historical ownership, and it helped a lot. And we added also to that a bit of the microservice station, and it helped even more. Miłosz [00:23:26]: I thought about even storming when you mentioned service mesh visualization. Miłosz [00:23:31]: So maybe instead of using this visualization after the fact, after you have the microservices in place, you could just have this high-level plan with even storming or not. Robert [00:23:45]: Seeing how system works before it works? How? Robert [00:23:49]: How is it possible? Miłosz [00:23:51]: What kind of magic is that? Miłosz [00:23:54]: Try to design it first and then see. But you know, it’s also not that easy, right? Because it’s not just a UML. Diagram where you have users, orders, and you see—okay, this is separate, but more like ‘yeah,’ maybe that’s why I would still recommend event storming because it focuses on behavior, not on the entities. So, if you map the events that happen, then the behaviors you can see there are some— those clusters of things that happen together— and it probably means they are a good place for some kind of boundary. And if you have, for example, two clusters that are joined by two events or something like that, you can clearly see, ‘okay,’ those things can be mostly separate and they won’t interfere with each other. They won’t be tied up. Robert [00:24:53]: And I think what’s also pretty good there, compared to some UML diagrams, is the context of time. And because, when you are looking at UML diagrams, you cannot see the order of flow. Even storming is more so for people that don’t know how to it looks like. So TLDR is that it’s basically some brainstorming session when you’re putting sticky notes for domain events or whatever events that are happening in the system. And you don’t need to do event-driven architecture for that later. So it’s more for the high-level modeling and how you implement that later, whatever. And what’s nice, it’s putting the in time order. Robert [00:25:32]: So basically, when you have flow, you can see when the flow is starting, when it’s ending, and it’s kind of in a magical way helping you to see how to do the split there. Anyway, I think it’s a big advantage compared to those UML diagrams or in general some diagrams. Robert [00:25:52]: Flow diagrams probably are a bit better because they also go from left to right, but I think they are not readable for very complex systems. You cannot make a flow diagram for entire system. Don’t try. But with event storming, basically you can do super big systems on higher level, but you can see the big picture of it. Miłosz [00:26:16]: So it’s a great technique if you’re just looking for the boundaries, or even if you have an existing system and you want to see how it could be done in a different way. Robert [00:26:25]: If you would like to hear a bit more about event storming, we’ve been discussing it one episode earlier, two episodes earlier, something like that. That so check it out but okay let’s come back to splitting. Miłosz [00:26:38]: So the nice benefit is of having everything mapped out on some whiteboard or you know some software. Is that you can look at it from different perspectives. For example, one thing you can do is try to differentiate core domains from the generic domains. So for example, you can see some cluster where there is more of your domain language of what your software is very specific. About, different from other software. This is probably your core part and it might be one cluster. And maybe there are some parts that are more supporting, more generic and they don’t need to be involved as much. So they don’t need to know all the details of your domain. Very often this helps you to simplify this part. For example, it can be some e-commerce software or some account. Miłosz [00:27:41]: Mounting software you can just buy, or you know, wipe code, or whatever— you don’t need to build it yourself from scratch if you don’t want to. Robert [00:27:51]: So, in other words, not all software in your product is equal. So some parts of software probably require a bit more of love. Miłosz [00:28:01]: And the less they know of each other, the cleaner your boundaries and the simpler the software as well. Miłosz [00:28:08]: So even if you build yourself this generic part, I would do it in a way that it could be reused in a different company. Miłosz [00:28:18]: It doesn’t mean you need to do it in the end, but it often helps to keep these boundaries clean and just keep some translation between them. Hmm-hmm. Of course, it can also be overdone if you try to make it super generic or use cases. So you have to be careful about that. But in general, I would try to not let the domain terms spill out of the core parts. Robert [00:28:48]: Yeah, I think it’s also some other direction that sometimes you can also watch out for with buying things because, it sometimes works that there are some services that are generic enough that can buy it, like sending emails, sometimes accounting software, but it depends. So sometimes when you are handling very specific things like marketplaces, you might not find accounting software that can handle it for you. I mean, for example, you are building some food delivery application. I guess you won’t find accounting software that will handle it for you. So it’s kind of interesting, because if you’re building this kind of application, it’s not your core domain, probably, but for other sites, it’s a very important domain. Unit also yeah— probably building a house, but parts of it will be generic, right? Miłosz [00:29:42]: Like printing invoices, that’s probably something someone solved some time ago and you don’t need to write it from scratch. Or if you do, we can keep it very generic. Robert [00:29:52]: Or maybe, for example, trying to understand what’s art. tax rate for each country is a good type. I mean, you can probably hire an entire team that is spending their entire time researching what the rates for every type of food or whatever that you can have in any countries. But probably you can just buy it from external company. And other techniques for event storming. So maybe not forever event storming, but I have also in mind one thing that you mentioned: sometimes you have parts of the system, some maybe services, maybe some libraries that multiple teams are kind of owning. And sometimes it’s on the pattern, but I think sometimes it’s also not, because sometimes it can happen. For example, from domain-driven design, we have concept of shared kernel, and it’s there for some reason. So sometimes you have some parts of code that two teams need to share, need to own. M. You can use it, but I would say that it’s rather an exception than something that you need to use by default, because if you try to use it by default, probably you’ll overuse it. Robert [00:31:02]: But I think it’s also important to keep it in mind that sometimes you have some things like that and it’s also fine. Miłosz [00:31:09]: Yeah, it’s a very important decision when you have two teams working with some similar entity or the same entity in different contexts. So first, there’s also the bounded context concept from DDD. What I think about it the most is exactly this situation when, for example, you have this user entity which can be also a customer entity in another context and, in other contexts, be something else. And you have multiple teams working with it. One naive approach would be to deduplicate and create one main user that decides about everything and has all the fields and behaviors and so on. But it that means all teams share this part and you know they need to agree on how to develop it or, from other side, if you’re one team, don’t that because it will just create too much overhead. Robert [00:32:05]: And this was something that we see often: there was one team and went too far with all those patterns. SPEAKER_2 [00:32:12]: And later, it’s like, oh, no, Domain Driven Design is making our software so complicated. Miłosz [00:32:19]: Yeah, but if you have multiple teams, you can use the bounded context concept. So you try to figure out how this user entity works. Different contexts may be separate; you know, maybe both teams can have their own user entity and they are completely separated and that’s fine. And sometimes you have this shared kernel, as you mentioned. So some parts that are so common and probably specialized. Miłosz [00:32:49]: Like you mentioned, maybe the tax rate calculation, something like that. But it doesn’t really make sense to separate that. You just agree, okay, on this small common part. We will work together and you know both teams— for example— need to agree on reviews and stuff like that on the design of it instead of the entire big system. So when I think about what’s difficult about this problem of splitting software well, I think it’s a combination of organizational and technical complexity. It’s not just one thing. Robert [00:33:23]: So sometimes I remember that there are some companies where the teams and services split and organization is somehow going from the top. Maybe it’s not from the top, but level or two levels higher than you are. Miłosz [00:33:39]: Or another way is that it became kind of the default to create microservices, and very often in a naive way. Miłosz [00:33:50]: So, for example, when I joined some project and there was some kind of migration happening from an old system to a new one. Miłosz [00:33:59]: A few people in the team assumed, let’s use microservices from the start. It’s kind of like a default approach for some people, but it can be a trap. Because you have to think, what problem do you want to solve this way, rather than just, ‘Oh yeah, it’s the industry best practice.’ Robert [00:34:19]: Worse if you have problem that you’re trying to solve, because sometimes people may think, ‘Yeah, we have problem with I don’t know scalability or ‘We had overcomplicated POC earlier and microservices will solve our problem.’ Yes. Miłosz [00:34:36]: So it’s helpful to think of microservices. Deployment patterns, something we discussed some time ago, I think. Miłosz [00:34:45]: It helps when you want independent deployments, for example, for different teams. That’s a good excuse to split software. Robert [00:34:56]: Or for example, you have that many deployments of your module or monolith that the queue is longer than you. I mean, queue is long enough that not every person is able to merge within one day. Probably it’s already designed that you can split it. Miłosz [00:35:13]: So it’s connected to teams and how you organize developers. Robert [00:35:20]: So in other words, probably we can say that microservices are maybe solving a bit of technical problems, but it’s mostly solving the human problem. Miłosz [00:35:30]: Humans working together problem. Miłosz [00:35:33]: There’s Tim Topology’s book. I’ve read some time ago. It’s great. I can recommend it. It’s specifically about this problem with the reverse Conway maneuver, I think it’s called. So basically, instead of creating a software first and then trying to assign people who work on what, you do it the opposite way. So you first design, as we mentioned, event storming, a bit similar idea. You first design the organizational structure you want to have in place, and then the software system follows that. Robert [00:36:08]: I love when such old patterns, like always low, are still working those days. Robert [00:36:14]: Even if people don’t realize, but, but I think. It’s good to know about some that something like that exists, because you can be a bit maybe more mindful when you’re doing such decisions. And it’s not like, oh, just everybody’s doing that or we’ve been doing it like that for ages, why we should change it. Yeah. Miłosz [00:36:36]: It’s also great that we are past hype. Era now and many people talk mainly about how complex microservices can be. So now it’s probably we are more balanced. Robert [00:36:47]: Probably won’t you won’t be put on the stake after saying that microservices is not the only way. Robert [00:36:55]: Maybe in 10 ways we’ll be again in the place when everybody is using monoliths. Probably not modular, something tells me. Probably it will be still a problem. Miłosz [00:37:06]: I think, as an ending note, we can recommend starting with monoliths if you are just getting started, especially if you are just a single developer or a small team. Yeah, I hope we should. And do it the modular monolith way. Robert [00:37:20]: Yeah, I hope that we’ve shown that there are alternatives and it’s really not that hard to do things in those alternative ways. Robert [00:37:30]: Definitely will not finish with this episode about modularization because it’s a topic that requires a lot of attention, in my opinion, and it can give a lot back— i mean, when you are doing it properly, it can really accelerate your work a lot, yeah, especially with modular monolith. Miłosz [00:37:49]: Because I think what we’ve figured out works pretty well. Robert [00:37:54]: And we’ve seen it work with teams. And we also see that it’s not that popular approach. So I think it may be also some unique skill to have. Miłosz [00:38:04]: Yeah. And if you’d like to see a real example of this, we have a domain engineer training coming up and we will have an example of a pretty complex application with different modules built as a modular monolith. And we mentioned parts of it in this episode as well. Yeah, I think it will be cool. So we’ll leave a link in the description if you want to learn more. So we shared our stories with Spitting Software. Let us know about yours so we can also learn something and maybe help you figure out something. Robert [00:38:40]: Yeah. So we’re here to help you. So this podcast is for you. And yeah, if you will be happy to answer in comments and maybe include something in the next episode as well. Miłosz [00:38:53]: Yeah, if you have any topics you would like us to cover, let us know. Miłosz [00:38:58]: See you in the next episode. See you next time. Bye. Bye-bye.

Quick takeaways Complexity comes from two extremes - projects fail both when they’re overcomplicated with unnecessary patterns and when they’re oversimplified for a complex domain Essential vs accidental complexity - essential complexity comes from the domain itself and can’t be removed, accidental complexity is created by poor implementation choices “Keep it simple” is lazy advice - achieving simplicity takes effort; closing your eyes to complexity just pushes it elsewhere Match patterns to the problem - using the same approach everywhere is a red flag; mix simple solutions for simple parts and sophisticated patterns for complex domains Ship fast and iterate - if you can’t deploy daily and fear making changes, something is wrong regardless of whether it’s over or under-engineered Introduction We discuss where complexity in software projects comes from and how to deal with it. There are two common extremes: projects that are overcomplicated because someone applied patterns they saw at a conference without understanding them deeply, and projects that started simple but became a tangled mess as they grew. Both can be equally difficult to work with. The key insight is understanding the difference between essential complexity (inherent to your domain) and accidental complexity (created by your implementation choices). We share diagnostic signals to identify unhealthy projects and practical advice on matching your tools to the actual problem you’re solving. Notes No Silver Bullet paper by Fred Brooks - discusses essential vs accidental complexity Domain-Driven Design (DDD) - a pattern for tackling complex domains Clean Architecture - another approach for managing complexity YAGNI (You Aren’t Gonna Need It) - principle against premature generalization Defensive programming - validating inputs early to prevent errors propagating Canary releases and rollbacks - deployment strategies that can add overhead Three Dots Labs blog - articles on Go patterns and architecture Go with the Domain ebook - free ebook about DDD in Go with 60,000+ downloads Wild Workouts example DDD project - complex Go project demonstrating real-world patterns Microservices architecture - discussed as sometimes adding unnecessary complexity CRUD applications - simple approach that works for some domains but not others The Domain Engineer training - upcoming training mentioned for early next year Quotes On one hand, you have someone who says we need to support 10 million users a day—this is one extreme— but on the other hand, you also have people who say, ‘just keep it simple and it will be fine’ and it sounds good. But to me, it’s one of the lazy advice. It’s easy to say to tell someone, ‘keep it simple but it’s not that easy in the end. Miłosz The complexity doesn’t go anywhere, no matter how you implement it. Miłosz If your team is unhappy working with the project. If the team brings up plan for migration or rewrite every other day, that’s a solid sign something went very wrong. You’re probably on one extreme already and it’ll be hard to get out of it. Miłosz If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite. Robert So you cannot close your eyes and say that this complexity doesn’t exist. It will pop up somewhere else and it will be much more expensive to handle at the end. Robert Don’t be dogmatic about using any tools because it’s never ending up properly. Robert Timestamps 00:00:00 - Introduction 00:01:21 - Two sources of complexity 00:05:14 - Complex vs complicated 00:09:20 - Essential vs accidental complexity 00:14:00 - The trap of generic solutions 00:19:15 - Complexity doesn't disappear 00:23:05 - Patterns and defensive programming 00:26:58 - Red flags in projects 00:32:05 - Avoiding premature optimization 00:35:12 - Don't be dogmatic 00:37:40 - Practice and resources Transcript Miłosz [00:00:00]: On one hand, you have someone who says we need to support 10 million users a day—this is one extreme— but on the other hand, you also have people who say, ‘just keep it simple and it will be fine’ and it sounds good. But to me, it’s one of the lazy advice. It’s easy to say to tell someone, ‘keep it simple but it’s not that easy in the end. Robert [00:00:18]: If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite. Miłosz [00:00:25]: It takes a lot of effort to have simplicity in the project. Robert [00:00:28]: Your eyes and say that this complexity doesn’t exist, it will pop up somewhere else and it will be much more expensive to handle at the end. It’s a very important concept to understand the problem. Accidental complexity and essential complexity. Accidental complexity is created by poor implementation. If you approach the implementation of your application in an overly complicated way, so we’ll put just some patterns that are unnecessary, you will have accidental complexity. On the other hand, if your project will be, for example, this complex financial product and you will try to build as a CRUD, you’ll end up with a lot of accidental complexity there. Miłosz [00:01:05]: Yeah, but you can reduce it if you use some patterns that play well with the domain. So you can reflect it in the code in a more natural way. Then it’s easier to work with. So the accidental complexity goes down. Robert [00:01:21]: I’ m Robert. Robert [00:01:21]: And I’m Miłosz. Robert [00:01:22]: And today we’ll talk about complexity in projects and from where this complexity is coming. And from my experience, there are probably two types or two sources from where it can come. So the first one. Probably. Many of you can relate to that. It’s the situation when you are joining some project and there is some team leader or some very senior engineer. And this engineer is dictating, let’s say, some good practices and some patterns in this project. Robert [00:01:55]: You can clearly see that those patterns are not adding a lot to this project. Actually, it’s adding a lot of unnecessary complexity. Often, this person is getting those techniques from some conference, maybe, or from some video, or maybe this person read some article. Later, this person is forcing everybody to use those patterns. Miłosz [00:02:15]: So it’s this ivory tower architect who dictates from the top how to do things and the solutions are already invented. Robert [00:02:26]: Yeah, so it’s often the problem. It doesn’t need to be architect because it’s sometimes maybe, again, some more senior person. But from my perspective, very often the problem in this situation is that this person doesn’t really understand those patterns very deeply. Robert [00:02:45]: As I said earlier. This person has seen those patterns in one video or maybe one talk and knows how it looks like. So you are going to conference, there is some idea, it looks great on the paper. In practice, Well, there are a lot of complexities and it doesn’t work as simple as it was thought on the conference. Miłosz [00:03:04]: And being an engineer on this team, maybe peer of this person, you will probably feel like this project is overengineered, right? There is too much patterns and it seems overcomplicated. It’s very difficult to get something done. Miłosz [00:03:22]: Because there’s so much work involved. Robert [00:03:24]: Definitely. And I think for many people, it will be the main source of complexity in projects, but maybe you have some other idea what can be another issue that can happen in projects. Miłosz [00:03:35]: So on the other hand, I can remember other types of projects where everything seems very simple, but at some point it’s very difficult to continue development. Miłosz [00:03:50]: Project lifetime, it’s sufficient to have some very basic patterns in place. Maybe no patterns at all. Robert [00:03:58]: If you have small enough application, maybe you can just do only one file and sometimes it’s starting as some proof of concept or maybe side projects and because of some lack reason this is becoming pretty successful project at the end. And it’s becoming also the problem because it started to be it started as a proof of concept. And at the end, it it became a successful product. Miłosz [00:04:22]: These are fun. You can just add new endpoints, add new database tables. Miłosz [00:04:30]: Works fine, but then that team grows and at some point it starts to be very difficult to extend this project because you know everything is coupled together when you change something you don’t know what else is going to change and usually at this time you you might be long for some patterns actually to be used in place and I started researching some architecture, how to approach this thing. It has two extremes, right? One is overcomplicated, the other is no patterns at all and both can be equally annoying to work with, difficult to develop. Miłosz [00:05:14]: Thank you. Robert [00:05:14]: In this episode, we’ll cover a bit more. How to understand if your project is in the bucket of overcomplicated by using too many patterns or maybe oversimplified. So the problem that you are solving or the product that you are working with is already pretty complex and building is a crud with just frameworks. It may not work anymore. And I think it’s a very important skill that many people is missing. So I think many people are stopping on the side that, yeah, this project is very overcomplicated because we use too many patterns but it’s not only the case yeah so it’s a similar issue but Okay. Miłosz [00:06:01]: two extremes, right? But the root cause is kind of the same. It’s using wrong tools to the wrong problem. Or you don’t know what the problem is yet and you use only what you know. Robert [00:06:14]: Bye. It’s not only about the complexity of the project but sometimes it’s also the scalability. So we’ll also cover a bit about this today, because also you probably heard many stories like: ‘This application will have millions of users.’ At the end, we have 10 users. And it’s hard to add anything more because you prepare your application for big traffic. Miłosz [00:06:35]: And because of that, it’s very, very complex. On one hand, you have someone who says, ‘we need to support 10 million users a day,’ which is one extreme. other hand, you also have people who say, ‘keep it simple and it will be fine and it sounds good but to me it’s one of the lazy advice. Miłosz [00:06:55]: It’s easy to say, to tell someone, ‘keep it simple.’ But it’s not that easy in the end. Robert [00:07:03]: If just keeping projects simple would be that easy, all projects would be great. But unfortunately, it’s the opposite. It takes a lot of effort to achieve simplicity in the project. If you’re joining a new project and you notice that this project is complex. So in this case, do you wonder if it’s complex because of some patterns that make it overly complicated? Or maybe this project has some kind of baseline complexity that made this project complex. Miłosz [00:07:33]: I think it’s very different. Tell the difference when you just joined the project— actually. You might know that it is difficult to work with, but you might not yet know what it solves, what’s the business domain. Miłosz [00:07:47]: So it might be not that easy to tell if it’s over-engineered or it’s just inherent domain complexity that’s not going anywhere. Robert [00:07:58]: And I think what may be pretty useful here is distinction between complex and complicated, because I think those words are nicely encapsulating the different characteristics of this. So when we have a project that is complex, so it’s a project that has, for example, some complex domain like accounting, complex e-commerce, or is pretty old product supporting many use cases. So in this case, it’s complex, but it doesn’t need to be complicated. So, if somebody will not approach it properly, it may be complicated, but I think what’s also pretty interesting is that some projects are not complex, but they are just complicated. And in this case, it’s the scenario when, for example, this team leader went from a conference and just read some blog article, started to copy some patterns, and just unnecessarily complicated that. For the sake of it, instead of solving something. Miłosz [00:08:53]: Exactly. Yeah, but it might be hard to tell the difference if you are not very deep into the project. I remember working with some MVPs that looked complicated on the surface. And then you start rebuilding them or trying to understand what’s going on. And you see that, okay, this is not actually technical complexity. This is how the business works. So that’s why someone did this this way. Robert [00:09:20]: So some examples of projects that are complex, but maybe not complicated, are for some financial-related applications. So for example, some accounting systems that are supporting multiple countries and even hundreds of countries, different tax rules that can be different for multiple countries and some domain. May have some special requirements, like for example, taxi or something like that. And this kind of projects are very big. Baseline complexity. And by the way, we’re currently working actually on a training that we’ll be releasing at the beginning of the next year when we’ll be showing how to build such very similar domain actually. So you can see the description. Yeah, so check the domain engineer in Google. So you will find it. But what I think it’s pretty interesting in this kind of projects is that if you, at the first site, look on all these accounting rules and support of multiple countries, you might have a feeling that it’s super complicated, but actually depends how you approach implementation of that. Robert [00:10:37]: So no matter what you do, in the implementation part, it will be complex because it’s just a complicated domain. Miłosz [00:10:46]: The complexity doesn’t go anywhere, no matter how you implement it. Robert [00:10:49]: And often this is named as essential complexity. This is the complexity that is kind of created by the domain. And the opposite of that is. Accidental complexity. So accidental complexity is complexity that is created by engineers, basically. So when you are, for example, preparing your application for global scalability, millions of users. So this is where you are creating the accidental complexity. Miłosz [00:11:16]: Yeah. I think it comes from the No Silver Bullet paper from Fred Brooks, which shares the name of our podcast. It’s actually a good read. Robert [00:11:24]: What a cool incident. Miłosz [00:11:26]: Check it out. It goes into detail on essential versus accidental. But maybe we can just quickly go over it. Hmm, hmm. Miłosz [00:11:37]: Hmm. Robert [00:11:37]: Also, because I think it’s very important to understand that concept to understand the entire problem of accidental complexity and essential complexity. So basically, the accidental complexity is created by poor implementation. Because if you approach the implementation of your application in an overly complicated way, so you’ll put just some patterns that are unnecessary, you will have accidental complexity. But on the other hand, if your project will be, for example, this complex financial product and you will try to build as a CRUD, you’ll also end up with a lot of accidental complexity there. Miłosz [00:12:16]: Yeah, but you can reduce it if you use some patterns that play well with the domain. So you can reflect it in the code in a more natural way. Then it’s easier to work with. So the accidental complexity goes down. Robert [00:12:31]: Unfortunately, if you do the opposite— so if you have a complex problem to solve and you try to approach it in an oversimplified way— it will add. So we will have the base complexity, so the essential complexity, and you will have also accidental complexity. So this product probably will be a great mess that nobody would like to touch. Miłosz [00:12:51]: And suddenly, we developers often bring this on ourselves, focusing on the technical details, trying to build a framework that will capture cases in the future, stuff like that, which all can add up to extensive complexity before you need all the features. Robert [00:13:11]: It’s starting with a solution that is generic from day one, and it’s basically creating a solution that is generic. Always much more complicated than creating a solution for one use case. I have some examples when we’ve been adding some functionality and some teammates were proposing like, ‘Oh, let’s maybe create a library to do that.’ You will not need to do it in the future. But after trying to design that, we found that actually making it in a generic way is a couple times more complicated than just doing it in one way for one scenario. And sometimes it’s even easier to just copy some logic and have a bit different implementations if it doesn’t change that often in two places and independently. Miłosz [00:14:00]: Because again, having this generic solution be just very expensive, yeah, so you need to be very conscious of where to use the generic solution, where to when to abstract away the details, and for for what system you need, what will approach. Robert [00:14:18]: Let’s go back again to the problem of complex projects, because I think it’s also interesting that if you are working with these more complex projects, sometimes some people from other teams or some new joiners may look at it and think, ‘Oh, guys, you’re overcomplicated that so much. You’re using so many patterns and it’s hard to go over this code and understand how it’s working. It may look like this at the beginning when you are working with some more complicated domains. But people may sometimes miss the context here. And sometimes you may be in a rescue project, basically, that earlier failed. Because it was created as a CRUD, you are trying to save this project and implement it in a way that you can tackle this. Robert [00:15:10]: Essential complexity. And some people looking at that later may be missing this context here. Miłosz [00:15:22]: Overengineering. Yes. Yeah, but on the other hand, it’s also the opposite issue, right? If you oversimplify things from the beginning, it may be too late to change the architecture. Robert [00:15:38]: And I think it’s also sometimes interesting when, for example, the team was changed and most people left this team. Robert [00:15:47]: If it was a complex project, not overcomplicated, but the person that was trying to change it later. May have in mind that, oh, okay, it’s so overcomplicated, let’s try to rewrite it. So this person or this thing is spending like their half year or year to rewrite that to just realize at the end that basically, all the complicated or the complex, not complicated stuff is again there. Because when you start to support more and more use cases and edge cases, you’ll see that actually all those checks and all those complex parts were necessary there so yeah and they don’t go away when you change the approach. Miłosz [00:16:31]: Even if you change the technical approach, the underlying domain complexity doesn’t disappear. All right. And. If you’re like a developer focused on simple code, if you like simple code, you might be annoyed that there’s new requirements coming in. Miłosz [00:16:49]: I know you need to change the factor to capture it, but it’s just the reality. You need to support the business this way. And however you model it, the complexity is there. And if you try to avoid patterns at all costs and just do it simple, you may end up down the road with some weird solutions, like for example, instead of having a good transaction boundary where you need it, you start running cron . checks for consistency. In your system and report some errors, and for some teams this is like the business as usual approach. Let’s just fix it afterwards, but that’s so much— you know— with operating efforts. You are bombarded with alerts all the time. Miłosz [00:17:52]: You need to react and fix the system. Miłosz [00:17:57]: After the fact, instead of just designing in the first place in a way that it’s resilient and I think it doesn’t need to be. Robert [00:18:04]: We don’t need to reuse any advanced patterns to avoid that. So sometimes it’s basically just some defensive programming. So let’s take an example when we’re building this accounting system, for example, and you are not validating if, for example, the amounts are positive or non-zero. And you can accept any value and issue thousands of invoices that have a total value of zero. And. Okay, you can later clean it up. You maybe have some chrome that will void them or do whatever with that. But basically, it will be much cheaper to do this defensive programming approach earlier and ensure that it will not propagate further. It’s not only basically about uh operating overhead from side of your team, but it may be also some financial team seeing later that, oh, no, we have thousands of invoices that we need to do something about that. And those developers again. Miłosz [00:19:09]: Right. So you only push around the complexity this way instead of solving it. Robert [00:19:15]: So you cannot close your eyes and say that this complexity doesn’t exist. It will pop up somewhere else and it will be much more expensive to handle at the end. Miłosz [00:19:24]: Some analogy that comes to my mind is like if you decided you won’t have files bigger than 100 lines in your project. And you will have simple files, right? Small files. Sure, it’s great, but then you end up with thousands of them. And you push the complexity to the number of files instead of the length of the file. Miłosz [00:19:47]: Back in the day, there was some big question: how big should be a microservice, and some people suggested metrics like this. This comes to my mind. Miłosz [00:20:00]: Let’s build our huge application into really tiny services. So you solve this complexity of big code base, but now you deal with thousands of code bases. Robert [00:20:15]: So basically, divide and conquer doesn’t always work with software. It can work, but you need to know basically how to split them. I think it may be also an interesting topic for some other episode because, Well, not a surprise. There are some patterns and ways how to detect that. And spoiler alert, it’s not as small service as you can because, at the end, you can just end up with big mess and some sagas that you need to orchestrate some really small part of the flow. But I think it’s enough for today. Miłosz [00:20:51]: The problem here is you think the size of the code base is the issue, right? It’s similar to what we discussed before. The complexity is in the domain, so you can’t just change this. If you have a complex business and it needs a big application to work, this complexity will be there, however you model it. So you just need to make effort to model it in a way that’s best to reflect how it works and for teams to work with it. Robert [00:21:20]: And I think it may be misleading sometimes, because you may feel that splitting this big problem into smaller microservices can actually make this problem simpler, easier to tackle. On the other side, sometimes you may wonder. How? Robert [00:21:38]: Some application can have, I don’t know, 5000 of developers for running and developing such simple application. I know that it’s sometimes hard to judge an application from outside because you may not know all the complexity inside. But come on, thousands of developers— it’s a ton of people. Splitting it into hundreds of microservices can create an illusion that you just cut this problem into smaller and it’s easier. But there are cases when, basically, this application could be one service with 50 times less developers developing it. Miłosz [00:22:17]: Maybe you won’t need a whole platform team then to manage all this complexity and communication between them. Robert [00:22:25]: And if you add the extreme scalability requirements that are, I think, also often out of nowhere, because really, you can probably handle Twitter-like scale on a single VM. Just a reminder that you can have VMs that have one terabyte of RAM, 96 cores. On GCP, it’s even 150 cores per one machine. So come on, you can really handle Twitter scale on this one VM. Miłosz [00:22:54]: RAM is expensive now, so I don’t want to do that. Miłosz [00:22:58]: So coming back to the essential versus accidental complexity, what is the solution here? Robert [00:23:05]: So I think for the case with cron. Again, some simple solutions like defensive programming and very very very very very very defensive programming. So you have any input that this not as expected, don’t accept that. Don’t allow to propagate it further. And the other thing, when you are splitting problems into smaller, also watch out to not split it too granularly. Other thing that is coming into my mind is trying to tackle this essential complexity instead of always splitting that. Because there are some patterns that are helping to tackle that. In, for example, this one service, it can be clean architecture, it can be domain-driven design. Thank you. Very important disclaimer. I’m not saying about some dogmatic clean architecture or domain-driven design because it’s very easy to go into caricature of those techniques instead of some pragmatic approach. So don’t be this person that you see in one video or one presentation or one article. Robert [00:24:08]: Dig a bit deeper, try to understand what problems those techniques are solving, what problems they are not solving, when to not use them, and apply them in places where you have the essential complexity to not add more accidental complexity. Robert [00:24:27]: Sounds like a plan. Miłosz [00:24:28]: Yeah, that’s something I would recommend to consider where in the project you use what approach. If there’s one approach you use in your entire project everywhere, that’s probably an amber flag. Probably something is weird because The most projects don’t have only you know, only complex or only simple domains. There will be a mix of some very generic parts that, you know, we can use something simpler for it. And the core domain where you need your most sophisticated patterns. Robert [00:25:06]: Definitely. And also when you are using some patterns that you read about, you’re trying to apply that. Also, if you feel that it doesn’t fit, don’t try to push it by force because probably it may not be fitting there or maybe you don’t understand something about this pattern. Miłosz [00:25:21]: Sometimes you can look back after implementation and I remember we did it many times. Uh-huh. Uh, which created some domain layer and try to model something in it. And then we look at it and, hey, it’s like mostly boilerplate. Maybe we can just simplify it because there is nothing it gives us, no benefit. Robert [00:25:44]: It’s basically a crud. So we just can have one struct getting it from API, put to the database and it’s fine. Miłosz [00:25:51]: We have a mix of these parts of complex domains, complex patterns and some very simple ones. I think it’s a healthy sign of a codebase in general. Robert [00:26:01]: So watch out for being dogmatic. So it’s. Robert [00:26:05]: It’s coding, it’s not religion. Robert [00:26:09]: Many people will probably disagree, but it’s reality. Robert [00:26:13]: And I think it would be useful to maybe give you some advice from our experience. Robert [00:26:20]: What are some, let’s name it, diagnostic signals to understand if your project may be over complicated or oversimplified? Because I think we can probably find a couple of signals that may suggest that there is something wrong with this project. Well, I think it’s pretty interesting is the fact that those signals may be the same for projects that are oversimplified and overcomplicated. So you may remember that at the beginning we said that it’s the same problem, but from another side, basically. Robert [00:26:58]: It’s about half of the codebase in general. So what would be, from maybe your perspective, some red flags that the project is unhealthy? So it may be overcomplicated or oversimplified. Miłosz [00:27:11]: Probably the king of all would be if you can ship the project fast and do it over time, iterate often, and maintain it— that’s a good sign. If you can’t do it, something is wrong. So it’s like a big number one rule. Robert [00:27:33]: Some anti-examples here, maybe some old enterprises that are deploying. Every half year, for example. Miłosz [00:27:40]: Yeah, like the other extreme. Yes. For example, you need to hand over your deployment to someone who does it during the night because it’s, I don’t know, risky. Robert [00:27:56]: And you can also do it daily because it’s the scenario when you have an essential with accidental complexity. So it’s a very old project that has already a lot of edge cases to handle. But it’s also overcomplicated because maybe in some places it’s not planned strategically properly. Miłosz [00:28:15]: Yeah. Miłosz [00:28:17]: So if you can deploy daily, that’s probably one red flag. If you fear merging changes or just changing some areas of the project, because you don’t know what can happen. That’s another red flag, I would say. Whatever the reason, you should be comfortable deploying and managing stuff. Robert [00:28:40]: So in other words, if your project is crude and you are able to merge it daily or a couple of times a day, and you are not afraid to introduce new changes, it’s totally fine. It’s probably enough. But for example, when you have some complex domain and you are not able to do that. Well, you may have some excuse that, oh, it’s so complicated. We are working with money. We cannot merge every day. But yeah, there it is. The reality is that if you tackle this accidental complexity, if you handle this essential complexity properly, you can even deploy multiple times a day application. Pretty complex and is doing some risky operations. Miłosz [00:29:21]: Yeah, if you are working with money, I hope you just don’t just you know cross your fingers and deploy. But you are confident this will work. If you, if you’re not, then probably something is wrong. Maybe the project is oversimplified. Robert [00:29:37]: Other red flag that I have in my mind is when you have some new joiner in your team and it’s taking weeks to onboard this person. So it’s kind of connected. Previous points because, if you aren’t able to deploy daily, you have fear of touching anything in your project. The same will also apply to new joiners and even more, because you’re a new joiner. You see that this project is complex, and/ or people warn you or don’t touch this part. Miłosz [00:30:09]: You don’t want to see this. Miłosz [00:30:14]: I’ve seen projects like this. Robert [00:30:16]: Exactly, exactly. And in a healthy project, I would say that it should be quite safe to fail, basically. And it’s connected to defensive programming. Basically, and also having some good infrastructure in place that is maybe doing some Canary releases and rollbacks, but it can be also the opposite. For example, if you don’t have application with complex domain, and making mistake is not that big problem, but you have very complicated infrastructure for rollbacks, for Canary releases. It can add a lot of overhead for your deploying pipeline and, in this case, it even cannot deploy the simple application daily because the process of deployment is too heavy compared to what’s required for this application. Miłosz [00:31:05]: Also, if your team is unhappy working with the project. If the team brings up plan for migration or rewrite every other day, that’s a solid sign something went very wrong. You’re probably on one extreme already and it’ll be hard to get out of it. Robert [00:31:28]: I think many people may say, like, ‘projects like that don’t exist.’ I think it’s a nice excuse, but it’s not the case. Such projects exist that you can really deploy daily, even if it’s not very simple projects. Robert [00:31:46]: So, to maybe recap. Robert [00:31:48]: Those indicators a bit. So I think there are some questions that you may ask yourself. So the first one is asking yourself, Can you remove some stuff without fear that you’ll break everything, basically? Robert [00:32:05]: And when you’re thinking about those red flags, remember, it’s not just because some patterns that some people use sometimes, it may be because premature optimizations, like preparing for scale that will never happen. Miłosz [00:32:21]: Yeah, that’s a lot of accidental complexity there. Miłosz [00:32:25]: Or focusing on performance before you need it. So instead of choosing something more explicit and easier to grasp, you choose something very optimized, but not really needed at the time. Robert [00:32:41]: And sometimes it may be also preparation for the future—like, maybe at some point we’ll need X. Let’s do it now, so it will be easier. Unfortunately, usually it’s a trap. Usually it will never happen, or if this functionality will be needed, if we will need to add this functionality, we’ll find out that actually what we prepared earlier no longer works in this way. And we spent a lot of time on writing that and maintaining it. So, jagni. Miłosz [00:33:10]: Yes, it’s easier to get things that are easy to remove or replace instead of looking for all the possible edge cases. Robert [00:33:19]: Always have in mind that most of the techniques, tools, or whatever you are using, you’re using it to make your development lifecycle as efficient as you can. So, if you see that but something is making your development lifecycle worse, slower, and it’s slowing you down, you should maybe rethink if it’s needed or you’re applying it in the right way, or maybe it’s not needed at all. And at the end, you should aim to making some small incremental changes. And I know that it sounds easier than it is, it’s easier to say than do. But it’s a matter of experience and learning, learning and learning, and trying to be as close to this go to deploy multiple times a day. And also, watch out to not go into the trap that we already invested couple of months into this feature to make it super scalable. It doesn’t matter that we need to spend now weeks on maintaining that. Robert [00:34:21]: Watch out for this trap. Don’t be afraid to remove some stuff that is slowing you down. Robert [00:34:29]: Yes. So it’s important to be aware of that. So if you already invested a lot of time into something, it’s harder to work some framework. Some generic library trying to do everything. Some stocks. But it’s showing it similarly. Like you invested in something. Company and you see that it went down, and you are waiting years to recover. And probably just saying that and buying some better stock can help more. By the way, not investing advice. Robert [00:35:03]: I was about to say that, yeah. Robert [00:35:07]: And at the end, I think it’s important to emphasize one thing. Robert [00:35:12]: Don’t be dogmatic about using any tools because it’s never ending up properly. Robert [00:35:22]: Have an analogy of some worker that is, I don’t know, big fan of pneumatic hammer, and this person is just using pneumatic hammer everywhere. And it may work if this person is demolishing buildings, but if you need to nail nails, well, probably using nomadic armor for that. At best, you’ll hurt yourself. Robert [00:35:44]: But in this case, it would be just more practical to use the normal hammer. But from the opposite side, maybe there is some fun in using a normal hammer. Saying that a normal hammer is great and it’s simple— it doesn’t need any other energy source. You can just use it everywhere. But it’s true. However, if you need to demolish a wall, it will take ages. Miłosz [00:36:07]: For example, something that we mentioned in a previous episode about having a toolbox. You can choose your tools basically for the job. Robert [00:36:16]: And even if it sounds, maybe, very obvious. From other side, we’ve met a lot of people that were like these people with simple hammer or pneumatic hammer used for everything, and even if it sounds that obvious, they were doing it in this way. Miłosz [00:36:38]: So it helps if you ask yourself, why are you doing this? Why use this tool in the first place? If you want to solve something specific, for example, if you have a complex domain and that you want to model better, that’s probably a good reason to use something. Or if you kick off a project that you know is supposed to be a big one, you probably want to have some idea of the architecture of the design beforehand. But on the other hand, if you just kick off an MVP, you don’t need all this stuff. You can just start simple and grow. Miłosz [00:37:23]: and yeah watch out for extremes, what we mentioned before. So if you are far on the over complicated or over simplified spectrum, the project probably will be difficult to work with. Robert [00:37:40]: And the thing that we can probably recommend most, and we are saying it over and over, but play with technics sometimes in work, sometimes after work, also. So if you would like to be good at what you are doing, it’s it’s also good to practice this outside of work, because you’re also sometimes limited by the projects or maybe sometimes you’re limited by some dogmatic people that, for example, like oversimplified code and you cannot do much about that, but you can always experiment a bit in in the home. So one thing that we can probably recommend is checking our blog tree . tech and where we have a lot of articles that are totally free. We also created one pretty big Go project that you can play with and try to apply. Look how we applied some patterns there. And you can also try to play with this project and add some of your features there. And you can gain this muscle memory on how to use those patterns and later, when you will encounter some problem, you’ll have some tools in your toolbox and you’ll be able to experiment with that. Miłosz [00:38:51]: So we tried to create more complex examples, so you can see the ‘why’ behind the patterns, not just some toy projects where they don’t make sense. Robert [00:39:01]: Yeah, because if you have sometimes those dogmatic people about using some patterns, you can notice that very often they have very shallow knowledge because they’ve been reading some article. That again, are very shallow. So it’s like five minutes read. And they are thinking that they know everything about some technique after doing that, or doing it differently. So our articles are not simplifying things, but there are—pretty long, but they are long enough to prepare you for applying some patterns in the wild. There are some good articles out there. It’s not like only our articles are good, but very often there are some articles that are very, very shallow. And when you look on them, you can just see that applying what’s suggested there will introduce more troubles and this accidental complexity without solving any problem actually. So we would have your base complexity with accidental complexity, and at the end it will be how to work with. Miłosz [00:40:01]: So we also have the Go with the Domain ebook about working with domain-driven design in Go. And it shows refactoring of a bigger codebase. Miłosz [00:40:14]: This is not a toy project, but more like a real-world scenario. Robert [00:40:19]: And it’s, I think, already pretty popular. So last time when I was checking it, it had over 60,000 downloads. And we also received a lot of emails from people that it helped them a lot in the project. So for some people, it’s quite useful. Robert [00:40:34]: Check it out. It’s also free by the way, so just google go with the domain and I’m sure you will find that Cool. So I think that will be all for today. So thank you Miłosz. Thank you Robert. See you next time. That’s it.

Quick takeaways Domain complexity matters more than technical complexity - Most projects fail not because of technical challenges, but because they don’t handle the business domain well. DDD is a toolbox, not a religion - You don’t need to use every pattern from Domain-Driven Design. Pick what solves your actual problems. Start with the domain model - Understanding how the business works is more important than designing the perfect schema. Avoid solving imaginary problems - Spending months on frameworks or platforms before building actual features often leads to wasted effort. Strategic patterns are essential for everyone - Even if you don’t use tactical DDD patterns, thinking about core domains and module boundaries matters in every project. Introduction In this episode, we discuss why software projects become legacy code that nobody wants to touch. We talk about how Domain-Driven Design can help, but also why it’s often misunderstood or overused. Instead of treating DDD as an all-or-nothing approach, we suggest to use the ideas pragmatically - picking the patterns that solve real problems in your project. We share stories to show how focusing on domain complexity rather than technical complexity leads to better software. Notes Go With the Domain ebook - our guide to applying DDD ideas in Go The Domain Engineer training - launching early 2026 Domain-Driven Design (The Blue Book) by Eric Evans - the original book that introduced DDD almost 25 years ago Design Patterns book by Gang of Four - classic software design patterns book Event Storming - collaborative workshop technique for understanding domains (covered in the previous episode Quotes The bold assumption for today’s episode is that you can go deep into many of the ideas and they shine in complex projects, but very often you can use some more basic version of the key ideas and apply it to most projects actually. Miłosz If you are hiring specialists in many fields, the sentence ‘if you have hammer in your hand, you will see nails everywhere’ may happen. They will spend a ton of time on solving problems that may be not the problem in this company yet. Robert If you consider most of applications, like SaaS apps, they are usually very similar. If you change jobs, you can end up at another completely different company and use the same tech stack. It’s not much different to use the same programming language. But on the other hand, what the company does can be completely different, and this is the differentiating factor between them. Miłosz The worst thing about that is sunk cost fallacy. So you’re spending more and more time on maintaining that, losing a lot of resources, time, and nerves on maintaining that. But it’s no longer needed. Robert We spent many hours creating this messaging framework using generics. And in the end, we had this good interface that you can very easily publish an event, very easily subscribe to it. And the result was that it was probably the only game jam we never finished. Miłosz You should use proper technique for a proper problem. That’s it. Robert If you boil DDD down to one primary theme, I would say something like it’s about understanding the domain you work with and then modeling it well in the code. Whether you call it DDD or not, it’s like a mindset. Miłosz It’s kind of forcing you to think, ‘Okay, is this code really important to my business, or is it less important?’ And it can kind of affect how we are thinking about this code. Robert Timestamps 00:00:37 - Introduction 00:04:18 - Technical vs domain complexity 00:12:13 - Technology choices 00:16:43 - Framework trap 00:21:12 - What is DDD? 00:31:36 - Core theme of DDD 00:38:22 - Context matters 00:43:01 - Domain model 00:52:00 - Wrap up Transcript Miłosz [00:00:37]: Hello, I am Miłosz. Robert [00:00:38]: And I’m Robert. And this is No Silver Bullet Podcast, where we discuss mindful backend engineering. In this show, we share takes on software engineering that will help you to grow into principal engineer level. Miłosz [00:00:51]: Let’s talk about the usual arc of a software project. Miłosz [00:00:55]: So first we have the kickoff, and now we have high hopes for good design and this product will be great and everything is fine. This time we’ll do it. Right. Yeah. And then we work on it for some time. Then we deploy it to production. Everything is great. And suddenly it becomes a legacy project no one wants to touch. Did you ever see this happen? Robert [00:01:20]: Every time. Okay, not every time, but yeah, it’s very often the case. Miłosz [00:01:27]: So it’s interesting, right? Why does it happen? Why every project ends up as this big ball of mud no one wants to work with. Miłosz [00:01:40]: And the first version works quite well. We come up with something, it’s pretty much what we wanted, that’s fine. Miłosz [00:01:48]: And then new features come in and very often they are quite different than what the software did in the first place. Right? Miłosz [00:01:59]: So we need to change the design. But very often there is not enough time or developers are lazy or they don’t feel like it, whatever. But they don’t remodel everything to fit the new design. Instead, we add some workarounds or hacks and it works pretty well. Miłosz [00:02:20]: For a while, except that maybe some bugs, maybe some unusual behaviors. And we repeat it over and over, over months or years. And then we arrive at the state that we are afraid to touch the codebase. Is there a way to fix it? Robert [00:02:39]: Oh. It. Wouldn’t be surprising to say that it’s a problem that software always had. So probably there are some magic techniques that are trying to solve this problem. One of the most popular, let’s say, tools or techniques that are trying to approach that is domain-driven design. What? Robert [00:03:00]: If you heard about Domain Driven Design, you probably also noticed that for some reason it’s not the default way of many teams to work. From one side we have some solution, from other side probably didn’t solve all the issues that we have. Miłosz [00:03:16]: And it’s also sometimes confusing; sometimes it seems like something you would use only for enterprise projects, but actually it worked for us before. So we like talking about it and sharing the ideas. Robert [00:03:29]: And look how people are surprised sometimes to hear that it worked for us because many people tried to use domain-driven design and it introduced more mass for them. Or maybe they worked in a project when somebody tried to use domain-driven design and it’s the worst experience that they have. Interesting. Miłosz [00:03:45]: Mm. Miłosz [00:03:47]: So the bold assumptions for today’s episode are that you can go deep into many of the ideas and they shine in complex projects, but very often you can use some more basic version of the key ideas of it and apply it to most projects actually. Robert [00:04:06]: So— In other words, in this episode, we will cover How to not use domain-driven design, but some part of domain-driven design and how to make them useful in your projects. Miłosz [00:04:18]: First, let’s think about what is difficult about building software. Miłosz [00:04:24]: So let’s consider someone wants to start. and they want to learn some programming language right, what would you recommend them to learn? Robert [00:04:34]: Maybe not, why, what I would recommend, but what people are using. Miłosz [00:04:37]: Maybe it was a default, like us were here. Robert [00:04:40]: So. They probably need some programming language, right? Yeah. Kubernetes, OOP design patterns. Miłosz [00:04:48]: There are some frameworks, some good practices, some algorithms, maybe data structures, stuff like that. Robert [00:04:55]: Useful stuff that you are using on interviews, in other words. Exactly. And it’s actually, I think, pretty interesting that a lot of learnings nowadays is focused around things that are useful for interviews, but on the other side they are not that relevant when you are working in a job. And I would say that it’s even worse because you are solving problems that normally you should not solve in the problems that you are working like. Robert [00:05:23]: You should probably not implement your own databases and you implement binary trees because most graphics have that out of the box. Miłosz [00:05:31]: But at the same time, if this person wants to learn, wants to join a company, they probably need to learn some computer science theory, like what is big O notation. How to write optimized algorithms for sorting, how to create data structures, stuff like that. Robert [00:05:52]: It’s useful. Miłosz [00:05:54]: Yeah, it is. And it’s often what you need to know to join a company. But my question is, is it enough to ship good software? Robert [00:06:03]: Well, I would say that it’s not enough, and probably it’s even not needed, because well, we are people who are self-taught, so we’ve been learning programming ourselves, and we’ve been shipping multiple projects without that, basically, and many people were using that. They were maybe not perfect, but for the other side, they worked. Miłosz [00:06:27]: Yeah, so I would say it’s like half of the skills you need. Miłosz [00:06:32]: And it’s useful, it’s needed. It’s often what we focus on. But there’s also the other half of stuff you need— to ship good software. So what we discussed so far is the technical complexity, right? So making software fast and scalable, and optimizing memory usage. Miłosz [00:06:54]: Stuff like this. Robert [00:06:55]: Probably we can put it into the bucket of software engineering. So this is this hard part. Maybe not hard, but this. Yeah, this low-level part maybe. Yeah. Miłosz [00:07:06]: The coding part. Miłosz [00:07:09]: And the other half, how would you call it? Robert [00:07:14]: So if you are working with product, probably product engineering. Miłosz [00:07:18]: Yeah. So the product or domain challenges. So basically figuring out how the thing should work and what should it do. Robert [00:07:29]: So this product engineering or those domain challenges, it’s less about those technical parts, but rather thinking. Robert [00:07:42]: How to tackle complexity, how this feature should work, maybe even thinking if it will be useful. Because if you know if something is not useful, you can remove some functionality and decrease complexity with that. Robert [00:07:58]: It’s also this: less priced part, let’s say, and that it’s connected to communicating to people. Robert [00:08:06]: Yeah, figuring out together what you want to build. Miłosz [00:08:11]: Maybe also talking to your users? Robert [00:08:13]: I think it’s also partly related to some long-term maintenance. So it’s, I would say, partially part of the software engineering, but I think it’s often lost somewhere because, when you have software that is not that maintainable from other side, it’s also challenging. And I would say that a lot of people that have this enduring mindset, they like challenges. So if software is complicated, it’s great because we have a lot of challenges there to solve. Miłosz [00:08:45]: Yeah, the technical parts are very interesting to solve. Miłosz [00:08:49]: We like puzzles. At least I know I like them too. And the technical parts often have a very specific answer to them. Mm-hmm. Miłosz [00:09:01]: For example, we can have an SQL query, you can benchmark it, and then you can come up with a faster query. And it’s very easy to compare if it takes 2 seconds or 10 milliseconds and it feels great to have this measurable progress. But for many businesses, it’s not really critical at all. If your website handles a request in 50 or 70 milliseconds— or whatever it is— usually not a huge difference, except for the first page load, which is critical. But for many web applications, saving some data takes one second longer or shorter; it’s not a huge deal and probably not worth spending days on. Miłosz [00:09:52]: coding on. Robert [00:09:53]: As long as cost is not the important part, but again, it’s also some subset of products. For many products, it’s not that important at the end. Miłosz [00:10:03]: Exactly. Miłosz [00:10:05]: But for most businesses, the hard part is taking the problem domain they need to solve and representing this in code. Miłosz [00:10:17]: Especially if the problem domain itself is complex. Miłosz [00:10:23]: And usually the real world is messy in itself. Right. So we have no easy way to compare if something works or not, like with the technical parts. We often need to experiment, see if something sticks, see how users use it. Robert [00:10:47]: Basically, the feedback loop is much, much longer and it’s not that obvious. Like you said with SQL query, it’s easy to measure if SQL query is faster or not. But obviously you can measure some customer satisfaction or some feature, but it’s more indirect and it’s usually taking a bit longer to get back to you. And it’s also a bit harder to understand sometimes how it’s happening because you can change some functionality. And notice that, okay, some behavior changed, but sometimes you may have that many changes around it—so it’s hard to judge if it’s because of it, actually. Miłosz [00:11:28]: Yeah, so it’s much more vague. And if you consider most of applications, like SaaS apps, they are usually very similar. If you change jobs, let’s say you used to work on web application using Go and TypeScript. Miłosz [00:11:50]: You can end up at another completely different company and use the same tech stack. It’s not much different to use the same programming language. Miłosz [00:12:00]: Same framework, basically, protocols, databases are very similar. They don’t change much, but on the other hand, what the company does can be completely different, and this is the No. Miłosz [00:12:13]: the differentiating factor between them. It’s sometimes funny how companies try to make their technical choices something that makes them distinct. So I don’t know if you remember— About 10 years ago, there was this hype on MongoDB, for example. Oh yeah. Miłosz [00:12:32]: And some companies were very open about it. Robert [00:12:39]: Proud. So when you’ve been sending CVs and it was the job posting, it was ‘Yes, we are using MongoDB. Yes, we are web-scale, but of course, it did not really matter in the end. It’s so absurd. But it’s also interesting because during that time, some people were saying that it’s absurd, but from other sites, many people were following that and saying, ‘Yeah, it’s so great.’ You’re using MongoDB, your web scales. Robert [00:13:08]: Let’s see what we’ll see 10 years. Miłosz [00:13:12]: You can just as well use JSON columns in Postgres and achieve pretty much the same thing, except for some differences in very specific cases. Robert [00:13:22]: Or losing changes because of inconsistencies between nodes. Miłosz [00:13:28]: Yeah, but basically, if you take two companies that do the same thing and one uses one database and the other MongoDB, whatever. Miłosz [00:13:39]: It probably won’t impact that much the end result. How well they do in business, how many users they have, how good the product is. Miłosz [00:13:49]: There’s a lot more factors in it. Robert [00:13:52]: Okay. In other words, we can say that many companies, many people maybe solve on the wrong thing because when starting your project, they’re thinking, ‘Okay, this time we’ll use this framework, this database.’ Miłosz [00:14:06]: Yeah, very different this time. Robert [00:14:08]: Yeah, yeah. Robert [00:14:10]: This time everything will be great. And again, we are in the cycle of starting in the greenfield and ending up with legacy, like: It was done five years ago, 10 years ago. So probably. Robert [00:14:28]: The technology that we’re using— the framework, database, maybe not the most important indicator if project is successful or not, and by ‘successful’ I mean multiple things. So, if it’s maintainable, if it has users, if it makes money— probably from perspective of developers, the maintenance is the most important part. But again, it’s not the only thing. Miłosz [00:14:54]: Yeah, but we treated that as something that’s a core thing we need to solve. Sometimes. Right. Miłosz [00:15:03]: I remember some companies with the idea that we will dedicate top talent to this core platform framework— or something— and once we solve it, then adding new features and apps to it will be trivial. Just hire anyone who’s able to code in this language, and they will just use this great framework to implement a new feature in one day. Miłosz [00:15:34]: I don’t know where this idea comes from, but I think it’s quite common. Robert [00:15:40]: And I think one part of the problem may be that if you are hiring specialists in many fields, the sentence ‘if you have hammer in your hand, you will see nails everywhere’ may happen. I mean, if you find hiring specialists from multiple areas, they will spend a ton of time on solving solving problems that may be not the problem in this company yet. So, for example, this startup and it’s starting. They don’t have scale yet. And probably it could be run on a single VM. And later, you have, for example, some Kubernetes specialists. So what they can do, they can create multiple Kubernetes clusters for multiple environments and later serve. Requests per second and even 10 requests per second. It’s like nothing. You can do it with a single VM, but it’s not for free. Obviously, you are using your top talents, but for solving problems that are not important. Miłosz [00:16:43]: And I remember myself falling into this trap, probably multiple times. But one story that I remember very well is how I used to run some game jams with friends, just making game in two days, right? And one time we wanted to create a story-driven game with a lot of dialogues, so you can choose dialogue options and it leads you to a different part of the game. Somehow we came up with the idea that we will do it event-driven. So each option will be an event that sends and triggers something. We had this idea that, now, this would be a lot of events, which will be hard to manage. So we will need this even during the framework. So, if only we get this framework right, all that’s left is to implement the dialogues. Miłosz [00:17:30]: That would be it. So we spent many hours creating this messaging framework using generics. Miłosz [00:17:40]: C# and I didn’t work very well. So it was super fun to come up with it. And in the end, we had this good interface that, you know, we just, you can very easily publish an event. Very easily subscribe to it. And the result was that it was probably the only game jam we never finished. We had this technical solution, but we didn’t progress with the game at all. Robert [00:18:05]: So in other words, you are not the best specialist to be hired to build event-driven games. Exactly. Exactly. Robert [00:18:12]: I can create a framework for you. Miłosz [00:18:14]: I think it shows. It’s like a very condensed, you know. Scenario that I think happens on bigger scale as well, and I don’t even know where it comes from, but somehow we tend to think, if only we can create this framework first, and everything will be easy. Robert [00:18:36]: And sometimes, if you succeed in creating this framework or whatever it is, this prerequisite, and you’re starting to rework on the things that matter. Robert [00:18:47]: You’re starting to notice that, okay, actually, it’s totally not needed. Robert [00:18:52]: We just spent two months on that. And you know what’s the worst thing? The worst thing about that is sunk cost fallacy. So you’re spending more and more time on maintaining that, losing a lot of resources, time, and nerves on maintaining that. But it’s no longer needed. Miłosz [00:19:12]: Yeah, and you don’t want to get rid of it because you invested so much time. Yes, I think we’ve seen many examples like that, right? I think it’s very common when you have a platform team in a company that is supposed to make all teams work together. Very similar ways this is exactly the idea of having one great system that you just add tiny pieces to, and of course it makes sense for a bigger company. But sometimes you can get go too far. I remember one time we had the platform team create service mesh for developers. To use it, you have service discovery so you can communicate between services using just service names. Sounds great, and they spent half a year working on it. But first of all, no one asked for it. Miłosz [00:20:07]: And we really didn’t have any use for it later. It just looked great on paper. Robert [00:20:13]: I think we even mentioned that. Yeah, but maybe we don’t need that because we already are doing it in Middleware. Miłosz [00:20:21]: Maybe if you have five services, maybe you can just call it with the URL of Kubernetes and wow, it’s enough. Robert [00:20:29]: Yeah, and all the cross-cutting concerns could just handle it with common library and middlewares. That’s it. It’s probably again, it depends on context. If you have a super complex project with multiple programming languages, maybe. But in this case, we just have one technology there and it solved the problem already. Miłosz [00:20:52]: So this is all the first part, the technical part you mentioned. Sometimes it’s easier than the domain part, the product part. Robert [00:21:00]: And it’s very often. You can move it between projects as you said earlier. So it can be very similar across and you don’t need to reinvent the wheel basically. So we already know that this technical part is very often similar between the projects and the thing that is different is the domain. Robert [00:21:24]: part of the project. And we already mentioned once domain-driven design, that it’s supposed to be helping us with this. Robert [00:21:33]: Think that is unique to our company. and many of you, maybe heard about this earlier, but it’s not. Robert [00:21:43]: wide load up. do you have some ideas why it’s like that? and actually, what the point of indesign at the first point? Miłosz [00:21:51]: maybe it’s even difficult to define what the approach really is in a very short description. and i remember before i knew about the dd. i think my first contact was when my manager wanted to have the entire team join the training for off-domain design. Miłosz [00:22:17]: and they sent me over like a pdf description of it with lots of details what the training is about and i remember reading through the document. Miłosz [00:22:29]: and at the end i was like, what is this? i have no idea. How is this supposed to help me with my day-to-day work? I just completely had no idea what. Robert [00:22:41]: What’s the idea? So it was more covering some non-technical things and at this point you were thinking, ‘Okay, it doesn’t matter’ or ‘it was not sold properly,’ let’s say. Miłosz [00:22:51]: Yeah, I think it was missing the ‘why.’ And I think also DDD has a lot of unique names or terminology that you didn’t hear before. So if someone tells you, you will learn about bounded contexts. You’re like, ‘What?’ Robert [00:23:10]: I know about design patterns, but what is this? Or maybe aggregate and, oh, okay, so I can aggregate some data. Robert [00:23:20]: Okay, another reason why it’s not that widely adopted, and maybe now it’s very often missing interpret. It’s also the reason that point-driven design is out there already for a pretty long time. So the blue book, so the book that was introduced it. Introducing Domain Driven Design was released almost 25 years ago and the programming world 25 years ago was totally, totally different. Miłosz [00:23:54]: And it was based on years of experience from Eric Evans, so I guess you can say it was about software written in the 90s. So it sounds like a different era today, kind of. Miłosz [00:24:09]: But on the other hand, you can still hear about the DD a lot. Maybe not every company practices it, but there are conferences and books dedicated to it. Miłosz [00:24:23]: So, probably suggest to you that there are many good ideas. That’s how evolution works. Good ideas tend to survive for longer. Robert [00:24:32]: Unfortunately, there’s also a side that many good ideas are misunderstood with time. So some good ideas are useful at the beginning. More people are trying to adopt them. But unfortunately, if those good ideas are not straightforward, but require bigger investment to understand them. Some people tend to get shortcuts and not use them as intended or use them everywhere and it’s quick ending up with some kind of caricature of some techniques, tools. So in other words, many techniques or tools at the beginning were good ideas and were solving some problems, but with time, when people were adopting that, they were starting to adopt some caricature of this technique and it’s often started to do more harm than good. It was visible in discussions that we have in many teams that some people have seen domain-driven design in some projects, but it wasn’t really domain-driven design what they see. Robert [00:25:38]: They’ve seen some overengineering, maybe, just doing it for the sake of doing it. Yeah, so using it in places when it wasn’t needed. So it was just too simple application maybe to use that. And to make it worse, it wasn’t used in there. Right way, because the initial investment was just a big end. People tend to not spend enough time to learn something and, Well, it’s easy to imagine that after 25 years, many, many of projects like that was created and it created a lot of bad PR. Miłosz [00:26:17]: And you can hear people saying, ‘I don’t want to work like this ever again,’ because it was a disaster. It was a super complex project that was made overcomplicated. Robert [00:26:28]: The similar situation is also, for example, with clean architecture, when we sometimes hear that, ‘ah, no, clean architecture doesn’t work,’ and later we hear about seven layers and. Miłosz [00:26:39]: Yes, but on the other hand, if you read the introduction to the Global. I think the core ideas are still very relevant. The introduction describes the state of most software projects. How they get stuck, what are the issues. And now it’s almost 25 years later, everything pretty much in the same spot. Miłosz [00:27:08]: Sense. We got better with frameworks and some technical stuff, but the core idea of working with people on the product. Miłosz [00:27:21]: Not that much different. Robert [00:27:22]: Are you suggesting that we are reinventing the wheel in software every 10 years? No, it’s not possible. And it’s 25 years during. since releasing the book. I can imagine that we reinvented the wheel at least twice since that time. Miłosz [00:27:41]: microservices and then ditched them. Twice. It was a full cycle. For example, in the Eric Evans book, he says that XML would be a promising technology to communicate between services. Yeah, it’s something fresh to send XML over RPC. Yeah, so since then we probably reinvented this idea a few times. We’ve. Miłosz [00:28:10]: microservices and protocol buffers over gRPC. It’s still the same. Coming back to the question why it’s difficult to understand what DDD is, it’s very broad. It’s like an entire book of many, many ideas. and from design patterns to communication between teams. So it’s very difficult to say what doing GDD means. It’s just very broad method. Robert [00:28:43]: We can also say that it may be philosophical in some way. Hmm, hmm. Miłosz [00:28:48]: Yeah, and you can interpret the patterns in many ways. Robert [00:28:54]: The domain part of projects can differ between projects. So it’s hard to give one solution that fits all. So it’s probably because of that, but it’s also creating a lot of space to interpret things. Hmm. It remains one thing. If you are with us for longer, you maybe already know that some time ago we decided to leave our daily jobs. And now we are doing our own business. But it’s not the first time when we’ve been trying to do that. And we’ve been trying to do it multiple times earlier. But I think. One thing that probably didn’t allow us to succeed earlier was that we didn’t spend enough time to understand actually how to run business. Robert [00:29:43]: How it actually worked that you’re running a successful business. So maybe you had some idea that if you would like to have a successful business, you need to sell things. Uh, do some marketing, do some sales, and earn more than you are spending. Miłosz [00:29:58]: Easy, but just— you just need to read one business book, yeah, which is called ‘Business’ and explains how to do business. Robert [00:30:07]: Yeah, but in practice, it’s not that simple and it’s more sophisticated, let’s say. And if you will not spend enough time to understand how it works deeper. Robert [00:30:20]: The chance that your business will fail is much, much higher. I think the statistics are like in five years, 50% of businesses are failing, and in terms of startups, it’s probably 80% in Twitter or something like that. So it’s a crazy number actually. Robert [00:30:38]: And on the other side, it’s something that you can learn, but you need to spend a bit more time to learn those principles and to not do. Caricature of your business. Miłosz [00:30:50]: It’s similar a bit to DDD. It’s a very broad term and a general approach to solving and you need to learn many patterns. Robert [00:31:00]: And it’s hard to give one advice that will fit all cases. But it’s doable, but you need to spend some time and keep in mind that many, many, many people are not doing this exercise and are later surprised that it doesn’t work. Robert [00:31:18]: Like with DDD, you can also see that many people are saying that running business is super hard. Robert [00:31:23]: So I would say that it’s hard, but I think it’s also not. That hard that people think. It’s just about knowing the principles and spending enough time on it. Miłosz [00:31:36]: Right, so if you were to boil down DDD to one primary theme, I would say something like it’s about understanding the domain you work with and then modeling it well in the code. Whether you call it DDD or not, it’s like a mindset. More than patterns, maybe? Robert [00:31:57]: In other words, you should always do that. I mean, if somebody will tell you that, okay, you should not think about the domain, what problems you are solving in your software and how you are writing in an account like it sounds crazy. Miłosz [00:32:12]: Yeah, it’s like you’re deciding. Yeah, I will create a website. I just don’t know which one. So it sounds like you do something for the sake of learning the technology, which is fine. Sure. But if you want to create a product, it’s not really helpful. Robert [00:32:32]: My observation is actually that dark. Robert [00:32:37]: Two kinds of people that are often visible in the area of techniques like domain-driven design, architecture, take anything. Robert [00:32:46]: There are two comps that are totally opposites. One comp is saying that you should always use that. Because it’s great, etc. And the second is you should never use it. That is the worst idea in the world. And I think it’s pretty interesting that even if they are opposite, they have many in common because they are extreme in some way. Miłosz [00:33:06]: So they treat it more like kind of religion rather than something you want to use to deliver a product. Robert [00:33:17]: And in practice, probably you should be somewhere in the middle. Robert [00:33:23]: It may sound obvious, but you should use proper technique for a proper problem. Robert [00:33:29]: That’s it. Robert [00:33:31]: It’s important when you see some religious people about techniques or tools that are against them or love to use it always that it’s the same coin, but just other sides of it. Robert [00:33:46]: I think it— gets into one thing that you should watch out to not use any techniques or tools like DDD just for sake of using it. So you should identify the problems that you have and use the right tools for that. And the Winterwind design have couple tools that you can take out of that like a tool from a toolbox and just use what you need from that. Miłosz [00:34:12]: There’s probably no project that will use every pattern described in the book. Robert [00:34:16]: Yes, but probably you agree that every project should use some parts of domain-driven design. So it may sound controversial, but it’s getting into a point that you may be using some parts of domain-driven design. Even without realizing that. It may be because of reinventing the wheel. Again, it’s probably easier to just see that, okay, you have some tools like that, but many of them, some people just can find them, yeah, probably except for very trivial projects, but we don’t care about them here. Miłosz [00:34:51]: Like, if you create something that’s very trivial and doesn’t need any kind of design and architecture, and you probably don’t need to figure out how to do it better. Miłosz [00:35:05]: So we don’t worry about this here. Miłosz [00:35:09]: Maybe also something very technical like Linux kernel. I don’t know. But probably also there should be at least one idea from DDD that applies there. Maybe value object. Miłosz [00:35:21]: Yeah, but in general. Miłosz [00:35:24]: If you boil DDD down to some very key ideas, I think you should be able to use them if you know what you’re looking for. Robert [00:35:34]: I think it’s a similar situation like with the book Design Patterns. So pretty old one. I don’t remember actually how many years. It’s older than DDD. But I think 96 or 97. So still, pretty old book, but still pretty relevant and many of us may be using patterns from this book. Even without knowing that those are patterns from this book, because it was used by somebody else and we learn those patterns by replicating them. But still. We may be missing some of the context maybe of them. And if we read that, maybe we could understand them a bit more and apply them in the better situations also. Miłosz [00:36:15]: I remember there was a joke or people making fun of people in interviews. When asked about design patterns, they will tell something like, ‘Oh, yeah.’ There’s the singleton, and that’s it. But that’s also kind of the point. You know, there’s an entire book, and doesn’t mean you need to know all the patterns and use them all in a project— even if you learn about some, like maybe you can use a strategy here or decorator in another place. I think it’s still a good result. You don’t need to force them and rewrite your entire project to use, I don’t know, abstract factory method or something. Robert [00:37:00]: It’s always about identifying some problem that is really a problem, and not solving. Robert [00:37:08]: Some imaginary problems. Yeah. Miłosz [00:37:10]: Especially if you force the patterns. I think that’s the worst thing that can happen because it just leads to overengineering. You just read the book and you like the ideas, so you use them everywhere in your next projects and of course they don’t apply to all of them. Robert [00:37:27]: And I think it. It plays nicely with a couple of things that we covered in the previous episode that we’ve mentioned: that being a principal engineer doesn’t mean that you’re forcing to use some patterns. You should instead be able to convince people what value those patterns give and and convincing them that it’s worth using some patterns there. Hmm. And at the end of the day, it’s also a matter of solving your problems, not imaginary problems. Miłosz [00:37:59]: There is no one solution to suggest. Because there are so many factors in play, like the size of your team. If you are a 20-person team in a corporation, a developer working on open source software, or maybe you are a five-person startup, you will probably use a very different approach to software. Robert [00:38:22]: And also, what’s the timeline for the project? Because if you are in a startup that you’re starving for, if you don’t know if you will survive half a year, and you’re doing refactoring, it’s probably not the best idea, but I mean, it depends, obviously, but it’s again going back to solving real problems that you have. So, if you are a startup that has a half year of runaway and you’re fighting for survival, you should choose your fight wisely, basically, and not put maybe refactor some code. That you didn’t touch for a year and put design patterns there. But maybe you have some part of the application that, for example, you are doing a pivot now and you need to— it quickly, but you’re wasting a ton of time to modify this code, maybe doing this refactoring and putting those design patterns may be good at the other, even if you’re fighting for survival. Robert [00:39:26]: Because maybe it may allow you to build this new functionality or do this pivot within this time that you have left. Miłosz [00:39:33]: Have some idea that this project will be for you longer. You probably need some better maintenance here. But but maybe some some areas are more less important. Maybe you will get rid of them in two months if it doesn’t work out. But because it’s sometimes difficult to decide. Miłosz [00:39:57]: This might be a bit controversial, maybe, but I think you don’t need to go all in. Most of the patterns from DDD you could decide to use them in a more, let’s say, shallow version, other than deep version. So you basically think of the the first principles, the big idea behind the pattern. You don’t need to implement all of it and still get some good results. Because you often don’t know if the project you work on will be with you for longer, or maybe it’s just a prototype. Miłosz [00:40:33]: But I think there’s a balance between having some very scrappy code and something that’s very enterprising. While design software. Robert [00:40:44]: It’s actually an interesting thing because it’s one of the things that Domain Driven Design covers with core and supporting domains. Robert [00:40:56]: It’s kind of forcing you to think, ‘Okay, is this code really important to my business, or is it less important?’ And it can kind of affect how we are thinking about this code. Robert [00:41:09]: I know that many people like consistency in the code, like, ‘Okay, let’s do.’ Something in the consistent way everywhere. End. It may sound like a decent idea, let’s say, but it may be also dangerous because you at some point start to over-engineer some parts because some parts of the application are more important, more often touched, more complicated, and some are simpler. You may just lose a lot of time for maintaining this consistency for sake of nothing. Basically. Miłosz [00:41:44]: So this episode is more like an introduction, but in the next episodes we want to cover more patterns from both shallow and deep end, so you can better decide what to use. Robert [00:41:58]: But I think we should put big emphasis on the thing that people are very often missing. So the strategic part, because Tactical patterns— those strategic domain-driven design— are the patterns that are not that much about how you write the code, but it’s more about strategic thinking, like how the modules are interacting, how modules are split. Very often cross-team collaboration. Robert [00:42:28]: How important are multiple parts of the code? And what’s interesting, everything what I just said, I think you should probably think about that in any project that you are doing. You’re using domain-driven design or not. It’s cool to have some tools that are helping you to do that, and some frameworks— let’s say it’s not a framework, but some way of thinking how to approach this. And from other side, you have technical patterns that you’re using in the code, and those ones are not always needed. And you don’t need to always apply them. Miłosz [00:43:01]: So we won’t be diving deep into patterns today, but let’s look at the core idea of DDD. Miłosz [00:43:12]: The domain model or model-driven design or knowledge crunching, as it’s specified by the book. What is it about? Robert [00:43:23]: There’s one thing in multiple projects that is a bit like running business, like we said earlier. So it’s obvious that you should earn more money than you are spending. so you have a successful business. In software, it’s a matter of understanding what kind of problem you are solving and how it should work. Robert [00:43:45]: It makes all of us like, yeah, we are starting a project. We should understand how it works and how we should implement that. From other sites It’s not happening. Miłosz [00:43:59]: yeah because sometimes the business is complex by itself and even for simpler ones like consider e-commerce seems like a simple domain it’s just people People buy products and that’s it. But if you dive deeper, there’s capturing payments and delivering the products and getting the inventory somehow. So it’s more complex than it seems. Robert [00:44:24]: How it looks in practice? People are thinking, ‘Oh, okay, it’s simple.’ We know what to do. Let’s go to implement that. Exactly. Two years later. Miłosz [00:44:33]: Consider, if you wanted to to open an online shop. Miłosz [00:44:38]: Oh. If you ask a software developer, they will probably tell you, ‘Yeah, you just need to host this WordPress website. With an e-commerce plugin and you’re done.’ Robert [00:44:52]: I think many software developers will say, ‘No, no, we need to implement it from scratch.’ Miłosz [00:44:57]: Oh yeah, that’s even better. But my point is, they probably won’t think about that— that you need to. Order the inventory from somewhere, store it somewhere, and then somehow send it out to people, which is the hard part of the business and your software probably needs to work with it somehow. Robert [00:45:16]: It actually works nicely with this part of strategic patterns of domain-driven design when we are considering what’s our core domain and what’s the value for our company. Miłosz [00:45:27]: And many projects are much, much more complex than e-commerce. If you consider accounting, for example, you can get into some very very specific edge cases depending on the country and if you do do it globally, then it’s yeah, it’s crazy amount of knowledge you need to understand and to implement. Miłosz [00:45:48]: Complexity in those projects doesn’t really come from the technical challenges, but from the business itself. It’s just complex to run this kind of business, so the software becomes also complex. Robert [00:46:03]: So, in other words, you have multiple dimensions, complexity dimensions, let’s say, to this product, and you can add more and more and more. So, for example, you’re starting with one country. Right. We need to scale globally. Okay. We need to support multiple countries, multiple continents. Great. Later, we need to support multiple types of product. So it can have multiple tax rates and multiple rules. It’s getting more and more and more complex. Miłosz [00:46:28]: And it’s always complex even before you introduce any kind of technical challenges. Just not considering software at all. This is already complicated. Miłosz [00:46:39]: So you want to model it well in the code, so at least you don’t need to deal with the additional complexity. Robert [00:46:51]: And to make it worse, often the initial idea is also changing with the time. So I have two good examples of the companies that were like that. So Netflix. Robert [00:47:01]: Maybe many of you know that Netflix at the beginning was an extreme platform. At the beginning they were just renting DVDs. So it’s interesting how it changed over time, how they were making business, and probably how much it changed how they’re operating. The other interesting business that changed over time is Instagram. Instagram wasn’t this kind of social media platform that you know now; it was more checking location applications. So I guess you were able to send people information like, ‘Hey, I’m here’ or something like that, not share pictures. Miłosz [00:47:40]: Yeah. And so imagine in both cases. Miłosz [00:47:45]: At some point, developers complained that the requirements weren’t clear. We didn’t know this component would become this thing it is now. Of course, it wasn’t clear because no one knew about it back then and they had to adapt somehow. Robert [00:48:00]: Those examples were more extreme, let’s say, but most companies are changing their initial assumptions pretty much, so it’s also adding this complexity there. Robert [00:48:14]: I also like example of the cyberpunk game that the cars are horses from Witcher 3 because they’re using the same game engine. Miłosz [00:48:25]: Yeah, so that goes back to the discussion of evolving software. The first version is usually simple, right? Everything works fine. But then someone tells you, actually, we’re not renting DVDs. We’re going to be streaming videos. Miłosz [00:48:42]: And at this point, you have to change your software to a completely different model sometimes. Miłosz [00:48:52]: It’s great if you could do it easily, if you don’t need to spend one year to rebuild everything. What helps is also what we mentioned in the last episode about developers working directly with domain experts. Robert [00:49:06]: So even if it’s a bit controversial, so we’ve seen how it can change things and again affect how you are implementing things. Miłosz [00:49:17]: So we don’t need a product owner or product manager acting as a proxy? Robert [00:49:22]: Always, because sometimes it makes sense, but sometimes it makes sense to just go direct. Miłosz [00:49:27]: But the idea of the domain model is to create a model that represents the problem domain as well as you can at the moment. Miłosz [00:49:37]: Developers can understand how it works, the more information they get from the people who work directly with the business. The better. The model doesn’t need to be physical, so it’s not a document or diagrams or anything else, although it helps, but it’s more abstract. It’s about understanding how it should work. And. Miłosz [00:50:03]: The ultimate artifact is the code. Robert [00:50:06]: And it also doesn’t need to cover everything. So I always like to compare this model part to a map. So map is not a perfect representation of real world, but it’s a useful representation of the real world. So you can look on the map and see the details that you care about and do the job that you need. Miłosz [00:50:28]: And it’s also not a data model, which is a common misconception. Miłosz [00:50:35]: Very often in software you can see models used for both the domain logic and some storage, for example, or transport. Miłosz [00:50:49]: For developers this is unnatural. Miłosz [00:50:52]: No. thing to do treat them as the same so we often start implementation with database schema, for example, because it feels natural to think of storage. Miłosz [00:51:03]: But here we think about the more abstract domain model, how the thing works, what behaviors it has. Robert [00:51:11]: So this is pretty big part. Paradigm shift and it’s also not consistent with how we’ve been learning earlier. So this is also the theme that was pretty common in the last two episodes. But how we are learning to code is very far away from the real world. And we are not prepared actually to work in the companies. And later it’s ending up with the. Robert [00:51:37]: golden cage concept from the previous episode. Getting out of that is not simple, but I think it’s mandatory if you would like to be successful and survive on the tough market and work in the better places. Miłosz [00:51:52]: Another tool from the previous episode is event storming, which is also closely related to figuring out the domain. Robert [00:52:01]: So today we’re more focusing focusing on giving you some introduction. Robert [00:52:09]: To— Maybe not Domain Driven Design really, but some parts of Domain Driven Design that we’ve been using in the project. Sometimes we’ll name it Domain Driven Design, sometimes not, because again, its patterns are secondary. I think it’s more important to focus on problems, not on the patterns, because when you are focusing on patterns, you are just with this hammer in your hand and looking for nails everywhere. So I think it’s more important to focus on problems that we’re solving and how we can do that. And yeah, this is something that we’ll be doing in the next couple episodes and it’s connected to the fact that now we are working on our new training the domain engineer and that will be available at the beginning of next year in January. Miłosz [00:52:56]: Yeah, so in the next episodes we will look at more use cases for the strategic patterns. Robert [00:53:02]: Yeah, so we’ll give you some more practical tools also, how to approach that. Miłosz [00:53:08]: So let us know in the comments if you have any topics you would like to hear about. You can also check our ebook, ‘Go With the Domain,’ which is about working with domain-driven ideas. Miłosz [00:53:22]: In Go specifically. Robert [00:53:24]: Yeah, so we will be preparing agenda for next episode soon. So let us know on YouTube comments or Spotify comments. So read all of them and it will be. Cool to help you with some projects or problems that you have. We have many, many, many stories from projects that we’ve been working in, but we’d also love to introduce a bit more there. Miłosz [00:53:50]: So that’s it for today. Miłosz [00:53:53]: Subscribe to our channel so you can see the new episodes as they come. Miłosz [00:53:59]: And hit that like button. Robert [00:54:01]: And if you don’t want to miss when the domain engineer will be available, just Google the domain engineer and join our newsletter if you are not there. Yet. So we’ll send you an info when it will be ready. And you will also notified when new episodes will be out, because we know that sometimes notifications from multiple platforms are not sent. So when you’re in our newsletter, you are sure that you will always get the notification. Robert [00:54:29]: Okay, so I think that’s all for today. So thank you, Miłosz, for your time. Miłosz [00:54:33]: Thank you, Robert. See you next time. Robert [00:54:35]: See you in two weeks. Bye-bye.

Quick takeaways Developers are often kept in “golden cages” - treated as coding monkeys receiving tasks without understanding the broader context or product vision Requirements are usually incomplete or wrong - understanding the business context helps developers spot issues early and deliver what’s actually needed Collaboration with stakeholders is crucial - developers have valuable product insights and should be involved in planning, not just implementation Event Storming is a powerful collaboration tool - a lightweight workshop technique that brings developers, product managers, and stakeholders together to align on complex topics Change takes time and trust - transforming company culture requires starting small, showing results, and convincing people through solving their actual problems Introduction In this episode, we talk about why software projects in regular jobs are delivered much slower compared to side projects, and what you can do about it. We share our journey from building hobby projects as teenagers to working in professional environments, and the differences we encountered. Developers are often kept isolated from product decisions and treated as “coding monkeys in golden cages” - just receiving tasks without understanding the why behind them. We discuss techniques like Event Storming that can help break down these barriers, and improve collaboration between developers and product managers. Notes Event Storming - a workshop technique created by Alberto Brandolini for collaborative domain modeling Quotes Five days of coding can save one day of planning. Robert If you want to be promoted, for example, delivering is the very best way to do it. So instead of just waiting for requirements and complaining that it doesn’t work, you can try to help whoever needs what they need. Miłosz How it often looks like: it’s like companies are trying to make the programmers coding monkeys in some golden cages, basically. Robert Instead of complaining that we never get anything done, you tell them, yeah, this feature could have been one week, but we spent two months. Miłosz The worst thing that we noticed is that the performance of how fast we’ve been able to deliver new stuff was much worse. Robert But what if instead, you have this domain expert, this stakeholder meet with developers directly and tell them what they need. And then usually developers will tell them, oh my god, this is so complicated, it will take two months. But if we drop this one button from the dashboard, it will take one week. Miłosz Timestamps 00:00:00 - Introduction 00:01:11 - Our story 00:05:06 - Slow projects 00:07:10 - Golden cage problem 00:09:30 - Incomplete requirements 00:13:14 - Developer time optimization trap 00:15:15 - Level 2: Collaboration 00:19:09 - MVP and simplification 00:23:33 - Talking to stakeholders 00:28:40 - Levels of engagement 00:30:29 - Why you should care 00:34:30 - Event Storming 00:44:37 - Getting started 00:50:17 - Practical advice 01:02:22 - Changing culture Transcript Robert [00:00:59]: I’m Robert. Miłosz [00:01:00]: And I’m Miłosz. And this is No Silver Bullet Podcast, where we discuss mindful backend engineering. In this show, we share takes on software engineering that will help you grow into a principal engineer. Robert [00:01:11]: In this episode, maybe let’s start with a bit of our story, because I think it’s pretty important to understand that, how we started to build some production systems and from where our experience came. So probably some of you know that we met with Miłosz in the high school, so a pretty long time ago. We were just talking before recording. That it was, I think, 17 years ago, so a pretty long time ago. Actually, I know Miłosz two months longer than he knows his wife. Miłosz [00:01:42]: That’s pretty long, and the reason I know you longer is because we are already working on projects before everybody started. So we started with some toy websites and then grew into more complex projects. Robert [00:01:59]: And many of those projects actually became pretty successful, so many people were using them. The downside was that, well, there wasn’t great business sense at the time, and well, a lot of people used that, but it wasn’t enough for making for a living of that. So we decided, okay, let’s go to university. Maybe we’ll learn something useful. Long story short, we did it. So we left the university at some point. So we quickly dropped out. Yeah, because, well, it was probably a bit boring since we’ve been already coding for multiple years and we’ve seen that, OK, maybe we’ll learn physics and mathematics, but… That’s not probably relevant in the software that we are writing. Miłosz [00:02:41]: We still work on our projects in the meantime, but there was almost no money coming from it. We had to just start regular jobs, which was fun at the time, because we didn’t have this professional experience, let’s say that’s our own. Robert [00:02:55]: I just remember that. During the time, the market also wasn’t that easy. I mean, we’ve been over decoding for a couple of years already, and I was already afraid that I will need to go back to my parents’ home because I I struggled a bit to find a job, despite having a couple years of experience already. Miłosz [00:03:11]: Even the basic wage we made back then was great for our student life, compared to hobby projects. Robert [00:03:21]: That’s true. But I think it wasn’t worth thinking about that because, since we’ve been working already on multiple projects, we’ve been working, so we have two people— team basically, so me and Miłosz. It was a bit shocking to start to work in the normal job, let’s say, let’s name it in this way. So, you know, in the way that you are no longer your boss, but instead you are getting requirements from somebody else. You are not inventing the project and product that you are building. It’s totally different dynamics. And for some people, I know it’s fine because they don’t need to touch this ugly business and product they can just code. But we’ve been different. We’ve always wanted to create something and invent something. And it was a bit shocking. Miłosz [00:04:08]: The upside is you don’t need to worry about… If the mana is coming through. Someone else handles that for you. But alongside this, someone tells you, here’s this system that has been around for 10 years and you need to add some feature to it. And usually it’s not an easy task. Robert [00:04:27]: Yeah, yeah, exactly. And you know, you have other challenges, like you need to collaborate with more people. So it’s no longer two of us, but the team is bigger, you need to take maintenance into account more because there’s 50% of a chance that it will be the person that will be maintaining it later. In this case, it’s totally different because you’re working with someone’s legacy code that some person who worked here 10 years ago wrote. This person no longer works there. Code that we have written 10 years ago is still running out of there. Miłosz [00:04:58]: You have to agree with other people in the team how to do it, probably, because you are not the expert anymore. Robert [00:05:04]: So it’s creating a lot of challenges. Robert [00:05:06]: Those challenges are pretty important because the worst thing that we noticed is that the performance of how fast we’ve been able to deliver new stuff was much worse. And in this episode, we will discuss some tools and techniques that should help with that. Robert [00:05:27]: Can give you some tools to handle situations when you are not working in a solo project, but in a team and you need to collaborate in some way and create software that is maintainable. In longer term than one month. So can you keep the same pace as when working with two people or just a solo developer? So if those problems sound familiar for you, please stay with us. So we will give you some hints how to overcome those problems when you are working in a daily job. So at the beginning, let’s think why projects in your daily job are delivered slower. So in other words, why, when you are working with your friends after work or on your pet project, you are much more productive? What do you think it is? So we already mentioned one thing: when you’re working on the project alone, you are basically your own boss and you’re working on your ideas, which is great. But the downside is that it’s also much harder to make business out of that. I mean, we are software engineers, we are good problem solvers, but from our experience, very often we are not best businessmen. Miłosz [00:06:32]: Yeah and we often start side projects and you know, some websites maybe web apps, but often it’s just something for fun because it’s for learning. And this is how we often are learning, and it’s great. Yeah, but making it into a business is a much different story. Miłosz [00:06:52]: Which might be also why developers usually in companies don’t have that much impact on the product side. So very often they are kept in a silo of software development. Robert [00:07:10]: Yeah, I actually like to name it a bit more brutal— and they need a golden cage. So and how it often looks like: it’s like we are companies are trying to make our programmers coding monkeys in some golden cages, basically. I know it sounds brutal, but this is how it looks in multiple components, basically. So what do you mean by coding monkeys? So there are a couple elements of this problem, let’s say, and also a couple of levels of how companies are approaching this. So the most extreme scenario is when you’re this coding monkey in your golden cage and you’re just receiving tasks to code. I even heard about worse cases when some developers even received empty classes and they were diagrams, empty classes and some UNLs and just please feel the methods and don’t care about anything else. It’s similar to LLMs today, just like a human instead. Yeah, yeah. Robert [00:08:13]: And funny thing, if many people were working like that entire life, well, I can imagine that they may struggle now a bit because… So this is actually, it’s not very performant at the end because the software that is created in this way, obviously the chance of working is very, very low and making some… value. But still, it can be replaced in some way. Miłosz [00:08:36]: So engineers in this scenario are just kind of implementators of some idea. From the outside. I think especially in product companies it might be an issue because developers usually have great insight into the product area as well. Miłosz [00:08:53]: It can work better in some very technical domains where you need developers just to work on some specialized algorithms or something like that. I guess even waterfall can work in such a scenario. Miłosz [00:09:09]: But if it’s a product company, it’s much different. Robert [00:09:12]: Some of you may wonder, ‘Okay, why is that?’ I can just sit in my nice cage and code. It’s great. But… Unfortunately, it’s pretty unproductive. If you’re working with software a bit longer, you probably already know that requirements that you receive, they’re in most cases… Maybe not wrong, but… Miłosz [00:09:30]: Incomplete. It’s probably the biggest issue developers have in their day-to-day work. I can’t work on this. This is not clear. This is missing some requirements. And it’s discouraging because you can’t do anything with it. You can just, you know… tell your product owner or manager or whoever that, sorry, this is not complete. I can do anything with it. And then the product owner needs to go back to whoever knows what to do and figure it out and come back to you with clear requirements, which is never the end of it. Robert [00:10:08]: And needless to say, it’s super unproductive. When you’re working alone, you can ask yourself, ‘Ah, okay, it doesn’t make sense.’ Maybe let’s do it in this way. When you have proxy… Project manager well, it’s very ineffective because probably this project manager also doesn’t know. So this project manager needs to go to another person, that person ask another person, and it can also probably explain why often those requirements are not perfect. Miłosz [00:10:36]: And worst case scenario is probably when the product owner also decides who gets to do what. Miłosz [00:10:45]: Split the tasks between developers so the workload is even across the team. So that’s probably the biggest anti-pattern you can see in project management. Robert [00:10:57]: Yeah, exactly. Because, well, if the project manager is assigning tasks to each developer, probably he may not or she may have not enough context to know. Which person has most knowledge there, or maybe some other things that if team will be responsible for assigning those tasks. It will be much more effective because they have more context to do this decision better. Miłosz [00:11:23]: Yeah, that’s a better approach because the team knows what they specialize in, so they can share the knowledge as well between the team members. Everyone knows a bit about what’s going on in the project. It’s not like there’s one person responsible for everything. Robert [00:11:44]: And I think it’s quite interesting because it’s based on crazy and pointless ‘why’ to work in this way. But there are multiple reasons why, in many companies, it’s the default mode. I remember that. Robert [00:11:58]: In many components, when I was starting at the beginning, it was unfortunately the default mode of working, and it was often some work needed to change that. I mean, it was possible and it wasn’t that hard at the end when you’ve shown what problems it’s generating, but… Yeah, unfortunately, it’s often the default way of working. Do you have some ideas maybe why companies are doing it in this, well, probably crazy way of assigning tasks to people and closing them into the cages? Miłosz [00:12:32]: I guess maybe it’s just the way things always used to be. Miłosz [00:12:38]: Just the default mode of, you know, we have those developers who are implementers and product people who come up with ideas. So the development team becomes the downstream of those ideas. Miłosz [00:12:56]: Ping pong between each other. Robert [00:12:59]: Yeah, I also found one interesting reason for that. And the reason was that developers are expensive. So their time is expensive. So it means that we should use as small amount of their time as we can. Miłosz [00:13:14]: Yeah, basically use as much as we can for all the tasks we need. Robert [00:13:20]: So basically, let’s optimize developers’ time so they can code as much as they can because, well, we’re paying them for coding. This is why we’re paying them that much. Miłosz [00:13:28]: It’s basically like in a factory. You want to utilize all the devices to do the work for you. Robert [00:13:37]: And you’re trapped, because, okay, from one side you are… ensuring that those developers are using 100% of their time on coding. But if you think about that a bit for a bit longer, you will notice that, okay, maybe they’re coding more. But useless things, because even if we will force them to code some stuff, and they will write them after it will be shipped to production after in this kind of companies, half year let’s say, you’ll see that, okay, it’s actually not what we needed. The worst part is it all feels like a good job. Yeah, and for the developers. You may also feel very productive, like I’ve been coding for 12 hours and it’s so great. Robert [00:14:18]: Yes, but there is nothing to show for it. You more like wasted the time. Probably one sentence here will be very accurate. And I really like to remind it over and over. So five days of coding can save one day of planning. And this is basically that. Yes. So how… Miłosz [00:14:35]: talking about or coming up with ideas how something should work feels much less productive than just typing away and creating this. stuff. So we are very eager to jump to implementation, maybe build some prototype. Sometimes it’s useful, but very often it would be better to spend some time. Trying to understand what you really need to build, what will be useful for the users, for example. Miłosz [00:15:06]: But it doesn’t feel as productive as this starting coding. So developers don’t really have an issue with this most of the time. Robert [00:15:15]: So let’s all go maybe one level up in our ladder of how good or bad putting developers into cages can be. So the second level. That I noticed in some companies is some a bit better situation. So in this situation, product owner is going to the team. With some requirements gathered from stakeholders. And in this case, the product manager or product owner is not the person who is creating the tasks, but it’s the person who discusses basically what we need with the developers and he’s working or she’s working with developers to learn how to implement that. Miłosz [00:15:57]: So they present some high-level idea and then developers split it into smaller tasks or come up with implementation, maybe propose some solutions. Robert [00:16:06]: So we no longer receive tasks. It’s no longer assigned to people. We can talk to our closest stakeholder and discuss what we would like to implement. Miłosz [00:16:17]: You get even better knowledge sharing this way. So everyone is on the same page. Also easier to work with code later. Miłosz [00:16:24]: Because everyone has insight how this will impact the end product. You don’t just work on some obscure subtasks on the other end of it. You see the full picture. But there’s also a challenge here, right? It’s not really common knowledge how to do it. Miłosz [00:16:44]: It can take long hours to come up with the final solution. Robert [00:16:49]: And can go into never-ending meetings. Robert [00:16:55]: My observation, obviously, it’s always about balance. But my observation is that often it’s when you have hours of discussion. It’s basically alternative because sometimes even better alternative. I mean, the worst alternative can be that, okay, you are going to saying that, okay, ‘We are discussing already for too long.’ Let’s go. Let’s do what we think should be done. Yes. And later, you’re not spending hours on planning, but you’re spending weeks on implementing something just later to see that why we did it in this way. It’s crazy. It doesn’t make any sense. And nobody’s using that. Robert [00:17:33]: So I think it’s one popular trap that we’re trying to optimize this discussion part. Sometimes, as you said, we don’t have good tools how to discuss requirements with product owner and/ or product manager. We’ll get into that soon. So we’ll give you some tools. Robert [00:17:50]: Yeah, it can lead to that, but it’s, I think, important to ask ourselves the question, what’s the alternative here? And often, the alternative is not the best. Miłosz [00:18:00]: outcome of this can be simplification. Miłosz [00:18:03]: I think you need skilled developers that can have this. Product vision a bit and can talk with product owner, for example, and suggest some solutions. But because often the reason the meetings drags on so long is that we talk about two big problems at once. So we want to create this full system or big feature up front. And there are many edge cases. Developers love poking those edge cases and they will come up with many many issues. So then you have like 20 things you need to tackle before you start the development because there are so many edge cases to cover. And very often, if you start with a simpler version of the same feature, it may be much, much easier to discuss and start. And also very often you… you later realize and actually, you know this is like 80% good, maybe we don’t need most of what we thought we needed in the beginning. Robert [00:19:09]: I think it’s some kind of meta advice that it’s already coming in multiple episodes that we had earlier. So some golden standards that we have tried to have in teams that we’ve been working in. To be able to deliver each task within one day and to be able to deliver some, let’s name it, story within one week. Because, if you’re implementing, you’re creating some functionality into something existing. But it helps us to… Simplify those discussions because, if you are discussing some feature for multiple hours, and you would like to implement within one week, you already see that okay it cannot happen. Because, if you are discussing some feature for multiple hours, the implementation of that will just take much longer. So it’s good because you can start a discussion like, ‘Okay, it’s already scope creep. Let’s try to figure out what we can do. Some kind of how we can make some kind of MVP out of that, and start with that, and iterate on that. I said that it’s working for features that you’re already building on already existing functionality. Robert [00:20:17]: Because, in this case, you can do it faster. It’s a bit different when you are doing kickoff of new project. Because, if you are doing kickoff of some project, sometimes you need a bit more time to have something that is functional. So, but I would say that you should also watch out to not exceed, let’s say, one month. For example, and have in one month some MVP that already delivers something, and somebody can already use that, and you can already receive some feedback on that. And again, I think it may sound very obvious, but… Robert [00:20:48]: It just works, basically. So it’s one of those advices that are too simple to be true, but this is how it works. Miłosz [00:20:57]: Splitting work is not easy, for sure. We’ll probably need another episode on this. But coming back to this… with product owner. This is probably an upgrade on the previous approach. Miłosz [00:21:09]: Product owner deciding on everything themselves, but it’s still… not perfect, right? Because this is a single point of failure. This one person, they understand the domain, they talk with stakeholders or users. Miłosz [00:21:24]: But they still don’t know everything. So they are kind of a proxy between the development team and whoever needs the product built. Robert [00:21:34]: And if you are working this way, probably more than once you have situation like you are asking some question to your product owner, he or she is seeing you for a while and I don’t know, I need to ask this stakeholder. I will go back to you in a week. Robert [00:21:52]: And it’s good because if this question is there, because maybe it will save you one week of implementation that it will be useless at the end. So that’s good. But from other side, you need to often wait for some time for having those. Miłosz [00:22:06]: Yeah, I remember those planning sessions where we were deciding what to… Take on for the sprint for the next two weeks, and of course, we poke holes in some plant features. We see some magic cases that are not solved, and the product owner says, ‘Yeah, you’re right.’ So yeah, I need to go figure out what to do. Meanwhile, here are some well-defined but very unimportant tasks you can do in your sprint. So you postpone the important work this way, and because you don’t have anything better to do, you start working on something smaller but probably something that’s not really game-changing for the company. Robert [00:22:50]: But sometimes it’s good, because very often you have this backlog of things that we should do later, a . k . a. Miłosz [00:22:57]: never. Sure, but this way you never do these game-changing features. Probably the product needs the most. That’s true. Miłosz [00:23:08]: Because they are so important and risky, no one wants to just… start doing them with some simpler approach. It feels like we need this perfect implementation or specification for it. So you wait until you have it planned perfectly. Robert [00:23:28]: So what’s the crazy alternative for that? Miłosz [00:23:30]: Because I know— because we tried this crazy alternative. Miłosz [00:23:33]: Yeah, so if you go one step further, I guess you could have… developers talk with stakeholders directly? Oh, no, it sounds so expensive. Miłosz [00:23:44]: Or, you know, just domain experts in general, whoever this is, I don’t know, maybe your users or your CEO or whoever. Robert [00:23:54]: That are often the source of those requirements. So there are people with knowledge, there are people with some interest, let’s say, but basically… those are people who give you the direction of work that you are working on. And those are the people to whom your product owner is going to ask how to do something. Exactly. Miłosz [00:24:15]: So it might be your VP of sales or your operations specialist in another team. Robert [00:24:23]: Or even your CEO. Depends on the company. Robert [00:24:27]: Um, Or it may be also an operation theme. So it really depends on the functionality that you’re implementing. Robert [00:24:34]: Usually, if you’re implementing some functionality, there is some person that is mostly interested in that and the idea maker. Miłosz [00:24:44]: go to your product owner and they will tell them I need this new dashboard built and they will reply okay I will talk with my developers in two weeks later. Miłosz [00:24:58]: I will let you know two weeks later what we think. But what if instead, no, you have this… This domain expert, this stakeholder meet with developers directly and tell them what they need. And then usually developers… will tell them, oh my god, this is so complicated, it will take two months. But if we drop this one button from the dashboard, it will… take one week. And then the stakeholder says, oh, really? I don’t need this button at all. Miłosz [00:25:31]: Wow, impossible. Now we save two months of development and also two weeks of planning to figure this out. Robert [00:25:39]: And this is not a story that we invented. It’s actually what happened multiple times for us when, basically, we could spend ages to implement. Something that the stakeholders would say that it’s not that important. But if we have proxy as product owner, it will never happen because the communication between developers and stakeholders will never happen. And to be exact, so it’s not about gathering all stakeholders in the conference room and discussing. Everything because it will be super inefficient. We’ll go into tools very soon. It’s more about having them meet with developers in some way. At some point. And also not doing it for every feature. So to be exact here. So we’ve been doing it for some bigger project kickoffs. So we’ve been not doing it for… every story that we’re implementing because it would be super expensive, especially that our stakeholders were living in a different country. So it meant that gathering, for example, five stakeholders, and discussing everything with them, would cost flights, hotels, etc. Robert [00:26:46]: So it was pretty expensive, but still it will work for us and we’ve been doing it multiple times because, if it wouldn’t, it will just… Miłosz [00:26:55]: Yeah, but this is a bit extreme scenario, right? When we fly over people to meet in the conference room. But this same idea can also be applied on a more simple level. Robert [00:27:04]: It’s easier if you have stakeholders nearby, let’s say. Miłosz [00:27:09]: Yeah, but even if you have someone in another team that uses your software and they have some issue, you can just have a call with them and have them show you directly what’s the problem they have. It’s the same idea applied to a more shallow level, so you don’t need your… Thank you. product owner to be a proxy between you two. You just talk directly to them and you understand, okay— this is the issue they have. So now I will figure out how to fix it instead of waiting for perfect requirements to figure out what to do. But I think the basic idea here is that it’s difficult to come over, having developers think about the solution and offer some simplification. And I remember some product owners trying to do it themselves. They somehow… In a way they knew the software developers were working on and they tried to figure out themselves how complex it would be to add a feature here or there. It’s a disaster always. I guess today they will ask AI which feature is the most complex and how I can simplify it. Miłosz [00:28:27]: Because it won’t work, because you need this expertise to understand what’s going on really. That’s why you need to close the gap, have developers in a discussion. Robert [00:28:40]: Yeah. So to maybe recap this. So I would say that there are three or four levels of how you can basically be engaged in developing the feature. Robert [00:28:56]: The top level is pretty close to how you are working on this feature alone. In this situation you are together working with stakeholders— people that basically can make decisions if the feature can look in this way or not, because this is very important. So to have people that you can ask, okay, can we… skip this and they can say yes, no, and you can go further and not wait one week for decision. It doesn’t need to be like that for every story. It’s probably better for kicking off a big feature. So when we’ve been working on some features like some bigger epics, let’s say, that we’ve been working for Hazier, for example, or some new functionality. This is what we’ve been using. Robert [00:29:41]: One lever lower, you have situation when your product owner, product manager, is gathering requirements and you are working on them together. Robert [00:29:51]: In case of doubt, the product owner can go and ask some stakeholders if it’s problematic or not. It works nicely for some stories because usually you can wait for a while and you’re planning it in advance, so it’s not an issue that it can be a bit delayed. Robert [00:30:09]: I would probably not recommend the ‘golden cages’ approach when basically you are receiving some specification or some task from somebody else and you are just just writing this without really affecting how it’s implemented because it’s very ineffective. Miłosz [00:30:24]: So we said that it’s better from product perspective. Miłosz [00:30:29]: Work directly with domain experts and so on. But why should developers care? Because, for many developers, they are fine being just an engineer and waiting for requirements. Is there anything for them in this scenario? Miłosz [00:30:48]: What do you think? Robert [00:30:50]: So, I think that there are two things that you should keep in mind. And, as you said, so… Many developers, most, hard to say, but many developers don’t care that much about, let’s say, component success. They are just here working to code and they don’t care that much. But I think it’s not that simple because… You could be such this kind of developer in the market when everybody could get a job easily. But as we already said, it’s not like that anymore. And you need to be a bit more outstanding to get job easily. And to understand job market now, it’s probably important to understand about what companies… care when they’re hiring developers. And well, they care about hiring people that will be useful for them. Let’s put it in this way. Miłosz [00:31:44]: So basically, if you are able to deliver stuff that they need, they’re kind of happy and you need to make help them make more money, basically, so they can share some of it with you. Now you know there’s less funding than 10 years ago, it was very easy to Oh. Miłosz [00:32:03]: get VCs to invest in whatever idea you have. So you need to actually deliver something more often. And it’s quite a rare skill to work directly with product. And I think for me the funny thing is: We somehow don’t test this on interviews very often. Also, no one teaches this. Miłosz [00:32:27]: Universities or even your job, you’re mostly taught to write code. I would even say that we are probably more prepared to be those coding monkeys. Yeah, exactly. If you go for a tech interview, you will encounter the slit code exercises or you will be asked to write some algorithm. Miłosz [00:32:51]: On a piece of paper or on a whiteboard. Perfect for coding monkeys. Robert [00:32:54]: Those are requirements. Don’t ask questions, just implement them. Miłosz [00:32:59]: Yeah, the technical details. But then you start… Working and turns out those things are really not that much used. They are not that important. You will spend more time figuring out what to build and what not to build. Robert [00:33:15]: And often you’re hiring this kind of people that are great in algorithms. And later you are finding out that they are expecting them to give them tasks, requirements, and they can just code that. And obviously there are companies where it works, but in the product companies… We already covered why it’s not working. Robert [00:33:36]: It’s also pretty hard to change your mindset if you’ve been working for entire career in this way. So this is obviously the important thing to think about. I mean, if you’re working this way. Robert [00:33:50]: And you will be working this way for your entire… Okay, maybe not your entire life, but for longer time, it’s harder later to switch. And companies are also— now better in picking, if people actually have some skills to be a bit more engaged in the in the way how product is developed. Miłosz [00:34:13]: It’s harder to hide your incompetence behind some algorithm. and technical stuff. For many product companies, it’s not that important. For the technical skills you have, I ignore it. Miłosz [00:34:30]: You know, data structures you can implement on a whiteboard. They are not that much relevant. They need to deliver something quickly to the market, and iterate over it, and do it over and over. So, if you can communicate well, you can talk with domain experts, you can figure out what they need to build, and you can deliver it. And that’s someone companies probably want to hire. Robert [00:34:54]: And even if somebody will manage to be hired, many companies are also pretty open to let some people go if they will not meet some requirements of how they expect that some people will be operating. Miłosz [00:35:10]: And on the other hand, if you want to be promoted, for example, delivering is the very best. Probably why to do it. So instead of just waiting for requirements and complaining that it doesn’t work, you can try to help. Whoever needs what they need. And in many companies, probably in most companies, this will help you get promoted because they will tell your manager or whoever that you do a great job. It’s really quite simple. Miłosz [00:35:45]: I think we can say it works. Robert [00:35:49]: And we mentioned earlier that nobody is actually teaching us how to do that. So how to work with requirements of product owner. Robert [00:36:01]: Nobody’s giving us some tools how to do it efficiently. I mean, There are a lot of resources how to build classes, how to do architecture. Robert [00:36:12]: How to, I don’t know, use vMotion to be super fast in your IDE. And it’s great, but from other sides, I’m not sure if it’s giving that much return on investment. It’s easier to learn, for sure. Robert [00:36:26]: And you know, it’s coding. It’s—uh— more cool definitely than working with product. But from other sides, I know that many people are also complaining how much time they were spending on meetings. Robert [00:36:41]: Actually, okay, maybe if you’re spending a lot of time on meetings, maybe it’s worth to spend some time to understand what you can do to do it more efficiently because it’s also a skill and often you may say, ‘Okay, it’s the responsibility of the product owner, I don’t care.’ Yes, but unfortunately it often doesn’t work this way. And in many companies that we worked earlier, if we wouldn’t initialize to do some changes if it would work like it was working earlier, like people will be coding monkeys in case. The default mode. Yeah, yeah. So I think it’s important to get some initiative and to go outside of this comfort cage or comfort zone. Oh, it sounded so coaching. Robert [00:37:30]: Thank you. Yeah, it is what it is. I mean, sometimes you need to do some things that you are not that comfortable with, and it’s fine. Miłosz [00:37:37]: Yeah, but I think it’s also important to mention it’s not for… for company’s sake or just because it’s some moral thing to do. It’s basically how you can grow your career this way. Because you become someone who can deliver stuff, and this is what companies want to pay for. Exactly. So I would say do it for your own sake, for your career’s sake. And it will affect your productivity much, much more. Robert [00:38:08]: I can attest that you can have a bit less of bad meetings that you have. And if you have some tools how to do that. Okay, so we’ve already mentioned a couple of times about that we have some tools, techniques earlier, and probably you are waiting to learn what those tools or techniques are. are. So you can try to use them, experiment with them, and maybe improve a bit how you can do that. Miłosz [00:38:35]: So, is there any tool you recommend for this? Robert [00:38:39]: I think I’m not sure if you agree, but I’m quite sure that one of the tools that created most impact in the teams that we’ve been using was brainstorming. When we’ve been working some things that at the beginning was working this default mode close to golden cages, it helped us to… to trigger some discussion between developers and non-developers and to think more how to implement those features. Miłosz [00:39:11]: Yeah, so maybe if someone doesn’t know event storming, we can… Uh, give a quick TLDR: this is a brainstorming session where you gather developers and domain experts in one room and have them use sticky notes. I know how it sounds—yeah, but basically, the idea is to have this lightweight conversation about complex topics so everyone is on the same page. Robert [00:39:42]: Even if it sounds, again, maybe I like, again, some coachy, agile, Scrum Master-ish thing. Robert [00:39:52]: It’s actually a very good technique created by Alberto Brandolini, who is a software engineer. He’s not some agile coach or whatever. Robert [00:40:02]: software engineer, so it’s very close to implementation. So the artwork that you’re creating is very close to implementation and it can be mapped to the code. Nicely and it gives us some nice framework to discuss some functionalities that we’d like to implement. And we’ve been using it in multiple contexts. So we’ve been using it for this smaller planning for these weekly stories, but we’ve been using it also for some bigger strategic features that we’ve been spending, for example, here. And it’s not about creating a big upfront design. It was rather to have some high-level idea on how everything interacts, how we would like to interact with other teams, to see complexity where it’s there. Because very often… There is one problem with stakeholders. They are going to you with some functionality and they are saying that ‘Why so long? Miłosz [00:40:57]: Yeah.’ Miłosz [00:40:58]: At least you have a way to have everyone start to discuss a topic in a different way than just sitting and looking at a presentation. And at first everyone is a bit confused, but then very often you have these lively discussions about the product. Robert [00:41:20]: I can also recommend you to maybe Google event storming so you can see how the artifact at the end looks like. But the idea is that basically you are using notation of domain events. Robert [00:41:32]: In the simplest kind of session and trying to model how our system works. And it’s nice because it’s creating some visible artifact that’s showing the interactions in the system, but it’s also showing the complexity there. I mean, if you’re working on this artifact together, with product owner or stakeholders, it’s kind of closing all those questions: ‘why is it that complicated because if system is complicated, it’s usually visible in the board— that’s a very visual representation. Miłosz [00:42:04]: And this is not a technical representation, right? So it’s not about diagrams or database schemas. Robert [00:42:09]: It’s something that stakeholders can also understand, but it’s also cool because you can also translate it later to your code very directly. Miłosz [00:42:17]: So very often, if you do it for an existing system that has been around for a while, you will see how developers use a bit different names than other people in the company and start to close this gap. Miłosz [00:42:33]: It’s very interesting when you ask your colleagues from other teams to map out how it works from their perspective. Hmm-mmm. It’s like you see a bit different system than you are used to. It’s the same thing, but they use a bit different names. They don’t know all the details. And if you do it together, you get a lot of insight from this. Robert [00:43:01]: And as I said, so for us, it was game changer in multiple things that we’ve been in. And from my perspective, it’s really a shame that it’s not adopted much wider. So, the domain-driven design community is probably best in adaptation of that, so a lot of domain-driven design practitioners are using that by default. I would say that outside of this cycle, it’s not that widely adopted. Again, I think it’s a shame. I’m not sure actually why, but maybe it’s the thing of the forum that for a lot of people it’s like, ‘ah, some agile or coaching stuff, sticky notes again.’ Maybe it’s… because of some cynism. I don’t know. Miłosz [00:43:43]: You also need someone who facilitates this session so it doesn’t end up with… People discussing stuff in smaller groups and not participating together or just going to some off-topic. So it’s not that easy. Always to run it you have to figure out how to do it a bit, but I think it’s worth it. In a way, it’s probably better than having one person speak and everyone sleep on the chairs in the conference room. After the meeting, you decide to run another meeting because you don’t know what to do next. Robert [00:44:20]: I think you need to also have some more open-minded people for that. It’s great if you would like to try that, but you’re not always working in teams when you have good climate for running it. I know how hard it sometimes can be. Robert [00:44:37]: It can be the reason. And in this episode, we’ll not cover into details how to run this kind of sessions, because probably it will be… pretty long. There’s also one very good resource from which we’ve been learning how to do that. This is the ebook from the creator of the technique. It’s named Introducing Event Storming by Alberto Brandolini. We’ll link this in the episode notes. Robert [00:45:07]: This is what I recommend you to check. But we have some tips how to try to convince your teammates, your product manager and people around to try that. Probably the first tip will be to try it maybe on a smaller scale. So don’t start with this big session of planning for a half year, getting your stakeholders from abroad. And don’t start with booking. Miłosz [00:45:38]: plane tickets? Robert [00:45:39]: Yeah, so I would recommend to start with maybe those planning sessions that you are doing with the rest of your team, maybe with product manager. And try to learn how to run those sessions. Because we’ve been running sometimes some bigger session with 20, 30 people. It’s much, much harder than with just your team. Miłosz [00:45:57]: That’s for sure. Especially if you have someone who’s skeptical of this idea, they will just try to discourage everyone to continue because, oh, we know how everything works already. Why do you do it again? Miłosz [00:46:14]: I remember sessions like this. Robert [00:46:16]: But also don’t try to look for excuses like, ‘I’m introvert, I don’t like to do that.’ I’m also, and I did that, and you can just do it. No worries. Miłosz [00:46:25]: It can be tiring, but there is some end goal in all this. You want to figure out the domain. It’s not just a meeting for the meeting’s sake. Robert [00:46:36]: It’s tiring, but from other side, I think it’s also very satisfying when it was successful and later you are sitting to implement that and you can later feel how this implementation is just flowing. And it’s also causing a lot of questions and solving a lot of problems that we’ve been mentioning at the beginning of the episode. Robert [00:46:57]: So, if you would like to start with brainstorming, the first good place to try to adopt that is within your team. So, run some smaller sessions. We recommend in-person sessions since the dynamics are a bit better, but I know that also a lot of people doesn’t have this privilege, let’s say, to work in the office. So, if you’re working in a hybrid way, we would recommend to gather and do this in person because it’s working just much better. Because if you’re doing it on Miro or any other online tool, it’s just harder to maintain everybody’s attention because everybody will try to open some other tab. You can do that, but it’s super hard. Miłosz [00:47:40]: It also happens in in-person meetings when people come to the session with their laptops. Yes, yes, and they are always very confused when you ask them to close them and stand up. Miłosz [00:47:53]: The first sessions are always fun. Robert [00:47:54]: But this is mentioned in the ebook, so we’ll not go super deep into that. And it makes sense. So the first thing you should also discuss it with your team. So you should have some better understanding of what problems you may have with delivering and planning. Robert [00:48:13]: You can use them to try to convince people that. Let’s try to maybe do it in a different way and try to get buy-in from your team to try that. Miłosz [00:48:23]: I know it’s often difficult, and we often get asked how to convince my team to start using and driven design. This is a similar idea here. Miłosz [00:48:35]: I think maybe you can find, if it’s not your entire team, a people like likely-minded as you who want to do something better and try to figure out together how you can improve how you deliver software, basically, and like we’ve been mentioning at the beginning—so, in our podcast, we would like to give you some tools that can help you to be promoted to principal engineer, for example. Robert [00:49:04]: So getting buy-in of people is a very important skill to be principal engineer. Robert [00:49:11]: We believe that if you’re a principal engineer, you’re not the person that is forcing people to do something. It’s rather showing people some benefits of your ideas. They will just commit to your ideas and they will be happy because it’s giving some solution for the problems that you have. Miłosz [00:49:28]: Yeah, so if you want to start, you can just take your team, invite them to a session, try to figure it out together, or you can lead if you feel like it. Miłosz [00:49:44]: Hopefully, you come at something that’s better than your current way of working and this way they see the value. And similar with your… or product person, product owner, or product manager. You can invite them to the session. They should be probably also eager to find better ways to work for all of you because it’s also there in their interest, unless they are, you know, kind of micromanagement freak. Robert [00:50:17]: But from other side, this micromanagement is often coming out of something, and at the end of that, if you… go deeper, it’s ending up that they would like to deliver stuff faster. Miłosz [00:50:28]: If you use this card, it’s nice to yeah and it’s it’s usually quite easy to point out the issues you already have. So you can look back at what happened in your recent plannings, let’s say. You can point that out. But this feature took us two months because of all the issues we didn’t know, of all the gaps in the requirements and so on. Maybe we could do it faster the next time. Maybe we can learn from it. Robert [00:51:07]: It’s also one important argument that is pretty close to the time. So it’s money. I mean, you can say, ‘Okay, last time we one month, which is this amount of money, because this is the average salary of the developer in our teams, if we could spend maybe this next time on planning it better, we could just save this money at the end.’ Miłosz [00:51:34]: Probably the best way to introduce new concepts is to build. Trust first. So, if you just just join the team and you see how that everything is not done the way you would do it— Probably telling the team on the third day that they should reorganize is not a good idea because you don’t know anything about the team and the company. But if you deliver consistently, you build trust, and then people will be more eager to listen to your ideas. You can gradually introduce new ways of working. Miłosz [00:52:16]: So I would start with this. One common issue with meetings or sessions is the never-ending discussions. Miłosz [00:52:26]: They tend to make the meetings hours long. Robert [00:52:30]: And there are some tips about that in the ebook, but I think it’s also worth mentioning because, even if you will read this ebook, it’s easy to get into some never-ending discussions anyway. Miłosz [00:52:44]: Especially if you have… If you run this big picture strategic sessions with 10 or more people in the room and you start mapping the entire flow on the board, people tend to run into discussions because you know, for every system you have in your company, there will be someone who says, ‘Oh yeah, this is the issue I need this fixed’ and Miłosz [00:53:08]: It can take a lot of time and it will derail you from the main topic. So it’s useful to have someone act as facilitator and cut that. the discussions early. Robert [00:53:20]: I’d probably use better work to maybe park it because, if you cut it, somebody may feel like, ‘why are we cutting that?’ It’s probably better. So, what we’ve been doing usually during the session was using the red sticky note. So, we’ve been writing down the problems that we’d like to park, maybe later during the session, maybe after the session, and put it to the board. So, the person that has this problem will feel that, ‘okay, it’s also something that it’s not disappearing, it’s there.’ We won’t forget about it. Yes, yes. But important. Let’s go back to that. Because we’ve already mentioned the trust earlier. So, if you will be parking things and not going back to that later, you may also have problems with this trust part. Miłosz [00:54:07]: Yeah, so later we had separate discussions, maybe in smaller teams. Maybe not everyone needs to be involved in all of the red cards. Maybe some of them we will… decide just are not important and forget about them or ignore them. But for most, we need to revisit and discuss this again. But this way, we will focus on one thing at a time. Robert [00:54:31]: Another thing that I think is also in the ebook, but worth mentioning. So this is the time boxing problem. We’ve already mentioned it earlier, but I think it’s one of the biggest issues. So it’s important to emphasize it and mention it again. Robert [00:54:48]: Getting some balance between cutting discussions but also not skipping discussions that are important. And it’s hard to give some good algorithm how to park some discussion and when to follow it. It’s probably some skill that you need to develop with time, but, as we said earlier, sometimes cutting some discussion may mean that it will go back to you during the implementation and making, changing some. During implementation, it’s much, much more expensive than during planning because you need to go back to planning. Maybe something that you already implemented is not compatible with that and you need to undo that. It’s important to keep it in mind. Experiment. So maybe sometimes try to discuss a bit longer. Robert [00:55:39]: Maybe sometimes try to cut, and also set expectations that, usually, when we’ve been doing some planning sessions, some vendor storming sessions, we’ve been like, ’excuse.’ After that, because it’s it’s been sometimes hours to do that, but at the end it was worth— I mean, very rarely— we had like, ‘okay,’ we’ve been spending on even storming too much, and we designed it too well. Miłosz [00:56:04]: We could better deliver something this time. Miłosz [00:56:08]: This comes a bit back to the topic of cutting scope. If you plan huge features, it will be very difficult to discuss all of it. Yes, but again, cutting scope is not always easy, depending on the project. Robert [00:56:24]: And again, it’s mentioned in the e-book, but in the… book, you can read that event storming has multiple phases. So at the beginning, you’re generating a ton of ideas. And with the time you’re removing more and more of the things that are not relevant. And usually, at the end, maybe not at the end, but close to the end of the session, you should do some reality check and to think: ‘Okay, is what we plant here deliverable? In depends if you’re adding some feature or doing something big, maybe a week, in worst case one month.’ Robert [00:57:04]: If it’s one month, probably it will be in practice a bit longer. But if it’s longer, probably you should look at it and see what we can cut out of that. Robert [00:57:15]: To make it functional, usable, let’s say, and work on this basically. It’s not obvious, but we’re often forgetting about that and we are trying to implement a feature that It should take half a year, and it’s going to be two years later. Miłosz [00:57:34]: Yeah, because no one challenges it. You start planning. Some feature that comes from someone from another team or your CEO or whoever and no one challenges you. Miłosz [00:57:47]: You need the entire thing. Maybe you need just 80% of it or 50%, and it will be much simpler to deliver. Robert [00:57:55]: Okay, so we’ve given you some, let’s say, brief introduction to event storming. Again, it’s not something for episode because I think even one episode will be not enough for that. But if it sounds interesting for you, please leave us a comment. Maybe we’ll do some episodes just about event storming. But. Yeah, I hope that it will give you some overview why it makes sense to try to think out of this thing out of the cage. Miłosz [00:58:29]: Maybe let’s reiterate some ideas. What can someone do if they see this is an issue, if they want to work better with the product? Robert [00:58:40]: Yeah, so I think at the beginning, you should assess in what level of cage you are, let’s name it. So, if you are receiving tasks and it’s assigned to you, or you are maybe closer to the way when you’re planning tasks with your product owner but you’re never discussing with stakeholders. Robert [00:59:05]: Tchum! Think on which level you are. Mm-hmm. Uh, later. Discuss with your teammates if they also see those problems. You should be able to identify the problems with feature planning already. On this episode, so it should give you some weapons. Miłosz [00:59:24]: Yeah, try to figure it out together. Just not try to force. Your teammates to do some of your ideas right away. Try to think together what are the issues you want to improve. Miłosz [00:59:41]: Can write down specific problems you have or even situations. So I remember we had a very issues like this— one task has been on hold for one month because of something, I don’t know, some miscommunication or whatever. Robert [01:00:00]: Or maybe you lost a lot of time on implementing something that used to be useless at the end and your stakeholders said… This is not what we asked about. Miłosz [01:00:10]: Yeah, we delivered something and then it was scraped because no one needed it. And… probably the more specific it is the better because you can then rise it with your manager or product owner or whoever is in charge Instead of complaining that we never get anything done, you tell them, yeah, this feature could have been one week. but we spent two months. Robert [01:00:38]: In other words, use the language and arguments that other side care about. You know about what product managers will care. your teammates probably they care about their car so you can also say that okay maybe would like to if would like to get better job later or be promoted, maybe we should try to think about that. And again, if this convincing people to change approach to use your approach. It’s a very important skill of principal engineers. Robert [01:01:10]: Being principal engineer doesn’t mean that you are forcing people to do something. It means that you are convincing people that your solution will help them with their issues. And that’s it, basically. Miłosz [01:01:21]: And if it really does, then you get trust. Miłosz [01:01:26]: You need to accept that some people just won’t care, and it’s also fine. Don’t try to force them to do anything, whether they are developers or not. And I would instead try to find people who care and try to work with them. So if your product owner doesn’t care, doesn’t want to change how they work, maybe to go to their manager and tell them there’s an issue. Basically, find someone who cares about work getting done in this company. Robert [01:01:58]: If it’s not possible due to political reasons, just maybe do it within your team. Robert [01:02:05]: We don’t recommend that, but it’s better to do event storming session within your technical team without product owner than not doing it at all. Miłosz [01:02:16]: There’s also the last advice we didn’t mention in this episode yet, which is change your job. Miłosz [01:02:22]: It’s always the solution. Robert [01:02:26]: But again, please don’t use it also as an excuse to not include your product manager, because you would be surprised. I mean, how often they care more than you. You think. They just have a bit different goals, but… Miłosz [01:02:39]: And they also might have been burned working with developers in the past. Exactly. If they worked with someone who’s just like, yeah, I don’t care. I will code and give me what to do. They might not be eager to work with you directly. So this goes both ways. Robert [01:02:54]: Yeah, but from our experience, it’s not the case in 96% of cases. It’s very, very rarely a case that it’s… those are people that I don’t care. So please try and you will be really, really, really surprised. Also, I think it’s important to set expectations. Robert [01:03:16]: There are no magic techniques that will make all the planning accurate because ideas will always change during implementation. We are going closer to planning better. Robert [01:03:31]: It will also always take some time to do planning. It’s not magic tool. There is no magic tool that will totally remove meetings. It’s just about making it more efficient. Also be prepared that, okay, if you, for example, decide to use event storming. It will be not super successful from the first session, but it’s also a pretty simple technique, so you’ll be surprised that the result will be pretty good anyway. But also set some expectations. Be prepared to experiment and also improve after each session. Maybe do some evaluation after every session to ask people, okay, did you like that? What you didn’t like? And improve, improve, and improve. Miłosz [01:04:13]: Especially if you change company culture, which is very, very hard to change. And especially if it’s been like this for a long time. Robert [01:04:22]: But it’s possible. We did it more than once and later people were eager to work in this way. So it was cool. And it was also cool later to transform it to other companies and also have some success stories that helped us too. Robert [01:04:39]: Be in different places later. Okay, so I think it will be all for today. So if you have any questions about windstorming, about getting out of the cage or any similar topics, let us know in comments on Spotify. Miłosz [01:04:54]: Yeah, also share your stories maybe. I’m sure there are a lot of stories how projects are run. Let us know in the comments how it works. Works for you. Robert [01:05:05]: If you have any questions, just ask us. We are answering all the comments, so we’ll be also happy to help you. Robert [01:05:14]: Would be interested in listening about event storming, for example. Let us know so we can maybe also record entire episode just about event storming. Miłosz [01:05:23]: Hit that like button and subscribe to the channel so you don’t miss the next episodes. We plan to release a new episode every two weeks now. Robert [01:05:32]: Yep. So, thank you, Miłosz, for your time. And see you in two weeks. Miłosz [01:05:35]: Thank you, Robert. See you. Robert [01:05:36]: Bye.

A DNS outage is like a flash flood. It hits you and disappears as if it were never there. “It was probably the DNS” became a meme. But, half the time, we do not know the exact cause. The outage is seldom long enough to diagnose, sequester, replicate, simulate, and test the failure conditions. How can we be certain that this type of failure will not compromise our systems again in the future? The DNS can fail spectacularly in a variety of ways. A better way to address intermittent failures, such as DNS outages, is to accept them as unavoidable occasional execution disruptions. We might not be able to address each of them exhaustively, but we can design our system to handle almost any interruption of its internal logic. We can harden the system against the disruptions to attain durable execution. What is Durable Execution? Durable execution is a guarantee that a program output is correct regardless of failures anywhere in the execution process. Competitive business applications require durable execution. When it is absent, an employee will spend an hour or two fixing each incorrect output when it is detected. In large business applications, incorrect output often goes undetected for weeks, resulting in expensive chaos around holidays. Are we talking about bugs? No. Bugs are the program. Some intermittent bugs erode durability, but the durability loss is most often a result of an interruption of the execution process. The interruption can be caused by a network outage, service failure, security breach, or a delinquent update. All of those reasons are common. Durable execution does not care what caused it. It is a promise to recover from almost any failure or combination of failures and deliver the correct output. Event-Driven Architecture (EDA) is the industry best practice for supporting durable execution. However, most EDA production systems are not provably durable. What happened? Note In the second guest post on our blog, we welcome Dima Kotik, who contributed the SQLite Pub/Sub backend to Watermill. Please give Dima a warm welcome! 🎉 Miłosz & Robert Durable Execution Requires Immediate Input Storage There is a difference between a guarantee and a wish. A guarantee has some proof, a history of recovery from a variety of system crashes. It begins with emulating crash conditions in integration tests. After writing many such tests, you will realize a simple but profound principle: durable execution begins with writing execution state into storage. There is no other way to proceed, because anything present in random-access memory or ephemeral storage is liable to disappear if the system crashes. The first step of a durable system is to stash its input before any processing takes place after parsing and validation. Security is more important than durability, so input validation takes precedence. Stash the Input After Parsing and Validation Wait, what? I have to commit all the system input immediately into storage? Yes. Is that even possible? Not all at once, but this is not as daunting a task as one might imagine. If you are using event-driven architecture with an publisher/subscriber management library like Watermill, you are already halfway there. Watermill is one of the most popular event buses in the Go ecosystem. Watermill is an unusually flexible library in that it can bridge events from a large number of different backends. Your core business logic may stream events to a PostgreSQL database, and your input may initially be stored in a SQLite databases of your edge nodes to achieve durability without sacrificing performance. Watermill can link the two to tango together seamlessly. Unfortunately, many business applications that utilize Watermill continue to use the Go channel ephemeral backend here and there despite the documentation urging developers not to do it. This backend is a basic in-memory implementation based on Go channels intended for testing purposes. The events are queued into an asynchronous channel. They will disappear when the system crashes. They will clog up the server with a long queue if the execution process slows down for any reason. They will compete for resources with a critical server process. In other words, do not write input into random-access memory. You are asking for trouble. There is an easy persistent drop-in replacement for an ephemeral backend. It is not a silver bullet, but it is an easy first step towards durable computing. Try the new Watermill SQLite backend. It does not require setting up any additional infrastructure, and it is more performant and reliable than any other storage method because it does not need a network connection or a custom transport protocol. It writes to a file on a disk. It does not require CGO. Suppose we are processing book orders, and we would like to store the orders as events in an SQLite database: db, err := sql.Open("sqlite", "orders.sqlite3?journal_mode=WAL&busy_timeout=1000&cache=shared") if err != nil { return err } defer db.Close() db.SetMaxOpenConns(1) // driver limitation publisher, err := wmsqlitemodernc.NewPublisher( db, wmsqlitemodernc.PublisherOptions{ InitializeSchema: true, }, ) msg := message.NewMessage( watermill.NewUUID(), // serialized order event []byte(`{"order_id": "1", "title": "1984", "customer": "Queen Elizabeth II", "address": "Buckingham Palace"}`), ) if err := publisher.Publish("book_orders", msg); err != nil { return err } For cloud instances, ensure that you are writing the SQLite file to a persistent volume, which is mounted explicitly. The root file system is typically ephemeral. You may want to eventually migrate the collected input into your main database after thinking about performance implications. The matching subscriber is not any different from normal Watermill usage other than the backend initialization: subscriber, err := wmsqlitemodernc.NewSubscriber( db, wmsqlitemodernc.SubscriberOptions{ InitializeSchema: true, }, ) if err != nil { return err } messages, err := subscriber.Subscribe(context.Background(), "book_orders") if err != nil { return err } for msg := range messages { // consume messages from the channel processBookOrder(db, msg) } The SQLite publisher and subscriber are drop-in replacements. They can be also used without Watermill router. In other words, you can just use low-level Publish and Subscribe methods. Full examples are provided in documentation. Nothing else needs to change in your code to achieve this first step towards durable execution: write the input events to persistent storage. For complete durability, however, the event handlers for each step of processing the book orders must have two provable qualities: Idempotency: the handler can consume the same event multiple times without generating more than one downstream event. Atomicity: the handler either completes successfully or fails completely. It must never succeed partially. The combination of these properties with persistent storage ensures durable execution. When the a step in business logic fails for any reason, it is attempted again from the last safe system state. Prove Idempotency with Event Duplication Idempotent event handlers must not change system state when processing identical events. In other words, any repetition of input any number of times makes no difference to the system. We can achieve this effect by de-duplicating events, but it is best that the program logic itself tolerates repetition. First, ensure that the event message is acknowledged only after the all the business logic has been executed successfully: func processBookOrder(db *sql.DB, msg *watermill.Message) error { // use only one database transaction per event handler tx, err := db.BeginTx(context.Background(), &sql.TxOptions{ Isolation: sql.LevelReadCommitted, }) if err != nil { return err } defer func() { if err != nil { _ = tx.Rollback() msg.Nack() } else { err = tx.Commit() // acknowledge the message ONLY AFTER successful transaction commit // what happens if the the execution fails here? if err != nil { msg.Ack() } else { msg.Nack() } } }() // process input using the business domain logic // ... return nil } What happens if the transaction succeeds but the message acknowledgment fails on line 16? If the handler is idempotent, the event will be reprocessed, but the system state will not change. An easy way to achieve this is to constrain the database table field order_id to be unique. Then, drop the error if it is caused by a unique constraint violation: // Example for SQLite3 "modernc.org/sqlite/lib" if err != nil { if liteErr, ok := err.(*sqlite.Error); ok { code := liteErr.Code() if code == lib.SQLITE_CONSTRAINT_PRIMARYKEY || code == lib.SQLITE_CONSTRAINT_UNIQUE { // duplicate order id is normal behavior // it means that the event was already processed // ignore this error return nil } } // ... } // Example for PostgreSQL's "github.com/lib/pq" driver if err != nil { const UniqueViolationErr = pq.ErrorCode("23505") if errors.Is(err, UniqueViolationErr) { // duplicate order id is normal behavior // it means that the event was already processed // ignore this error return nil } // ... } What if there are multiple SQL table key constraints? The best solution is to have an idempotency test in place. Then, methodically groom the business logic until it tolerates repetition and passes the test. Use duplicator handler middleware to clone messages and check that the final program state is the same as running the test without the duplicator. Prove Atomicity with Chaos Engineering Once we achieve idempotency, how can we prove atomicity for every step of the business logic process? At-least-once atomic execution is theoretically facilitated by Watermill, but there are subtle ways of subverting it. If you rely on low-level primitives, always acknowledge the message after all the business logic finishes. And, ensure that business logic is granular enough that all of it runs anew in case of an interruption. A simple rule to follow is that all state changes in one event handler must apply within the same database transaction. If you have two database transactions, you will never achieve durable execution. If you are publishing another event inside the event handler, that event publisher must operate within the same transaction as the event handler. The outbox pattern is essential when making any additional changes to the database together with publishing the event. How do you prove atomicity with a test? Write a chaos middleware to pair it with the retry middleware for your testing setup: func ApplyChaos(h message.HandlerFunc) message.HandlerFunc { failNextOperation := true return func(msg *message.Message) ([]*message.Message, error) { result, err := h(msg) if err != nil { return nil, err } failNextOperation = !failNextOperation if failNextOperation { return nil, errors.New("world chaos cancelled the event handler") } return result, nil } } This middleware will break the event handlers half the time. The retry middleware should ensure that the operations complete even with half of the system experiencing outages. How do you prove that an event handler is atomic? Write another middleware that pollutes the message context: func PolluteContextWithChaos(h message.HandlerFunc) message.HandlerFunc { return func(msg *message.Message) ([]*message.Message, error) { polluted, cancel := context.WithCancel(msg.Context()) defer cancel() go func() { <-time.After(time.Millisecond) cancel() // simulate premature cancellation }() msg.SetContext() return h(msg) } } If all your input and output operations are context-aware (they should be - it is best practice), the premature context cancellation will simulate an execution interruption inside the event handler. This simple example cancels after one millisecond. Simulate a variety of pseudo-random cancellation patterns. Never use true random, because it will make the tests brittle. Check that the output is the same, and you are well on your way to a durable system. Conclusion Business applications need durable execution. Programs that have an innate ability to recover from almost any error or crash are a joy to work with. Durability can be achieved with proper system design and a rigorous testing discipline. You can see how durable execution can get complicated quickly, but the first step is the easiest. Write the input event to a persistent volume. It will give you some durability right away. You can work out the rest of it later. SQLite Pub/Sub backend for Watermill is a great choice to try this. This new driver works without CGO and, like Watermill, is free and open source. Cloud providers have begun to offer distributed SQLite clusters for use cases that offer some durability and the fastest possible read speeds. Those projects are a strong indicator that SQLite-powered storage is a viable and increasingly desirable option for durable execution: LiteFS keeps all the classic SQLite conventions like the single-writer constraint. This gives good reasons to anticipate that in the future durable execution could also become fast even with global transactions. Turso comfortably runs on edge or mobile devices with poor connectivity. Cloudflare durable objects are an SQLite persistence companion to Cloudflare Workers. There are two disadvantages to those commerical offerings compared to Watermill over SQLite Pub/Sub: (1) the vendor lock-in and (2) some difficulties in binding database transactions to event handlers reliably. This means that Watermill might get you farther past persistence towards idempotency and atomicity on the journey to provable durable execution. Give it a try!

It’s been almost a year since the last Watermill release post, and we have a bunch of news to share! In case you’re new here, Watermill is a Go library for building event-driven applications the easy way. It’s like an HTTP router but for messages. Watermill is a library, not a framework, so there’s no vendor lock-in. Watermill is now close to 9k GitHub stars and 79 contributors, just on the main repository. It supports 13 Pub/Subs including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS, SQL, and RabbitMQ. See the documentation for more details. Let’s look at what’s new. SQLite Pub/Sub Watermill now officially supports the SQLite Pub/Sub. The messages are moved between publishers and subscribers using tables in an SQLite database. A huge thank you to Dima Kotik who contributed the new package: github.com/ThreeDotsLabs/watermill-sqlite. See the documentation. SQLite Pub/Sub is simple to set up, as you don’t need to run any external services: everything is just a file on disk. It supports persistence, so you don’t lose messages when your application restarts. So far, we recommended PostgreSQL or MySQL Pub/Sub for when you don’t want to set up a separate message broker. Now it’s even easier to start with SQLite. It can also be a great alternative to using the GoChannel Pub/Sub in smaller projects. We’ll soon publish Dima’s article about when it shines. Stay tuned! The implementation has two packages: modernc.org/sqlite and zombiezen.com/go/sqlite. You can choose either one when creating the Pub/Sub. Here’s an example of creating a Subscriber with the modernc driver: import "github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc" // ... subscriber, err := wmsqlitemodernc.NewSubscriber( db, wmsqlitemodernc.SubscriberOptions{ InitializeSchema: true, Logger: logger, }, ) messages, err := subscriber.Subscribe(ctx, "messages") for msg := range messages { fmt.Printf("Received message: %s, payload: %s\n", msg.UUID, string(msg.Payload)) msg.Ack() } And then publishing messages: import "github.com/ThreeDotsLabs/watermill-sqlite/wmsqlitemodernc" // ... publisher, err := wmsqlitemodernc.NewPublisher( db, wmsqlitemodernc.PublisherOptions{ InitializeSchema: true, Logger: logger, }, ) msg := message.NewMessage(watermill.NewUUID(), []byte(`{"message": "Hello, world!"}`)) err := publisher.Publish("messages", msg) We consider this a beta release: the full test suite passes (and it’s pretty extensive!), but we want to gather more feedback from real-world usage. If you try it, please share your experience. You can use the main repository’s issues or join our Discord. Watermill Quickstart We recently launched Watermill Quickstart, a free hands-on training that teaches you how to use Watermill in your project. We keep looking for the most effective way to learn programming topics. It’s clear that reading docs and books is not enough: you learn best when you actually practice by writing code, failing, and fixing it. The common advice for newcomers in any technology is to “create a pet project.” But coming up with an idea and sticking with it long enough to learn something isn’t trivial. The Quickstart guides you to build this kind of project. You code in your own IDE, so the next time you want to use what you learned, you already know the setup and have the experience. If you try it, let us know what you think! Watermill Quickstart Watermill v1.5 While there are no major new features in the main Watermill package, we reviewed a ton of PRs and merged many improvements and fixes. You can see the releases page and the full changeset on GitHub. Thank you to everyone who contributed, and please keep the PRs coming! Let’s look at some API changes and deprecations. CQRS Marshaler updates cqrs.ProtobufMarshaler is now deprecated and replaced with cqrs.ProtoMarshaler. Ideally, we’d remove it completely to get rid of the github.com/gogo/protobuf dependency, but that would be a breaking change. We recommend using ProtoMarshaler, which uses google.golang.org/protobuf instead. There is now an easier way to decorate marshallers. If you want to add some extra logic when marshaling or unmarshaling commands or events, you usually wrap the marshaler in a custom struct. Now you can use cqrs.CommandEventMarshalerDecorator to do that with less boilerplate. cqrsMarshaler := cqrs.CommandEventMarshalerDecorator{ CommandEventMarshaler: cqrs.JSONMarshaler{}, DecorateFunc: func(v any, msg *message.Message) error { event, ok := v.(Event) if !ok { return fmt.Errorf("%T does not implement Event and can't be marshaled", v) } partitionKey := event.PartitionKey() if partitionKey == "" { return fmt.Errorf("PartitionKey is empty") } msg.Metadata.Set("partition_key", partitionKey) return nil }, } Small API changes You can now create messages with context using message.NewMessageWithContext. msg := message.NewMessageWithContext(ctx, watermill.NewUUID(), payload) This is a trivial addition, but we hope it’ll help make it clear that it’s the message that carries the context. For newcomers, it can be confusing that Publish() takes no context argument. This API is similar to net/http with NewRequestWithContext. A tiny change that should improve the developer experience. Similarly, we deprecated the Router’s AddNoPublisherHandler in favor of AddConsumerHandler. When we initially designed the Router, we assumed the default behavior of handlers would be to always publish messages. In practice, it’s the opposite: most handlers consume messages and don’t publish anything, or publish the message in other ways. AddNoPublisherHandler sounds a bit awkward, so we decided to rename it. Context in GoChannel Pub/Sub There is now an option to propagate the message context to the handler when using the GoChannel Pub/Sub. This has been something many people have asked for, especially for passing some arbitrary values through the context. You can enable it by setting the PreserveContext field to true. pubSub := gochannel.NewGoChannel( gochannel.Config{ PreserveContext: true, }, logger, ) Since GoChannel works in-memory, you can access the context you set when creating the message in the handler. However, keep in mind this is not how most Pub/Subs work. When using Kafka, Redis, or any other “proper” Pub/Sub, the context is lost when the message is sent over the network. Use it only if you know what you’re doing! If you switch to another Pub/Sub later, this will break. Watermill SQL v4 We finally released watermill-sql v4.0.0 after a long time as a release candidate. You can now use it with packages other than database/sql, like pgx (Postgres driver) or ORMs. This includes support for transactions, which is key for the Outbox pattern. Shoutout to @julesjcraske for contributing these changes! Subscriber and Publisher constructors now take a Beginner interface argument instead of *sql.DB. subscriber, err := sql.NewSubscriber( sql.BeginnerFromStdSQL(db), sql.SubscriberConfig{ SchemaAdapter: sql.DefaultMySQLSchema{}, OffsetsAdapter: sql.DefaultMySQLOffsetsAdapter{}, InitializeSchema: true, }, logger, ) Migration: Wrap your *sql.DB in sql.BeginnerFromStdSQL(db) when passing it to the watermill-sql API. Pgx has its own BeginnerFromPgx(db) constructor. When working with transactions, use TxFromStdSQL(tx) or TxFromPgx(tx) instead. If you use another driver or ORM, you can implement your own adapter that implements the Beginner or Tx interface. See the implementation. The other major change is that we decided to make SchemaAdapter and OffsetsAdapter interfaces more flexible. Instead of raw arguments, they now take a params struct and always return an error. We will stick to this approach for most APIs going forward. While more verbose, it’s easier to extend without breaking changes. type SchemaAdapter interface { InsertQuery(params InsertQueryParams) (Query, error) SelectQuery(params SelectQueryParams) (Query, error) UnmarshalMessage(params UnmarshalMessageParams) (Row, error) SchemaInitializingQueries(params SchemaInitializingQueriesParams) ([]Query, error) SubscribeIsolationLevel() sql.IsolationLevel } type OffsetsAdapter interface { AckMessageQuery(params AckMessageQueryParams) (Query, error) ConsumedMessageQuery(params ConsumedMessageQueryParams) (Query, error) NextOffsetQuery(params NextOffsetQueryParams) (Query, error) SchemaInitializingQueries(params OffsetsSchemaInitializingQueriesParams) ([]Query, error) BeforeSubscribingQueries(params BeforeSubscribingQueriesParams) ([]Query, error) } Migration: If you use the default schema or offsets adapters, you don’t need to change anything. If you have custom implementations, update the method signatures to match the new ones. Watermill GoogleCloud v2 The Go Pub/Sub client library v1 will lose support next year. We updated watermill-googlecloud to use the new v2 library. There’s only one breaking change: SubscriberConfig no longer exposes the SubscriptionConfig field. Instead, it now has the GenerateSubscription function. The function returns *pubsubpb.Subscription, which mostly maps 1:1 with the old SubscriptionConfig. Migration: Replace SubscriptionConfig with GenerateSubscription. googlecloud.SubscriberConfig{ ProjectID: "tests", GenerateSubscriptionName: subscriptionName, - SubscriptionConfig: pubsub.SubscriptionConfig{ - RetainAckedMessages: false, - EnableMessageOrdering: enableMessageOrdering, - }, + GenerateSubscription: func(params googlecloud.GenerateSubscriptionParams) *pubsubpb.Subscription { + return &pubsubpb.Subscription{ + RetainAckedMessages: false, + EnableMessageOrdering: enableMessageOrdering, + } + }, Unmarshaler: unmarshaler, }, We considered mapping the config to make this change non-breaking. While it’s possible, we wouldn’t get rid of the old dependency, and we’ve seen reports that the old and new clients don’t work well together. So this means a major version bump, but the migration should be straightforward. Hall of Fame 🏆 We had 43 unique contributors to all Watermill repositories since the last release post! 💪 Thank you all for your help! @Abdullah-AlAttar @abramovd @AceFire6 @addshore @alexandregv @Amr-Shams @avlajcic-axilis @chaindead @chengehe @co60ca @dkotik @edebernis @ekazakas @EvanMad @HansvdLaanNedap @HolyKingCrusader @jgersdorf @jjcollinge @jlevesy @jpradass @julesjcraske @marczahnfc @matdurand @minxinyi @NathanBaulch @nkonev @Orbsynated @petergere @pxwanglu @rhhammond @samartha-kar23 @seamusv @stephanvanzwienen @sysradium @thpk @tjnet @unaik @weiye-Lee @whosehang @xico42 @yashb042 @ygaros @ytnsym Docs refresh We updated watermill.io a bit, so it’s easier to tell what the library is about. It should also be easier to choose the proper learning path. Once again, give the Quickstart a try and let us know how it went. We consider making free training like this for other open-source libraries as well. If you have any suggestions, please share!

The peak of hype isn’t the best moment to reflect on AI. Will it take your job, or is it the next fad like NFTs? Are AI startups ridiculously overvalued, or are the companies that sleep on AI doomed? Time will tell. LLMs are far from perfect, but I’m excited they’re here anyway. Not because of a silly promise to make me 10x more productive, but because they can solve some problems that were previously unsolvable. If you enjoy building software like me and Robert, it’s a new tool worth trying. And there’s a chance AI experience helps you create a better product or land a good job. Did you ever get stuck debugging your project at 2 AM, and went to sleep frustrated? We spent a few weeks adding an AI Mentor to our learning platform to help our students avoid this. As always, it wasn’t as easy as most tutorials show. Trivial use cases, like a weather-checking assistant, are fun to make, but aren’t much more useful than the default chat interface of any model. I’ve heard before that implementing an AI feature is easy, but making it work correctly and reliably is the hard part. You can quickly build an impressive demo, but it’ll be far from production grade. We can now confirm this is true. Plus, we’ve seen something not talked about that much: how an AI feature can fail in production and mislead users. I want to share what we learned in detail, hopefully saving you time on research. Let’s set aside the world-changing stories for a moment and see what we can build with this stuff. Note If you’re just starting out I know many developers stay away from AI because they consider it overhyped, unethical, or useless. They also feel massive FOMO because there’s currently no other topic in tech. If you feel this way, I’m not trying to change your mind. I encourage you to approach it like an engineer trying out a new technology. Look for what problems it can solve, while most people are busy applying AI to everything. Once you use it in practice, you’ll have a good idea where it’s useful and where it falls short. You’ll make up your mind much better this way than reading hot takes from CEOs and tech bros. Why AI Mentor? We’re not VC-backed and don’t seek to sell our company, so there’s little value in slapping “AI-First” on our homepage. We wanted to use LLMs to solve a real problem our users had. Our Academy platform is all about learning by building real-life projects. One of our challenges is helping students finish the training. If you have ever tried an online course, you may know this problem — you put it off after a few days and never return, distracted by something else. One reason to put learning off is that you get stuck. While working on complete projects, it can be difficult to spot a bug. Example solutions don’t always help, since each project is unique. We help students move forward by reviewing their solutions. Often, it’s a small typo or mistake that’s easy to catch with fresh eyes. There’s no learning value in fighting with such bugs for hours. Frustrated and stuck, some people give up. Last year, we thought LLMs could help. You can feed them entire codebases and ask them to point out or even fix errors. And they are always available, in contrast to our limited time for customer support. Back then, the models were expensive, far less impressive (GPT-4 had just been released), and there were few sources to learn from. But most of all, we lacked time and postponed the project. We came back to it earlier this year. (If you want to see it live, the first few modules of Go in One Evening require no login.) Milestones We decided to start with an MVP and focus on the fundamentals, so we’re not distracted by nice-to-haves. We picked a few milestones and planned to implement each one in production before moving on to the next one. A single “Help me!” button on the website for Go in One Evening. Users would receive a single message with a hint on what’s wrong with their code. Conversations. While a chatbot wasn’t our main goal, asking a follow-up question about the solution could be helpful. Enable the Mentor for Go Event-Driven. Go in One Evening is a simpler training with shorter exercises. We assumed solving the issues found there would be easier compared to Go Event-Driven, where students build more complex projects. (The final project is about 4k lines of Go code.) The mockup of our initial idea. The end result. The “Help Me!” Button We created a proof of concept that should be easy to extend. We started with two HTTP endpoints: a POST for requesting help (for now with a constant “Help me” message) a GET streaming all messages over Server-Sent Events (SSE). In theory, a single endpoint could do both, but we wanted it to be asynchronous by default. Users can click the button and refresh the page without losing the response. Separating writes from reads is a good practice, especially for long-running tasks. The downside is a bit more complex setup, but we use proven tools that we know well. We prefer event-driven patterns for asynchronous architecture. We use Watermill and Google Cloud Pub/Sub to trigger the Mentor and send updates back to the user. All of our events end up in BigQuery, so we get storage for conversations out of the box. This is important because we need the history of messages to include in the next prompts. High-level overview. This seemingly simple setup already has some challenges, even without considering the AI part: How to handle race conditions between the frontend app, the Pub/Sub, and the SSE handler? How to handle errors and retries? How to deal with BigQuery’s insert and select latency? How to approach message ordering? Events Event-driven patterns turned out to be a good fit for working with LLMs. Behind the scenes, there are many moving parts, and most need no input from the user. Failures and retries often happen, but they are all hidden behind a single response. It’s why we don’t recommend using a single HTTP handler for the whole process. There are delays and interruptions, and you need to handle them, so your users don’t see the mess. Most LLM APIs support streaming chunks of messages. You can show them on the UI before the model has finished generating the response. SSE is a perfect fit for this. The first response of the SSE handler is a list of all messages (for the given user and exercise). Then, we stream all chunks over the Pub/Sub to the SSE handler and then to the browser. We had a great experience working with SSE before, and watermill-http supports it out of the box, so it integrates nicely with our architecture. We’ve improved the implementation by adding support for generics (although it’s not public yet, we may release it at some point). It’s all hidden behind a high-level interface. The HTTP handler receives a channel of incoming events and a channel to send responses back to the client. func (h *Handler) HandleEventStream( r *http.Request, events <-chan *events.MentorMessageSent, responses chan<- any, ) { // Send the initial response with all messages messages, _ := h.getAllMessages(r) for _, msg := range messages { responses <- msg } // Process incoming events for event := range events { // ... responses <- response{} } } Note If you want to learn more about Server-Sent Events, we have a dedicated post about implementing it in depth: Live website updates with Go, SSE, and htmx. Calling LLMs We initially used langchaingo for calling the LLMs, but the API felt a bit bloated for our needs. We created a small wrapper on top of the official SDKs to be able to switch the models quickly. The design of most LLM APIs is very similar. You can describe a prompt call like this: type Model interface { Prompt(history []Message) (Message, error) } It’s a function that accepts a chat history and returns the next message in the chat. In practice, we ended up with a few more details: func (l *LLM) Prompt( ctx context.Context, history []Message, user User, opts PromptOpts, ) (*Output, error) { // ... } type PromptOpts struct { MaxOutputTokens int Model ModelName Tools []ToolSchema } func (o *Output) Listen() (<-chan Chunk, error) { // ... } func (o *Output) WaitAndGetAll(ctx context.Context) (string, error) { // ... } This is a tiny abstraction over the LLM API, just enough to make it easy to switch between models. Similarly to SSE, Go channels are the perfect API to stream chunks. It works well with the whole asynchronous model, and it’s great to have support for it on the language level. Prompts & Context Engineering We spent a lot of time iterating over the prompts and the context we fed to the model. For example, we wanted the Mentor to hint at the right direction, but not simply give the complete answer. We ended up with quite a long prompt for something as simple as “Help with this exercise”. We have many files related to each exercise (the user’s solution, an example solution, and many others). Even though models now easily handle long contexts, we still need to be careful about how much we feed them. Deciding what context to provide needs careful balance. If you dump too much and the context window grows, the prompts get more expensive, and the models are less likely to follow the instructions. If you provide too little, the model may not understand the problem and give a useless answer. There are really no rules that work for every case. One thing we found helpful is strong language in the prompt. The models seem to follow wording like “ALWAYS” or “NEVER” better than “should” or “shouldn’t”. Free-form chat After the initial response, users can ask about the solution or other things related to the exercise. This part was quite straightforward since we had already made the MVP ready to work with threads of messages. The key here is getting the historical messages to include in the future prompts. In our case, we fetch the chat history from BigQuery. We retrieve what we can concurrently, so the time to the first response chunk is short. Solving Complex Projects This approach worked pretty well, even though we used a single prompt. For shorter code snippets, it could guide students on what’s wrong with the solution. But for complex exercises, it failed miserably. With big projects, the model often made mistakes. It tried to fix issues that didn’t exist or completely missed what was wrong. We improved it a bit with better prompts and more context about common concepts. But for the most complex examples, it wasn’t enough. So we experimented with a more advanced approach, which you could consider an agentic system. Fixing the Solution The tricky part of generating hints is that we can’t be sure whether the model is right. It’s how your AI feature can (and likely will) fail in production. To counter this, you need to be able to verify the model’s output. In our case, we built a system similar to coding agents like Claude Code. We use it to fix the user’s solution before it gives any hints. If it succeeds, we have some confidence that the model is on the right track. This approach solved the issue of the model making wrong assumptions. If it tried one approach, and it didn’t work, it would realize the issue is elsewhere. Among other things, the agent can edit the user’s submitted files. These changes happen only in the Mentor’s context — the user doesn’t see them. It also plans what to do, can summarize the changes, and a few other tricks. Different models excel at different tasks, so we can use the best one for each step. In the code, an agent is pretty much a for loop plus tool uses. It’s a bit like a regular chatbot, except the model “talks” to itself. You keep the history and add messages with feedback on the environment. The context grows with each iteration, so it makes sense to limit the attempts. Parallel Runs The new approach sometimes brilliantly succeeded. But other times it was stuck, failing to edit a file, or just not getting the original issue. The good part was that it could now tell the user it didn’t know the solution, rather than making things up. That’s a huge improvement — if someone is already stuck, we don’t want to mislead them further. But it wasn’t able to fix the solution on each attempt. You had to retry a few times if it failed. We noticed that if the model didn’t succeed in the first few attempts, it would only get worse. It made more sense to start over. So we started running a few agents in parallel, each trying to fix the solution. Once one succeeds, we stop the rest. Managing concurrent code like this is a pretty common pattern in Go. We feel it’s a great fit for AI engineering thanks to its concurrency model (compared to other languages, like Python). wg := sync.WaitGroup{} wg.Add(workers) for i := 0; i < workers; i++ { go func() { defer wg.Done() for cfg := range configsChan { ctx, cancel := context.WithTimeout(ctx, workerTimeout) defer cancel() select { case <-ctx.Done(): return default: } result <- Solve(ctx, mentorContext) } }() } go func() { wg.Wait() close(result) }() for { select { case <-ctx.Done(): return nil, ctx.Err() case r, ok := <-result: if !ok { return nil, ErrCantHelp } if r != nil { return r, nil } } } We use many combinations of models to increase the chances, hence the configsChan. Different models excel at different tasks, so we can use the best one for each step. Also, if a cheaper model manages to finish the task fast, we can save some money on the expensive ones. Often, a single call couldn’t reliably figure out the solution. This shotgun approach improved the chances. The tradeoff is cost: we pay for all started parallel runs, even if we finish them early. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! QA: Tests & Evals In many projects, testing is an afterthought. It turns out it’s crucial when working with LLMs. And testing AI products can be a nightmare. It doesn’t seem that different at first. Calling a model is similar to using any other API, so you need to prepare test cases with inputs along with the expected outputs. But there are some caveats. Challenge #1 is judging the responses. An optimistic scenario is when you make the model return a structured output, with a well-known list of valid values. Good old unit tests can work well if you have enough test cases. But if you let the model respond in free-form text, like in most chatbots, it quickly gets complex. You need an evals system where you judge if the output is correct. You could either use a human to do this or another LLM call. Whatever you choose, expect to read lots of data. You need a realistic set of inputs and outputs, and you need to review and label them. Once we kicked off the project, we spent a few days manually going through possible test cases based on production data. It seemed unproductive at first, but if we skipped this part, we would have no idea if what we built works. It also helped us understand the problems people actually face. Challenge #2 is time. If you want to use the smartest or thinking models, a request can easily take tens of seconds. But even with faster models, running a few prompts in a sequence is far from unit test speed. This slow feedback loop is annoying to work with, especially if you tweak the prompt slightly and want to see the impact right away. Challenge #3 is cost. The best models are also the most expensive. If your context grows (some of our evals have context with 100k+ tokens), each test run costs real money. Because of #2 and #3, forget about running a big set of evals in CI on every commit (unless you have too much VC money to burn). You can run some of them, or do it only once in a while. In our case, the trivial test cases (fast and cheap) are not that interesting, because even the less capable models solve them easily. The longest and the most expensive ones are what we care about the most. In regular tests, you can mock the expensive or slow resources. It makes little sense here, as you need to validate the same models you’ll run in production. Since we run parallel workers for each mentor call, it also impacts the evals. While not a big difference in the production environment, it makes evals much more expensive. It was also easy to hit the API limits. So, in the evals, we run the workers sequentially, starting with the cheapest ones. It slows down the tests, but saves us some money (this makes a difference, especially for bigger contexts). On top of all this, LLMs are unpredictable, so you may want to run a few iterations of each eval for more realistic results. This multiplies both time and cost. You need to balance many factors. We didn’t use any framework, but a script written in Go that runs the evals and collects the results. Because evals take a long time to run and could be interrupted by some error, we implemented a cache and retry mechanism, similar to what go test does. It’s frustrating to spend five minutes waiting for tests, only to realize you forgot a simple change and have to start over. A few of the evals we run for the Mentor. Failing in production After rolling out the first version of the Mentor, we monitored the responses to see how it did. Shortly after, came this: Mentor hallucinating. This is wrong! I started sweating when I saw this. People pay us to teach them, so it’s a huge risk if they learn obviously wrong things. In this case, the student tried to understand if assigning a slice to another slice in Go results in a deep copy. The model happily agreed, even though it’s not true. We take teaching people seriously. I almost wanted to scrape the entire feature when this happened. We contacted the student right away to clarify the issue. Thankfully, they appreciated the detailed explanation and weren’t angry. To counter this, we switched to a better model for conversations and added more context to the prompts (details below). The worst part about hallucinations is that the model is very confident about being right. When you just start working with LLMs, you’re surprised how often it happens. You need to be ready for it, because it WILL happen in production. This is why I’m not comfortable learning something in-depth just from an LLM. If it’s something you can quickly check, it’s not a big deal. But learning fundamentals the wrong way is another story. RAG and Sources Initially, we assumed we didn’t need to use RAG (Retrieval-Augmented Generation), and we simply always added some documents to the context. But after the first issues, we decided to implement a simple RAG with things like Go documentation, our blog posts, and other resources. We have a simple database of content in Markdown format. The RAG hype presented it as some mindblowing technology, but it was rather simple to implement. It took us just two days to have an MVP in production. It helped that we could split the work and implement two parts in parallel: building the database and querying it in the mentor code. As a bonus, we also link to the sources in the response, so students can read more about the topic. The list of useful links. Predictability If you expect a structured output from the model, like “true” or “false”, or a set of answers, testing is much easier. But dealing with free-form answers, like with most chatbots, is entirely different. One day, I started tweaking the temperature setting, assuming lower values should work well for our needs. Then, I realized there’s no way to tell unless I compare multiple responses manually or write an eval that focuses on this difference. And even then, I’m not sure if the reason was my change. It’s very difficult to get predictable results from LLMs. Small changes in the prompt have effects on the whole pipeline. The more complex the system, the less predictable the outcome, and the harder it is to test. After you change the prompt, you need to run the evals. But probably not all, since they are slow and expensive. But then, if you test only a few, you don’t know if it covers all cases. If the response changes, you can’t be sure it’s due to your changes. Maybe it’s just a different random response. Or maybe the model has been updated overnight (you have no visibility into it). To have a better idea, you could run more iterations of each eval, but this increases both time and costs. The developer experience of changing the prompt and waiting for results is quite poor. It’s nothing new: it’s like working with a slow or flaky CI/CD pipeline, where you commit changes and can’t spend the next five minutes on anything productive. The context switch is painful. You can counter this with parallel runs, but you can hit usage limits this way, and it’s then easy to burn money within a few seconds. How you model your evals is key. We ended up with two kinds of evals: Strict evals, where we expect a given percentage of scenarios to pass. We can use them reliably in the CI pipelines. Graded evals, where we run a set of scenarios and manually compare how the results changed according to grading. Shifting the Mental Model How do you cope with software that doesn’t follow instructions? Often, you like the LLM’s output 90% of the time, but the remaining answers are confusing or outright wrong. This can be frustrating if you’re used to working with code. There are many ways to reduce hallucinations, but there’s no way to avoid them completely. Realizing this helped me change how I think of LLMs. I see people go to two extremes: Treating LLMs as sentient beings, trusting the model to do exactly what they ask, if the prompt is good enough: “Bro, I found a way to make GPT never lie to me, just use this prompt.” Skeptical about using LLMs entirely, making fun of prompt engineers having to beg the model to do what they want. A middle ground I found is to treat an LLM as a function that is excellent at parsing vague inputs, and the output is often brilliant, but sometimes nonsense. I treat prompts as code, but with less strict rules than I was used to. With this mindset, it’s easier to be pragmatic and remember not to blindly trust the output. But also not getting discouraged after something breaks. ✅ Tactic: Expect LLMs to fail LLMs are impressive but unpredictable. Don’t treat them like the code you’re used to. Be ready for failures and hallucinations. If you can, verify the output. Agentic systems or autonomous agents? The top 2025 trend is running autonomous agents that make decisions independently. But thinking of an agent as a sentient being that can figure out what to do in every scenario is a trap. Code always works in contrast to an unpredictable LLM. We didn’t need to go all in on the agent running all workflows by itself. For example, we can just decide (in code) what steps the coding agent should take next. When writing regular code, you need to control the agent loop more directly. It’s tempting to think you can just include in the prompt what the agent should do in every scenario. But the reality is, you have no guarantee it will follow your instructions exactly. Our agent isn’t super complex, but it doesn’t need to be. If you can predict when to hand over to regular code or another LLM call, just do it in the code. No need for an agent framework where agents know about each other and can decide who to “talk” to. Especially if you can’t verify the output, how can you trust the agent to make the right decisions? This agentic hype feels like the early days of microservices when we told ourselves the story of independent entities talking to each other. Then everyone jumped to implement the solution, without considering whether it actually solved anything. Many teams learned the hard way that distributed systems are difficult to get right. Even with classic, perfectly predictable code. I’m not eager to add unpredictable models to the mix. ✅ Tactic: Use LLMs where they shine If you can handle the problem with regular code, there’s no need to use LLMs. Don’t think of agents as sentient beings that can figure everything out by themselves. Where is the complexity? The length of this post shows how much goes into one AI feature, and not all of this effort is related to LLMs. Most of it is good old software engineering and product design. I will risk saying that building the Mentor was 80% regular software development and 20% AI-specific parts. And 80% of the latter was creating and running evals. Most of the complexity is in the orchestration. All well-known patterns and good practices still apply. This agent-orchestrating code is your domain logic. Once again, Go shines for such use cases. It’s trivial to run concurrent workflows, and channels are a great way to communicate between different parts of the system. The logic is easy to follow. (And Go is easy to learn!) Models We accidentally picked a hot moment in model releases. We started with Gemini 2.0 Flash — it had a 1 million token context window, a generous free tier, and was relatively cheap overall. Then, Gemini 2.5 Pro came and made Flash look silly. It had a much better understanding of code. Then GPT-4.1-mini came and made Pro look expensive and slow. In retrospect, it’s good we didn’t implement this feature one year ago, since the models got so much better over time. Picking the right model is a balance between speed, cost, and quality. You need to be able to switch models easily for tests, so make sure your system is not tightly coupled to a specific model. We eventually landed on OpenRouter. You can use the Go OpenAI SDK to call any provider’s model, which is great by itself. It also has no strict rate limits (we hit limits running evals on OpenAI). Finally, it centralizes billing and lets you set limits and budgets (something unheard of for Gemini — crazy!). The downside is some extra cost and latency. Limits & Costs One feature of our internal LLM library is limits. We designed it so that calling an LLM is not possible without using the limits. Even tests track limits in-memory. It’s too easy to lose track of how many tokens you use. One day, we spent $100 running evals. After that, we made the cost clearly visible in all of our tooling, including tests running in the CI. A cost spike because of running evals. Gemini is an especially awful experience. There’s no way to set budget limits, and the usage shows up with a delay. Thankfully, other providers offer basic services like budgets or prepaid payments. As mentioned above, OpenRouter also works great for this. We can set limits on a few different levels. The primary one is per user. It’s not a perfect solution, since more complex projects can quickly exceed the limit. It’s something we will try to improve in the future. The costs of running in production so far are just a tiny fraction of our infrastructure costs. The surprisingly expensive part has been running the evals. The faster you want them to run, the easier it is to burn money quickly. Thankfully, this happens only once in a while when we change the prompts or add new evals. Observability & Tooling As in any complex system, debugging AI features is difficult. Good tooling helps. Tracing is a great way to understand how the system works. We didn’t use anything AI-specific for tracing, as we like simple tech stacks and proven technology. We already use tracing in our project, so it was just a matter of adding spans and propagating them. A cool thing about tracing is how it shows which parts run concurrently. We also created a back-office dashboard to track the cost and usage, and a set of CLI tools to easily manage evals. Moderator We wanted the Mentor to avoid unrelated topics, as it was more likely to give incorrect answers then. We set some guardrails in the prompts, but it was easy to work around them. Our second approach was a separate moderator prompt. It has two functions: filter out irrelevant topics and decide if the user needs help with the exercise or just asks a question. If it’s just a question, we don’t need to fix their solution, so it’s a much simpler and faster prompt. It’s a classic example of how asking the model to do too much at once is problematic. Similar to classic software, the separation of concerns is a good idea. One prompt should focus on one task. Moderator stopping the conversation early. Encouraging students to ask for help Implementing the Mentor was one thing, but it wouldn’t be effective if students didn’t know it existed. People are already fed up with companies shoving AI features their way, and since most chatbots are dumb, it’s not tempting to click the button. We track the failures and show small hints in the CLI. Plus, we use the same SSE endpoint to stream hints and show an indicator on the website. Sometimes, students still ask us directly for help. We don’t want to point them straight to the Mentor, as our support is part of the product. Now, we can use the Mentor ourselves to find what the issue is, and then we just need to explain it to the student. Finding a small typo is much easier this way. The UI We’re not frontend experts, so I will shortly mention the UI parts for completeness. We use React and have vibe-coded most of the Mentor’s frontend code. We’ve been working with frontend for many years, but it’s not something we enjoy. Generating the code was a great way to start: we understand enough to know what we want, but we don’t have to write it ourselves. We find AI-assisted coding works well for software that has no “strict behavior”, and you can quickly tell if it works, like UI and video games. For example, if the generated CSS is silly, but looks fine, it’s not a big deal. Especially if it’s just a single component. Of course, if you go too far with it, it can turn your project into a mess. We use a polyfill for SSE. It supports features not in the official spec, like passing custom headers in the API (Authentication), and it works out of the box. One thing to consider with a chat interface is that you need to animate the chunks when streaming (delay the letters slightly). If you make them appear as they come, it doesn’t look as good. At some point, we added status messages to show that the Mentor is working on the response. Sometimes it takes a few minutes, so it’s easy to lose patience and think it stopped working. These messages are not fake — we stream them from the backend via the same SSE endpoint. Outcomes We ended up with something good enough to make it public and observe what happens. We see people use it, and most of the time, the response does the job. What we haven’t tried yet: Running an open-source model on our own. Fine-tuning models. Judging the responses by another LLM on the fly. Pre-computing hints before someone asks for help. Partially reading files instead of dumping the whole codebase to the context. Even without these, we feel the result is pretty impressive compared to most chatbots. The key lessons learned for us (hand-crafted by a human so you don’t need to paste this post into an LLM): LLMs are not magic, but they can solve some problems that were previously unsolvable. Building an AI feature is easy, but making it work reliably is a whole different story. The complexity is in the orchestration, and the old software engineering practices still apply. You don’t need a framework to get started. A tiny wrapper around the LLM API is enough. You need to watch out for: The hype stories don’t mention the full picture. Try things yourself! LLMs WILL hallucinate in production, be ready to handle it. Testing AI systems (evals) is difficult to get right and can be expensive. It’s easy to go over budget. Implement limits early. If you want to see the Mentor live, you can try the free part of Go in One Evening (no login required).

Quick takeaways Go in the AI era: Go is excellent for AI applications, as the built-in concurrency makes orchestrating parallel API calls much easier than languages like Python Architecture philosophy: Clean Architecture isn’t always necessary. Start simple and add layers only when you feel the pain of complexity, not because someone said you should Career transitions: Switching roles within tech is easier internally. Moving from sysadmin to developer works better within the same company where people already trust you Go design patterns: Small interfaces near usage is the Go way. Duck typing allows you to define interfaces where they’re used rather than in separate layers Distributed systems: Async communication often solves sync timeout issues. When dealing with chains of service calls, consider using messages instead of increasing timeouts Introduction In this special 10th episode, we answer community questions in our first AMA format before taking a summer break. We discuss Go’s role in AI development, Clean Architecture implementation, career transitions in tech, and distributed system timeouts. After the break, we’ll switch to pre-recorded episodes with improved production quality. We still plan running some live episodes like this one, so stay tuned! Show Notes The new Domain-Driven Design in Go training we announced Team Topologies book mentioned for organizational architecture Join our Discord community for more technical discussions — we’re happy to help! Quotes Architecture sounds like something complex that architects do. But it’s basically stuff like “if you have 10 services connected with synchronous calls, what happens with timeouts?” Miłosz Five days of coding can save one day of planning. And it’s often the case. And even if it’s not visible on the first sight, it’s often visible during review that is going for ages. Robert I really like this idea of fixing something once you feel the pain. If something is not painful maybe it’s fine even if it’s not the best. Maybe it could be refactored, maybe it could be split to modules or whatever but if it’s not painful you probably have more stuff in the backlog to take care of. Miłosz In many cases we hired people that didn’t write a line of code in Go earlier. It wasn’t an issue in the end, because those people were productive very, very quickly. Robert Often people are like, I am just a developer, give me the requirements and I will transfer them into code. So yeah that’s not really helpful for most companies, because they need people who can understand the business domain, can help people figure out what the product needs. Miłosz Go is also a great language to be generated, thanks to the syntax that is simple, But it’s also very strict. So, for example, if you have some agent that is generating your code, it’s easy to verify that. Robert Timestamps 00:00:00 - Introduction 00:03:32 - Go in the AI world 00:15:00 - TDD in Go 00:23:44 - Career switching 00:38:35 - Clean Architecture interfaces 00:42:10 - Distributed system timeouts 00:53:31 - Domain vs application logic 01:04:09 - Small teams architecture 01:07:30 - Under-engineering risks 01:16:12 - Future plans Transcript Robert [0:00]: All right, so welcome in our first AMA episode. Miłosz [0:04]: Hello everyone. Robert [0:05]: Last episode before short summer break. So we have a couple of questions to go over today. And just as a reminder, so if you have any extra questions, just feel free to leave it on the chat. We’ll also be happy to answer that. Miłosz [0:21]: This episode is a bit special because this is our 10th one and we will end the live format this way. Robert [0:29]: Or at least partially. I mean, I think if you like this AMA format, there is a chance that we may have more episodes like that. So we’ll ask real questions and we’ll run it live. But in general, it will be no longer a live podcast, it will be a podcast. So we’ll basically pre-record most of the episodes and you’ll be able still to watch them on YouTube, so it will not change. And yeah i think it should improve the sound quality the video quality because well it’s not the simplest way to make it live and to make it sound great because it’s live so it cannot i mean it’s easier to tweak sound after recording obviously it’s a bit harder during the recording the second thing is obviously video uh the third thing is that a bit less obvious that is that we We don’t have our own room to record, so sometimes we need to improvise a bit to record in different places. Like today, so when you’re longer with us, you might notice that we’re in a bit different background. This real background, so… Miłosz [1:39]: I hope the next episode should be much more professional, Robert [1:43]: Let’s say. But you will see. It’s a surprise after vacation, so after we’ll be back, I guess, in September. Miłosz [1:50]: Probably, yeah. Robert [1:51]: Yeah, yeah. So this is the thing. So yeah. Improve and yeah it will be also a bit i think um content quality should be also higher because when we are recording it live you know it’s also a bit challenging to go with the flow and try to avoid a lot of pauses and try to maintain the flow obviously when it’s not live it’s much much easier because we can pause we can just cut it later we can think about topics so but yeah i think it was It’s a nice experiment to try with the live format and, again, it’s not like we’re dropping that. It’s nice that we have already working setup for that. We have the space, but it’s just a bit changing of the bars. Miłosz [2:33]: We also received a lot of feedback. It would be useful to have some diagrams, some visual aid to what we are discussing. So maybe we’ll also try something like this in the future. So it’s not, you know, so you have a video with some helpers or something more interesting like coding, not only talking. Robert [2:53]: Yeah, but no worries. So if you are listening for that in the podcast applications, also we don’t want to make it hard to listen, just listen. So, you know, sound is always the highest priority here. What else? Yes, I think that’s all about the format and I think we should be able to go to the questions. Robert [3:16]: Or maybe you have something to add before. Miłosz [3:18]: Yeah, if you have any questions, leave them in the chat so we can pick them up. Robert [3:29]: All right so questions so the first question is what do you think the future holds golang developers in ai forward world so i think we can have i actually have pretty a lot to think to say about that because we have the mentor functionality in our online trainings and from what we see how people are using that it’s super useful and especially compared to some other ai solutions because i think it will be not only my experience that very often when you see some ai functionality it’s just unpolished i mean you can have impression that somebody just get the api of open api of open ai or cloud or whatever just put some empty simple prompt and that’s it or like for example if you have iphone i mean it’s not the best implementation in the world with everything and the AI implementation. Miłosz [4:24]: As a user, it’s hard to get right. You can guarantee the answers. Robert [4:31]: I think it’s also easy to have something that works, but it’s much, much harder to have something that helps. But I think it’s a bit off topic. The question is more about Golang developers in the AI of our world. So what do you think about how Golang fits with this AI world? Miłosz [4:50]: I would expect most developers to use APIs, just call the RLMs over an HTTP API. And something that probably everyone does in Go anyway. So technically it’s not that different. And if you write more agentic systems, like we did with Mentor, The hard part probably is the orchestration of it all. So there’s a lot of loops and parallel function calls. And I think Go is actually great here. We’ve seen it work well when we designed the mentor feature, because goroutines give you very easy-to-use tools to create parallel requests. Robert [5:47]: And to cooperate with Python, so I think we are probably pretty good people to talk about that, because we professionally work with Python for some time, and doing this parallelization in Python is hell, in a very big simplification. And Go is basically created for that. So running something in the background is just a matter of using the Go keyword. And it’s simplification obviously, but compared to Python, it’s like deadly easy. Miłosz [6:20]: Other than that, you will use probably the same tools you use now for development with LLMs, just calling them, and you know, it won’t be that much different. Robert [6:35]: But I think this parallelization, this is the key, because when you’re building some agentic flow, usually you’re doing some operations that are slow. They are not CTO-bound operations, but they are still calling some API, they are doing some operations, and just doing it synchronously, it’s super, super slow. And this is why I think Go is just perfect language. I think it’s not only my opinion that Go is a perfect language for building AI or LLM or Gen AI applications. Miłosz [7:09]: Because it works. We will have a separate post on our blog soon. It writes up in detail how I do it. In short, there are a couple of complex parts, right, working with AI. One is prompt engineering. The hardest probably for me was writing evals, so testing the AI code. It’s super difficult. And the last one probably is orchestration of it all, which is the same code you are used to writing. It’s kind of like business logic for your AI agent. Robert [7:46]: The only difference is that the result that you expect is not that strict. As you mentioned, the problem is that if you have written some prompt, it will maybe work in 90% of cases, but maybe just 10% is the hardest when it’s doing some bad things that totally unexpected, and you need to spend a lot of time to convince it that, no, no, no, it’s not about what… This is not Go-specific, but Go, I think, is helpful for that because, for example, the Go test package, it works pretty nice with that. So you can do some simple wrappers and write evals for that. So that’s pretty cool. Miłosz [8:28]: In general, expressing business logic in Go is quite easy, or at least works for us well. Because we have this explicit approach. You have no magic in place. You can just re-read code and you know what it does. So it helps when you define Domain objects, business logic and such, and similarly, when you write this agent logic, similar idea. So it’s much easier this way. Robert [8:56]: I think it’s one important thing to mention here that if you are talking about, writing some Gen.AI applications, it’s not data science. I mean, because often when somebody is talking about machine learning, AI, Python is mentioned, but Python is great for data engineering, for data science, but using general AI, I think it’s pretty far away from data science. It’s more backend at the end. I mean, you’re doing some API, you obviously additionally have prompt engineering, but I mean, it’s big simplification, but it’s very, very close to backend in general. And a lot of good practices that you know from backend systems are applying one-to-one. Miłosz [9:46]: You don’t need much background about machine learning or RLMS in general to start. Robert [9:52]: I think probably a lot of people can compare it to driving a car. It doesn’t need to be mechanic to drive a car. Maybe if you would like to go to F1, maybe you need to have some mechanics background to understand very special details of the car, but it’s just or top level. Miłosz [10:14]: Can we comment on the audio? It’s not clear. Pull your mic closer. Or even maybe using some specialized tools of API. Miłosz [10:29]: It’s basically the same idea. Robert [10:38]: Okay, so we mentioned Go, how it’s compatible with AI for our world. I think it’s one more thing, because AI is very wide, ARIA, let’s say. So we have building some JN AI applications. We have also Vibe coding. We have generating some code on production. I think it’s also very interesting, we will think about Go, because Go is also a great language to be generated, thanks to the syntax that is simple. but it’s also very strict. So, for example, if you have some agent that is generating your code, it’s easy to verify that. There are no many options actually how it can be generated. Like other languages, you can do things in 10 ways. That’s making generation a bit harder and it’s also harder to get the syntax right. But in Go, thanks to having it pretty simple, I think I also seen even some studies based on that and covering that. Miłosz [11:36]: You’ve got types also in compiling, which is a big plus compared to dynamic languages. Robert [11:43]: Yeah, yeah. So, and I think it’s also out loud that, okay, you can probably easily generate JavaScript that will compile, let’s say, but it doesn’t mean that it will work properly. And for example, with Go, this is the language that for the first time I have the ceiling that, okay, I’ve written something, it’s compiled, and it works. With languages that are script languages, it’s not that obvious, because it can execute, but at some point it can show exception that, okay, it doesn’t make sense, sorry. Miłosz [12:15]: Yeah, you’re relying far more on testing. Robert [12:18]: Yeah. Miłosz [12:20]: That’s why I remember working with Python. We need far more assertions about what, for example, what some method returns, which is just not an issue with strongly typed languages, which is, no, it’s like nothing new, but if you generate the code, the less time you need to spend reading it, the better probably. So you don’t need to analyze, you don’t need to test such as edge cases, because the compiler does it for you. That’s a B+. Robert [12:52]: Yeah, so Tldr, in our opinion, and not only our Go is great for that. Oh, and maybe one last point about that, it’s performance, because also some people might say, okay, Python is slow, but I know maybe Rust will be faster. But remember, so what’s the battle like in your Gen.AI applications? It’s not CPU. It’s usually I.O. call done to some external server, so basically it’s doing some network calls. And for that, actually not CPU matters that much. It’s more about goroutines. And Go is I think even more efficient than Rust in running goroutines because it’s not using OS, I mean, it’s using operating system threads, but it has also its runtime. Miłosz [13:38]: I guess you can do the same lightweight threads or coroutines in Rust but with probably not such a built-in API. But it’s also something that many people overlook. They focus too much on the performance. I’ve seen some AI frameworks that claim that they are the fastest framework in the market. And as I said, it makes zero sense because you will wait a couple of seconds for the alarm to respond anyway. So if you optimize nanoseconds or milliseconds of your framework, virtually no gain. It’s similar to how people tend to benchmark Go routers for HTTP, which apart from extreme cases is not that important. If you if you talk about network and you use JSON Marshalling and you will be kind of slow but good enough anyway so the nanoseconds don’t matter that much yep ah great Robert [14:57]: Okay so the next question uh so the next question that you received is about how software development looks in go is it just a driven development uh so tdd yeah so it’s interesting question so i would be interested to hear more about the background but probably will not know but yeah, I think it depends a lot and it’s not like everybody does TDD in Go. I would even say that probably more people could do that. Miłosz [15:34]: Yeah, I think TDD is very popular in Go, but probably similar to other high-level languages. I don’t think there’s a big difference there. It’s just TDD is not, no, it’s language agnostic, so it doesn’t matter that much. Robert [15:54]: And probably compiler adds one extra step because you need to make this code compiled at least, but it’s a different level. Miłosz [16:04]: Do you use TDD? Robert [16:06]: It depends. So yeah, I think it depends on what kind of code I write. So for code that I know how it will behave, and it’s kind of clearly defined what I expect for that, it makes sense, because I can start with tests earlier. One nice thing about TDD is also that it’s helping to make nicer interfaces of the things, because, okay, you know what’s the expected interface, and you’re writing code for that. Miłosz [16:40]: Or designing the API. Yeah, a nice tip. Robert [16:44]: But, for example, when I am experimenting, or I don’t know what I would like to achieve, or I know it may be a bit controversial, but I know that for this functionality I don’t need tests, because it’s just obvious enough, and it will not add enough value. It also doesn’t make sense. Miłosz [17:06]: I actually find it helpful when working with AI to generate code. Sometimes I will use AI to generate something like, you know, a very specialized algorithm. For example, the other day I had to process a file to, I don’t remember, remove some lines or whatever in a very weird way, and this actually is pretty cool that you can define the function interface, write the tests for it, which are very simple because you have just input and output, then use AI to generate the body of the function. Robert [17:43]: And AI has nice specification basically. Miłosz [17:46]: Yeah, you can even give it the tests. And what I like about it is that if it’s a small function and I can define the tests well, I don’t really need to care if it’s cover or corner cases. Basically, I don’t need to care how the code works. I can treat it like an external library when you use an external library you don’t read all of it to make sure it’s nice and I don’t know, Robert [18:15]: More optimal sometimes better to not know yeah. Miłosz [18:20]: Exactly and so this is similar so this is kind of TDD, I would say because I focus more on the tests and I can write down first and then generate it and now I can just take Robert [18:33]: AI TDD. Miłosz [18:37]: I can take a look on the generated code, but I’m like, okay, if it works, I’m fine. Whatever. Robert [18:44]: I think also to make it super clear, so there are multiple, let’s say, classes of tiers of code. So it’s some component critical code. Probably this approach may be not best, but sometimes you have code that you don’t care that much. It’s some simple functionality. Miłosz [19:01]: You don’t want to maintain over time. Robert [19:03]: Yeah, maybe you’re just writing some ugly tool to do something once. And I would say that even just writing tests for that, it’s a lot. Miłosz [19:15]: For me, it’s like algorithms in general. Like when you have to parse something, when you go line by line and save some results over time. I had a couple of examples like this recently. and this works pretty well because you can do it but it will take probably about an hour or an hour yeah unless you like doing this it’s Robert [19:37]: Also like a cool it’s a strange hobby but I know that people like that. Miłosz [19:42]: It’s like a cool puzzle to solve although people complain about hacker rank exercises so maybe not everyone likes that but yeah it’s But coming back to the question about what is the software development, how does software development look like in Go? I would say it’s like any other high-level language probably. There’s no specialized methods. Robert [20:16]: But also, from my experience, it’s also pretty, let’s say, opinion, but I have also some feeling that compared to some other languages, in general, some library code quality is a bit higher than in other languages. So maybe it’s also thanks to the language, to how strict it is, but this is my feeling that I have sometimes also. I don’t know, maybe it’s subjective, but it’s something in that. Miłosz [20:50]: Yeah, perhaps. Robert [20:53]: Probably it’s not related to this class-driven development. Miłosz [20:56]: There are some approaches, but I’m not sure if I would count them as a methodology, compared to the TDD. Like you know avoiding frameworks could be one example but it’s more like what the community likes to do Robert [21:11]: Not. Miłosz [21:12]: Really a methodology Robert [21:14]: But it’s also coming from the language design yeah it’s I mean language is designed in a way that it’s, good that you don’t need to go into frameworks because you have some interfaces already defined that you can integrate with so it’s a nice thing It’s some special thing about Go, but it’s not a big deal for others. One good thing. I guess that the person that asked this question may not… The person mentioned that this person is learning Go, but also keep in mind that Go has one big nice advantage. You can learn Go very quickly and be productive pretty quickly. So when, for example, we had our teams, in many cases we hired people that didn’t write a line of code in Go earlier. Miłosz [22:05]: Because it was difficult back then to find people like that. Robert [22:09]: Yeah, but from the other side, it wasn’t an issue at the end, because those people were productive very, very quickly. And it’s not like there is some magical technique that you need to learn to use Go. It’s basically, you can learn the basics of the language and use the principles that you use in many other languages. Miłosz [22:33]: Well, the risk here is that if you hire a team with people from different backgrounds, we have a diverse team, which is nice, but also people might start writing Go, like they used to write Python or Java or whatever. So they need to be kind of aligned. So we spent some time to help them learn. Robert [22:58]: And I think this is an argument that we heard very often, that in terms of complex codebases that people were, copying some patterns blindly from Java or from C Sharp, but at the end it was very complex. So yeah, it’s probably something to watch out. Miłosz [23:17]: Yeah, it’s a classic discussion around clean architecture. Is it still Go or is it Java? you should start doing it. Robert [23:29]: Stop doing Java from Go. Miłosz [23:35]: Well, I guess you can apply it to any language, really. Robert [23:39]: That’s true. Miłosz [23:41]: Go too far. Robert [23:44]: That’s true. Okay, so another interesting question that we had was about switching a role. So the person was a sysadmin, and the question is actually how to… So we can probably generalize that. So how you can basically switch a role when you are, for example, Robert [24:02]: sysadmin or maybe data scientist or something else and try to go into software engineer path. So maybe do you have some examples of people that did this transformation and how they approached it? Miłosz [24:21]: Yeah, I’ve seen it happen within teams. It’s much easier to do it within the same company than just trying to be hired right away. It’s basically like starting in software development in general. Everyone knows being a junior is not ideal because we need some skills, to take proven skills into a portfolio to be hired, and it’s the same idea. But if you are already hired in some kind of team that works closely with software, it might be easier to start helping this team a bit, learning how they work, I might be doing some part-time work there for a time so I can see if you like it and if they like it. Yeah, I had a friend who was in the chat switch from front-end to back-end this way. Kind of similar, I guess. Like sysadmin to back-end and front-end to back-end I guess, pretty close, maybe front-end is a bit closer because it’s software development, but still you need some skills you didn’t have before. Robert [25:49]: And I think, so what you said, it’s some kind of internal team transfer and if you are kind of person that people know within the company that they can trust you and you are nice, it’s a bit easier to do this transfer because if you are joining a new company, it’s harder because it requires much more it involves much more risk for this company, so hiring, this kind of person let’s say also it’s possible, but I think it requires much more time to prepare portfolio, maybe some project that you work in and yeah, it’s probably easier than just going outside of tech to tech obviously if you are within the tech. Miłosz [26:32]: So it’s much easier because you already have some contacts with the people you can spend some time with them see how they work I’ve seen people do it from like a technical support role for example because this was area where they had to know the technology very well already to help clients and they were in close contact with developers so it was quite natural to just over time move towards development and remember, Robert [27:03]: Be nice to other people because if you are not nice for those people, it’s much harder to do this move. But unfortunately, sometimes if you are doing this kind of big change, sometimes you need to expect some salary downgrade. Miłosz [27:20]: Unfortunately. Yeah, for sure. And title downgrade probably too. Robert [27:26]: Yeah, but it may be worth sometimes, especially if you don’t feel in your role that it’s something that you would like to continue with. And yeah, I think also one nice tip maybe to try, okay, if you are looking for a company where… Try to find a company when you can use both parts of your skill sets, for example your sysadmin. So obviously it’s not always easy to find this kind of company, especially if market is harder, but sometimes you may find a company when they are looking for a software engineer with sysadmin skills, for example, and it can be nice advantage. Or for example, if you’re switching from front-end to back-end, you may look for a full stack role and try to do less and less. Miłosz [28:14]: So this is the idea of this T-shaped skills, right? They are super proficient in one area and a few others, just a bit. But it’s great if you have a team and you have someone with, for example, cloud skills or SysApp skills. So the rest of the team doesn’t need to go that deep in this topic. And this person usually is happy to handle them. Robert [28:39]: And yeah, this is the thing that also you recommend to do to basically go very wide with the skills. Obviously at the beginning you can focus on one area, but, At some point, I think it’s nice to try other areas, for example, for a back-end developer, try a bit of front-end, a bit of data science, a bit of sysadmin, a bit of devops. Why devops is probably not the best term for that, but okay, let’s use it. And it’s nice, because maybe you may find a role that is different and you like more, for example. And I think some projects that you are doing after hours, it’s pretty good for learning that. But you might discover that, okay, maybe there is some role that you prefer more. And I think also this kind of people are also very valuable in the team, because if you need to collaborate with other teams, so with some sysadmins, with frontend, with some cloud ops or whatever, you have more knowledge and you can, you know, have a discussion and, you know, because if you are working as a developer, you often may need something from your cloud operations team. And if you have no idea how it works, it’s, you know, sometimes hard to request something. Miłosz [29:59]: Yeah, so you have more skills in-house. You don’t need to rely that much on external teams. Robert [30:04]: And it’s also nice because if you are working in a startup, you can just deliver more things because sometimes you may have some small front-end functionality or maybe some simple data science work and you don’t need to outsource it to other team, wait ages for somebody that will do that. You can just do it yourself. I think it’s great. And this is, for example, what we are doing in Trios Labs. So we are kind of lucky that we did many areas so we can basically work to people and we can do a lot of stuff because we can have some experience so we are not front-end experts we are not data engineering experts but we know the field enough that we can do it in the level that is and. Miłosz [30:48]: Just use it to beat something end to end so we are not stuck or leverage some tools to help you do it. Because if you would need to hire someone to do it for you, that would be super expensive. Robert [31:05]: Expensive and the overhead is also pretty big, because if you’re doing something alone, you can just do it, ship it, and it’s working. But if you are outsourcing it to somebody, it’s probably 10 times more expensive in terms of time. Miłosz [31:20]: And the communication overhead is not that it’s an issue to talk with people, but there’s always this scoping and discussions and agreeing on things, so this will take ages. Robert [31:36]: It’s not only for indie companies like ours, it’s even for bigger companies where you have teams and requiring something from other teams is just not the most efficient way. And there is a follow-up question for this question. If it’s worth to invest in some older technologies like Laravel or PHP. And I think it’s maybe also an interesting way actually to do some switch because obviously working with older technologies, okay, it’s maybe not the most interesting thing in the world. But from the other side, the upside is that less people are keen to work with that. So basically, the competition is… Miłosz [32:22]: And there is Legasse to be worked on. Robert [32:24]: Yeah, and you can actually learn a lot on the Legasse. Usually, I remember when I learned a lot, it was the hardest projects. The projects that are greenfield and everything is great, you will not learn that much during that. Miłosz [32:37]: I would probably consider if the language is growing or declining rapidly. I guess if you consider Java let’s say or maybe PHP too there’s many many projects building those languages. They probably don’t grow rapidly right now. I didn’t check the indexes but I just guess. It might still make sense to go get into them because there will be lots of work in a way. But if you compare it to something that’s declining by, I don’t know, Peril? Maybe Ruby? Yeah, so that’s something that you need to watch out for probably. Not to spend too much time learning something that will be obsolete. Robert [33:28]: Yeah, yeah. Miłosz [33:29]: But if you are just getting into tech and you learn one language well, it will be worth it probably anyway, because then you can pick up the next one much easier. Robert [33:39]: To switch between programming languages than between titles, basically. And yeah, probably one more dimension there, so local market. So if you’re looking for hybrid or on-site, it’s also specific. Miłosz [34:07]: Job market in general. Check what the market looks like, which is also not ideal, because it can change in a few years, but probably not that fast. So we should be safe at least for five years or so. For example, with Go it was a bit risky when we started working on it full-time because there weren’t that many jobs, but also it was we learned quickly, so it wasn’t a huge investment. Robert [34:41]: And the competition, if some companies started to ask about Go wasn’t that big. Miłosz [34:47]: Oh, yeah. Robert [34:48]: As we said, so we’re later looking for people keen to switch to Go because there were no Go people in the market. Miłosz [34:56]: Right, yeah, and that’s also interesting. Sometimes, you know, in a situation like there’s a niche language which some companies use and they are desperate to hire, and there are a few developers who are also desperate to be hired because they want to learn in this technology. But they don’t always match each other, depending on the location and so on. So, one more thought. Maybe it’s easier to stick to the proven but old technology than learning something fresh. You know, something that’s not stable yet. Because it might never get traction. Miłosz [35:41]: I think maybe Elixir is one example. I’m not sure how the job market looks like for it, but I remember learning it around the time I learned Go and And I think betting on Go was probably a more pragmatic choice. Robert [35:57]: Yeah, definitely. Miłosz [35:58]: I think by now, Go is much more popular. Robert [36:02]: Yeah, I remember that a long, long time ago, also, it started to be some Scala hype. And I guess that you can still have some Scala job. Miłosz [36:12]: But… Probably Kotlin killed it a bit, right? Robert [36:17]: It’s a bit different, but it also had some specific use cases. But I think the entry point also wasn’t the best thing. But at some point, it was trying to get some traction. People went into that, but I would say that, okay, maybe in some areas it’s still there, but it’s not longer hot. Miłosz [36:38]: So you have those early adopters, companies who jump on it and hire people, but there’s not enough of them. Over time, we just kind of die down. Robert [36:52]: It’s just a reminder about the thing that we are advocating all the time. Don’t focus on learning specific technologies. Obviously, you should have some baseline, but we covered it more in an episode about efficient learning. If you focus on the things that are language agnostic and timeless, it will work for you for much longer. Because if you focus on learning some frameworks, languages, obviously you need to have some base, but it’s not the most important thing. And this was also the key when we were looking for people to our Go teams, that we’ve been not asking about how to implement something in Go, because we just know that it doesn’t matter that much. What mattered for us was how they are solving problems. Miłosz [37:42]: How they Robert [37:43]: Can for example get some requirements and stuff like that and some general. Miłosz [37:50]: Stuff Firmina shared in the chat that Java killed Kotlin oh yeah I don’t know so I will trust you’re right long late leave Java I agree and Firmina also asks if one year and a half is enough time to let it go So, I think day and a half is good enough. Robert [38:17]: One evening I think it’s enough. Miłosz [38:19]: Yeah, it’s ideal. What’s the next question? Robert [38:27]: The next question is about clean architecture. You would like to take that. Miłosz [38:35]: Okay, so this is about where to define the data interfaces. Should you define it in a data layer or in the implementations? And this actually is something that where the language matters. At least if you compare Go to, let’s say, Java. And one cool feature of Go is implicit interfaces. Robert [39:03]: In other words, duck typing. So, if it’s ducks, it’s a duck. Miłosz [39:13]: In Java, in many other languages, you have to explicitly say this class implements this interface. In Go, we don’t need to do it. And it has this nice advantage that but you can keep the interfaces close to where you use them, so probably near the… not near the implementation of the interface, but near the thing that uses it. Usually the application layer or use cases or whatever you want to call it. Some people don’t like that, but I think it’s quite cool. you just need a good IDE to show you what implements an interface. That’s a bit hard. If you are a diehard emacs fan, that might be a problem. Or maybe you need some plugins to navigate. But other than that, it’s quite easier to keep the interface close to where it is injected. or used. And this solves this issue of a separate layer for the interfaces, like ports and adapters, where you need to keep them in one place. In Go, you don’t need this layer at all. It’s much simpler. Robert [40:41]: And I think it’s one big advantage of this duck typing and this non-explicit implementation, is that it’s much easier to implement interface segregation principles. Basically, you no longer need to have interfaces that have 30 methods, for example, for some big repository. Because instead, you can just have multiple interfaces with the methods that you need in this use case. And you can inject this repository with multiple methods there. But you are making the contract of the interface much smaller. Because you are just defining which methods are required. And it’s useful for refactoring, because sometimes you have a repository with that many methods, you can just cut it to multiple repositories and just inject it a bit differently. And it’s nice. And it’s also nice for testing, because when you’re also writing tests, when you have some stubs or mocks, you can just stub or mock smaller interfaces. Miłosz [41:40]: Yeah, that’s also great. Small interfaces and close to where they are used. That’s the idea of scenario. We covered this in more detail in the episode on Aclia Architecture. If you want to check it out, it’s the third one, Robert [42:01]: I think. And just a reminder, so if you have some other questions, listen on the chat, so we’ll also cover it. Like, what’s the joke? Okay, so let’s maybe take a question from the chat. if MocnyŻuk asked the question. So how do you handle timeouts in distributed system in sync communication, HTTP, GRPC? So probably it can be answered from multiple levels, let’s say. But in general… How it should be handled. So I guess the question might be, okay, if it’s about distributed systems, I guess it may be about some chain of calls. So one service is called by something and it’s calling other services. So in Go, for example, it’s handled out of the box with context. So when you have the client that is calling your server and the connection is closed, so it’s propagated basically through context and it should be propagated to HTTP Hender. It should be propagated to other requests that are going further. And if you are passing context, for example, to your database queries, it should be also cancelled and it’s doing some kind of chain of cancellation and at the end the connection is closed and everything is great. Miłosz [43:22]: What about more like a high-level approach? How do you define timeouts? Let’s say you have a service that calls another service. Do you set a constant timeout for it? Robert [43:36]: So usually i would say that at the beginning you have some endpoints from your message handler and usually it’s good to have some default timeouts let’s say like 30 seconds let’s say for http request and the idea is pretty simple pretty similar because you have this context cancellation that is propagated to everything else basically yeah. Miłosz [43:59]: I would keep it very very short or sync communication, especially for internal services. Because for external ones, external APIs, you sometimes can’t control it. And then I would consider hiding it behind a message. Basically publishing a message, and then this message triggers this endpoint, and if it times out, it will just be destroyed eventually. If it’s internal, I would keep it super short. Because long requests are usually a pain to handle. Robert [44:38]: But it’s important to not forget… So if you are talking exactly about Go here, so passing context, because sometimes I notice this problem that somewhere this context wasn’t passed, and context background was passed, and it may be problematic when you are doing some I.O. Call, like if it’s HTTP or some database call, it can run for much longer than you would. Miłosz [45:05]: Like to run. Robert [45:06]: And even if you have middleware with the timeout, it doesn’t matter because it’s not propagated and it will run, maybe not forever, but for pretty long. Miłosz [45:20]: Yeah. And working with timeouts, there’s one more difficulty. If you have a chain of services that call each other and they are slow, and then the timeout happens on the end, don’t really know what happened in the system, the last one. Maybe it was just network failure or whatever. Robert [45:43]: Especially you don’t have transactions. Because if you have transactions, if you have transactions, again, we are talking about Go, but context cancellation should cancel that and it should roll back things. Yeah. Miłosz [45:55]: So it’s often where people will consider using a distributed transaction, maybe. Robert [46:03]: Or at least some message in the middle, maybe event or command. Miłosz [46:08]: Yeah, so there’s something to watch out for. Ideally, those endpoints will be idempotent. Robert [46:14]: You should try it. Miłosz [46:16]: If it happens. Robert [46:17]: Or ideally, the operations will be fast, and the things that you’re doing synchronously, maybe it can add something to the queue, and it’s just responding pretty quickly, and everything else is done in the background and maybe you can. Miłosz [46:32]: Okay, I do have a follow-up. Let me read it. I have a scenario where a given call is propagated for various services via REST API. Therefore, you need to set up smaller and smaller timeouts the deeper you go in the tree of services. Oh yeah, okay, I see your point. So you have a long chain of services that call each other. And I don’t like the approach to use increasing timeouts from top to bottom. That’s why this question originated. How it should be handled? Right, so you have this chain. The first service needs to be aware. There will be lots of calls. Robert [47:17]: So I assume that if you are mentioning that it’s not like the entire chain is executed within one second, something tells me that it’s taking a couple seconds or up to 30 seconds, for example. Miłosz [47:30]: Yeah, maybe minutes. Robert [47:32]: So, I don’t know. So it’s hard to say without seeing the system, but it’s maybe a case for some more asynchronous communication, for example. Miłosz [47:43]: I would consider it, Robert [47:44]: Yeah. Yeah, because in this case, you may have a problem of cascading failures as well. Or also other solution that I know that for other people is crazy. Maybe we don’t need as many microservices. It depends, because I don’t know this case. maybe every team has its own service and its multiple teams and its very big product. That’s fine in this case. But if it’s, I don’t know, one team has 20 services, maybe it doesn’t need to be 20 services. Again, it’s hard to say. I’m more saying, more mentioning, more alternative, let’s say, to consider that, but… Miłosz [48:28]: But in general, if you have this spider web of service communication, I would probably try some asynchronous approach, because there are many issues there. For example, each service needs to react to some error. I’m not sure if this is like one service calling 10 or 10 in a sequence, but any time any failure happens, maybe it’s timeout, maybe it’s a network issue or some domain error. It needs to handle it and it propagates it up the stack and then the next service also needs to handle this somehow. Robert [49:06]: And if every service in the chain is storing something in the database, I hope in transaction, but from my experience, something tells me that it may be not always the case. Miłosz [49:17]: It’s more risky. But the transaction spans only one service anyway, so I would consider doing it with a message. Publishing a message with the first service, and then all the others reacting to it somehow. Robert [49:32]: So, start with the question. Do you need that many microservices, PCS. Miłosz [49:37]: Might be too late for that. Robert [49:40]: Yeah, okay. That’s true also. Miłosz [49:42]: But the interesting approach has the downside that you don’t get the error right away. I mean, you don’t know the result of the other services, what they execute. Robert [49:56]: Also, one more alternative, because, maybe it’s also not that big issue, and maybe you can increase this timeout a bit more. Miłosz [50:09]: Yeah, it’s not stuck out the most naive. Robert [50:12]: Yeah, I mean, it’s a naive option, but we also know how it looks like. In perfect world, it would be good to join microservices, use asynchronous architecture for the places where we need but sometimes it’s not important enough and we don’t have enough time yeah. Miłosz [50:32]: Depends how many activities are on there because if you add a new service to the chain every week, it will get much worse with times but if it’s something you don’t need to touch over the year whatever, we can just bomb the time out So, Robert [50:52]: I hope it didn’t help. But yeah, jokes aside, so, well, favorite face. So, it depends, but… Miłosz [51:03]: Yeah, I would consider if async approach can work here. That could simplify it. Robert [51:08]: Even simple ones. So, also, the thing that we are often advocating for is, even if you don’t have any message broker, you have Postgres. You can have a very simple message broker on Postgres and it will be also fine for the beginning. Miłosz [51:22]: I love how helpful we are that someone asks about sync communication specifically and our answer is to use async. Robert [51:32]: Yeah, but this is the case of end with. Miłosz [51:37]: Yeah, but other than that, there’s not really much you can do. You can bump the timeouts, sell it everywhere to high value and live with it maybe it’s not an issue or redesign the system work in a different way So, some frontline service that would send an event and then waits for the finished event. If you wait for a finished event, it’s kind of like a timeout. You just do it over messages. Robert [52:16]: So, it depends if you’d like to wait for this event in context of request. So, basically, usually you do it in the way that you’re sending some event, and the request is finished. Miłosz [52:32]: And you can, for example, have a status in the database that each service writes to. And then you can display it on some kind of UI. Robert [52:41]: But, again, it’s not for free. Miłosz [52:44]: It really depends on the use case. Robert [52:46]: On the complexity of the system, how big is the team… Miłosz [52:50]: If you want to share more about the exact use case you have, You can join our Discord and we can help you figure it out more in detail because it’s hard to… Robert [53:00]: Yeah, I think the size of the team, the scale, it’s important here, because I think the answer depends on this context a lot, basically. Miłosz [53:13]: Hello, full dump. Nice to see you. Thanks for joining us. And drop your questions if you have any. Okay, should we move on to the next question, is also about Clean Architecture, I think? Robert [53:31]: Yep, that’s true. So, my team started using Clean Architecture about a month ago, and it has been quite a learning experience. But overall, we are seeing the benefits as we wrap our heads around it. One thing we got stuck on sometimes is when to determine the boundary between application and domain layers. This is the question that we all usually have. Or question, what is domain logic? This is also a pretty interesting one. So we’re using SQLRS in application layers. Sometimes the handlers are so simple for crowd operation that we don’t have any dependency on the domain layer. Other may be than entities. We have other handlers that do a lot more and that’s ended up with calling domain-specific logic. Is it common? So to recap, I think it’s about if we need to have domain logic basically in our applications or we can simplify that because it’s just some simple crud and it doesn’t make sense to do some domain magic there. Miłosz [54:41]: Yeah, I remember the long discussions about this we used to have. Robert [54:46]: What is domain logic? Miłosz [54:47]: Yeah, what is application logic? So, maybe we can start with that you don’t need a domain logic in every service you have. Robert [55:01]: Even if you have domain logic, because you can probably say domain logic about many things, like you’re doing if in SQL query, it’s domain logic. Miłosz [55:10]: Yeah, so how I like to think about it is like, instead of four layers or whatever you might have, just think about it as a split between implementation details and the logic. That’s a very basic split. You want this domain code or business code away from the SQL queries. And then, if you find it useful, you can split this logic into pure domain logic and all the other orchestration, application things that join things together. And it’s not like a huge deal, but sometimes they work with very complex domains, where this domain layer will be huge, because the business rules are huge basically. And you need to model what your software does. And there’s no other way than writing it down in the code. So it helps to have a separate layer that’s not polluted by any database, calls, or whatever it’s just pure functions and structs and lots of tests for it yeah, Robert [56:27]: And I think for what we advocate here is trying to be mindful about that and asking yourself a question whatever I’m doing, is it helping me or not? For example, okay, I’m not using domain layer I have a lot of logic in my application layer okay, maybe it’s not helpful maybe I should have domain layer or maybe the opposite side for example, I have domain layer with encapsulation and a lot of stupid mapping for CRUD so I’m just mapping fields between layers and not doing anything for that I don’t have tests for that and I see that it’s not giving me any value okay, it should be not there and it’s not easy to give a good answer if you should have a domain layer or not. You need to learn that. Oh, by doing. And I don’t have a better answer than just writing this kind of logic and over. And yeah, you will have some puzzle. Miłosz [57:24]: This is the hard part that you can’t really define that. And I remember having the same issue because we have some meeting where we agree, okay, guys, this is application, domain logic, blah, blah, blah. We set this you know the system in place so everyone knows and then you have this edge case like when you for example like this person said they do just crud and then you start to wonder you know should it be there should it be there mm-hmm It’s hard to give one answer to this, but most of the time it also doesn’t really matter that much. Robert [58:01]: Just try and be mindful. I know it’s not easy, and I think it’s also not easy with Skunko’s fallacy, because if you already spent a lot of time to build your beautiful domain, layer, and at the end you see that it’s just useless, it’s a bit hard to delete that, I know. Miłosz [58:21]: So if you could define very strict rules for your application, we wouldn’t need probably admin developers because we would just wipe code or we would just use those builders for software. Robert [58:40]: Or you will use spreadsheets. So I think it’s interesting, heuristic, or kind of. Like, if you’re building a crowd application and you’re feeling that this is enough, maybe you don’t need an application for that, maybe just something can do a super big spreadsheet doing that. Okay, it’s maybe not most optimal, but sometimes it’s more than enough for the beginning, especially if you’re working in a startup, because you might not know if it will be useful or it will be trolled away. and, you know, the spreadsheet for the beginning. Right, fine. Miłosz [59:16]: I think when you said that in the beginning, it was important to you. Just think of what you get. If you are just starting out, do you need this full design of all the layers and splitting? Probably not, because you don’t really know what will be there. But if you have application layer and so handler, and you find 500 lines of code there with 50 ifs and lots of conditions, lots of arguments and so on. Then it kind of feels natural to just extract this thing somewhere else. It can be another function, another struct, or if you have many places like this, just do a separate layer of it. And that’s basically the idea. Robert [1:00:03]: The downside is that often we don’t have enough time to do that, so the pain is starting to be bigger and bigger and bigger. But the more complex it is, the harder it’s refactor that. But from the other side, I think it’s also nice, because if something is bad enough, it’s easier to propose changes. Miłosz [1:00:22]: You feel the pain, so you know it will help to change it. Robert [1:00:25]: For example, Mocnej Zhuk asked about these timeouts. I’m guessing, but it may be two scenarios here. That those timeouts aren’t painful and it may be worth investing in migrating, for example, to event-driven architecture, but maybe it’s not painful enough and just increasing timeout, it will be good enough and, okay, we have some bigger problems. And the same is with refactoring. So, for example, if your model in the application layer is complex enough and it’s slowing down your implementation a lot, you might have time to actually refactor that. And obviously, it would be great to be right at the beginning and start with domain logic and have tests from the day zero, but also let’s be realistic that you may be wrong with your assumptions, that you may assume that, okay, it will be crud, but at the end, the complexity will explode. It happens. It happens even for us when you are working up. But the reason for that is not that it’s sometimes some lack of skill, but it’s often lack of understanding. That if you are starting to work on something you may not have full understanding of the problem space and you will discover it during implementation and there’s just no way that you guess that okay maybe you will have luck and you will. Robert [1:01:51]: Implementing in a domain logic. But you may have a bit less luck and you assume, okay, it’s simple, complex, it will explode. And even if you do your best to guess it should have domain layer or not. Miłosz [1:02:07]: You will be not always right. Next month, your boss can come in and say, actually, you know, we decided not to do this. Robert [1:02:15]: Or we signed a customer that needs this, this and this. Miłosz [1:02:18]: Yeah, can I just revert what you did last month? Because we need to change the domain model. And it’s actually fine, you know, when this happens. So your domain model is never ready. You need to adjust it over time. Robert [1:02:34]: So yeah, TLDR is totally fine to not have domain layer, just put everything to the application layer. Miłosz [1:02:41]: I use some of it and some DOM. Robert [1:02:43]: Exactly, exactly. And what’s even better, and it’s not like if it’s good or not, but it’s, I would say, necessary for applications to have modules that have domain layer and have modules that doesn’t have domain layer. And it depends a lot on the application, but usually this is a good idea, because trying to keep consistency for everything, it’s usually a bad idea. Because usually you have these modules that, even if you have very complex applications, not everything is critical. Miłosz [1:03:13]: Although it will be much simpler to have one standard for everything. Robert [1:03:18]: Yeah, but it will be just very expensive compared to how much you will gain from that. Miłosz [1:03:28]: So I really like this idea of fixing something once you feel the pain. If something is not painful maybe it’s fine even if it’s not the best maybe it could be refactored maybe it could be split to modules or whatever but if it’s not painful you probably have more stuff in the backlog to take care of and Robert [1:03:54]: You will never finish. Miłosz [1:03:55]: Everything yeah, you will just keep growing it that’s normal, that happens in every company every project Oh, I think we can answer this one. Robert [1:04:09]: But we have another one about clean architecture. Oh, okay. So, what about clean architecture and small teams with complex projects? So it’s a question about the episode that we had. So the most popular thing that we have. So 10k views on just YouTube. So yeah, this is probably why it generated… Miłosz [1:04:32]: So there we mentioned that architecture shines for bigger teams and complex projects. So if you have a smaller team or a simpler project, better to avoid it. The question is, what about small team and complex projects? Yes. Robert [1:04:51]: Yeah. Miłosz [1:04:54]: Yeah. Robert [1:04:55]: So it makes sense. So we are a two-people team, and even if we are a quite small team, we are using some lightweight implementation of Clean Clean Architecture, because if you were already doing that, it doesn’t add that much overhead, and you are doing it properly, and it can basically help in many, cases, even if you’re a small team. Again, the cost is not very big if you’re already proficient with doing that, and you can just have benefits out of that. It’s also some natural way of doing things, but again, it’s about doing it in a lightweight way. Miłosz [1:05:40]: I even do it for my own projects, very small ones, because it comes kind of natural. But yeah, the key point here is to do it in a very lightweight way. For example, not inject everything from an interface like we often see in Dan. That can be crazy. Robert [1:06:02]: Also, the thing that we mentioned in the previous question, you don’t need to have domain layer with the clean architecture. It’s not a must. Miłosz [1:06:10]: You don’t even need application layer if you don’t know. Robert [1:06:13]: And probably you shouldn’t have more than four layers, because we’ve seen the comments on Redis that do. Miłosz [1:06:19]: You remember how it was? Yeah, it was like Clean architecture is over-engineering. You shouldn’t have more than four layers in your project. We’re just free. I don’t know what some people do, but it sounds like over-engineering then, for sure. If you manage to keep it simple, you can do it for small projects, for sure. For small teams, complex projects. Sure. Why not? I’ll just watch out for making it overcomplicated for no reason. Once again, this question of why it helps can be helpful. I want to understand why you do it, not just do it because someone says, this is a nice idea. So for example, do you run integration or component tests? If you do, the bendless injection is what you need basically to do it in the same way. If maybe you don’t want component tests at all, maybe then you don’t need the prevention injection at all. Let’s try to understand if it makes sense or not. Robert [1:07:32]: And it was an interesting follow-up question and all that. So if it’s something like under-engineering, not spending any time on architecture at all. Miłosz [1:07:47]: Yeah my favorite quote here is that there is no there’s no such thing as no design there’s only bad design that seems similar I just don’t spend any time on proper design or any kind of architecture, you’ll end up with this big ball of mud. And sometimes it’s fine if it’s some project you do over the weekend and we’ll scrape or your vibe code. Robert [1:08:19]: Or it’s proof of concept that you will hopefully throw away after proving if it works. Miłosz [1:08:27]: But yeah, keep in mind it will happen. So is it what you want? If you want to maintain this thing over time, maybe just spend some time, a few hours to draw some diagrams, try to sort it out in your head how it should look like. Robert [1:08:45]: And I think it’s one interesting thing that we covered a lot in the… One interesting thing and side effect of this node design that we covered a lot in the previous episode, how it affects code review. So if you didn’t make any design in your team, so The code review, so the PR that you may submit, may, let’s say, surprise some people. And surprising PRs are usually not best because it will trigger a lot of discussions, that resolving will take ages, that fixing will take ages, and probably it will take more time that you will spend on design. And I know nobody likes meetings, but I think I like having big PRs that are discussed for ages less than meetings. And I like the sentence that five days of coding can save one day of planning. And it’s often the case. and even if it’s not visible on the first site. Robert [1:10:01]: It’s often visible during review that is going for ages. And later people are blaming that code review is awful because you have a lot of the comments, you need to resolve that. Let’s just push everything to main. But it’s not a problem of code review. It’s a problem of surprising people. And don’t surprise people with your code review. Miłosz [1:10:24]: Architecture sounds like something complex that architects do. But it’s basically stuff like if you have 10 services connected with synchronous calls, what happens with timeouts? That’s basically it. Try to think about it, what will happen, how to handle this and then prepare for it somehow or either fix it or just agree it’s fine and note it, document it somehow. Robert [1:10:55]: Or a different thing that I also remember that we faced one time is not thinking really how teams are split so let’s imagine that you have some process, and let’s imagine that the beginning of the process is handled by team a later team b is doing something the process later team a is doing something again and later some team c, yeah yeah yeah and it’s pretty problematic because nobody can really own some big chunk of the process because it’s really chopped for small parts that are between teams and it’s, hard to do some kind of handover of big. Robert [1:11:42]: Let’s say, events in this process. And again, it’s a matter of a bit of design and thinking actually how the process looks on the high level. And obviously, it can be fixed later, but it’s also problematic because often it means that, okay, if this part of the process is handled by this team and it’s between parts that are owned by this team A, maybe this part of the process that is in the middle should be also owned by team A. And, okay, it can be done, the service can be migrated to other teams, but it’s expensive, because it was written by other teams. Probably it will be harder to maintain, because you have no idea how it works. Maybe this team will be also not happy about that, because they will say, I already have 50 services to maintain, we don’t have capacity for that. and yeah it’s like. Miłosz [1:12:39]: A very high level architecture like an organizational there’s this team topologist book that talks about this I think it’s quite nice if someone wants more on this topic you can check it out Robert [1:12:52]: And we also mentioned in some previous question that even if you are not manager if you are not architect it’s nice to have this knowledge from other areas for example like team topologist that is, a bit higher level, but from the other side, you can also propose better things and be promoted much faster, because very often those people that are on higher, levels, they may not know that. Miłosz [1:13:20]: Yeah, that’s true. We can help them fix things, which can also help you in your career. Yeah, so it’s nice to have those more orthogonal knowledge. That’s pure development. That’s a good example of this, too. Robert [1:13:39]: Yeah, and this is the thing, that if you are going to more senior roles, you’ll notice that, okay, writing software is important, but there are many things around that. And to really be beyond this senior level, you need to understand much more things and also go beyond technical things, basically. Miłosz [1:14:01]: Yeah, and AI just confirms it. Writing code is cheap somehow I don’t hear many great stories about you know generated codes we are not probably doing much more meaningful work thanks to it to figure out anyway what you need to do in the first place and I think it’s Robert [1:14:26]: The thing that people that are most senior can get more out of that basically. Miłosz [1:14:34]: Yeah, exactly. So you get boring parts away, but you still need to figure it out on a high level and design it properly. Robert [1:14:45]: Alright, cool. So I think we are shortly going out of questions, as long as you not put something in the chat. Or as long as you don’t have something to add about previous questions that came into your mind. Miłosz [1:15:04]: So not as long in the chat if you have any more questions. Robert [1:15:08]: Last chance before our holiday season. Miłosz [1:15:12]: Yeah. So we can share a bit more about our future plans. Robert [1:15:17]: Yeah, so short term, resting a bit. So I think we had a super busy time now with Go Event Driven V2. So about doing the big update, also giving you support. Robert [1:15:33]: Any horror stories? Miłosz [1:15:39]: We need a separate episode about startup horror stories. Oh, yeah. But I also want people to share theirs because I love reading this. Robert [1:15:50]: One of my favorite ones is about sending a couple million SMSs to one person within one minute. Remember, add some trodling. Miłosz [1:16:04]: Thanks for the suggestion for me I think we need a separate episode about this and I hope to hear people’s stories as well Robert [1:16:16]: So after resting what are, our next plans I think we can officially confirm that we know what will be our next training that a lot of people were asking about. So do you want to share these news? Miłosz [1:16:34]: Yeah, so we plan to release our next training, which will be the Domain Driven Design in Go. And there will be also a hands-on training like Go Event Driven and Go in one evening. Let’s focus on Domain Driven Design. Or I would Robert [1:16:51]: Even say that obviously Domain Driven Design, it’s some way how to go there, but from the high level it’s more about how to build application with complex domain logic. So many questions you heard about this complex domain logic is for a good reason, because applications that are successful often have these parts that are very complex and require a bit more love than the crowd applications. And those are usually the unique part applications that are making money for the company. And the interesting thing is that, companies are kind of happy to hire people that can work with that because, one thing is that they need to usually touch this code update this code to make more money but from other side it’s also hard to find people that they can do that because it requires a bit different skill set to do that and and. Miłosz [1:17:50]: Often people are like i am just the developer give me give me the requirements and i will transfer them into code so yeah that’s not really helpful for most companies, because they need people who can understand the business domain, can help people figure out what the product needs. Basically, product engineering. So this training is about that. Robert [1:18:14]: We’re pretty lucky that we had a chance to work with some pretty weird domains that well, people try to use CRUD approaches sometimes there and it was actually quite interesting to see how this trying of simple approaches totally failed. Miłosz [1:18:33]: Or maybe they thought they are using DDD, but DDD was their own idea which also didn’t end up well. Robert [1:18:41]: Yeah. Miłosz [1:18:42]: Yeah, so we this would be also hands-on training, like the previous ones. We learned a lot from people’s feedback. We had over thousand trainees in both trainings. Robert [1:18:58]: Over 2000. I think two and a half thousand people. Miłosz [1:19:04]: So we received lots of feedback. We had some things we wanted to improve. That’s why we prepared the second edition of Go Event Driven. That should be much more fun to go through and easier in general. So we want to take those learnings to the next training as well. Make it more focused on what’s the most important and focus on real life scenarios Robert [1:19:34]: If you don’t like coding, sorry, it’s very hands-on. Miłosz [1:19:40]: You need to code. There’s no videos to watch. Robert [1:19:43]: And yeah, I think that this is a pretty common thing. You need to learn by practice, when to go simple way and where to go complex way. This is actually what was helpful for us. just working with many projects in a daily job, after work, and trying to get this muscle memory to have some idea what approach to use there. And this is why we are advocates of hands-on, because this is the only way actually how you can gain this skill. And you can get in work in the projects and do some crazy over-engineering stuff at the end, or maybe it’s a bit better to learn it in a controlled environment. Miłosz [1:20:33]: So you don’t need to confess to your manager why you need two months of rework and refactoring. So you can experiment on your own project or our guidance. So this will be in Go, initially, but we probably will have more languages later, but that’s probably not the start. Just as with the other training, we will have a pre-sale, which will give the best price. So basically we want to check if there’s enough interest before we start working on it. And of course you can ask for refund and so on. That’s the same rules as we did in Robert [1:21:19]: September so for now a bit uh a bit of rest a bit of thinking what we’ll do uh in quarter. Miłosz [1:21:27]: And also there’s also one thing we want to complete which is go with the domain ebook so we have some cool ideas about to release the second edition or the 1.0 maybe edition is a better name Robert [1:21:43]: Yeah because there are still some chapters that were not finished. Miłosz [1:21:47]: That’s long overdue for a few years, so finally you should have some time to do it. Robert [1:21:53]: But you know, probably it’s also good to share that if you have projects in your work that are taking more time than your thoughts, you’re not the owner. We also have this problem, but for a good reason often. And yeah, as we said, in September we’ll also do some updates of the NoSilverBullets form. It’s a bit secret, so you’ll see on September, but I think it will be quite cool. That’s it, I think. Miłosz [1:22:33]: I have no more questions. Miłosz [1:22:36]: Firmina is laughing at us, so that’s time to wrap up. Robert [1:22:42]: Yeah so thanks for being with us and see you in two months yeah thank. Miłosz [1:22:47]: You everyone for joining us next time Robert [1:22:50]: See you next time ciao.

Quick takeaways Prioritize reviews over new work - treat PRs as work that’s almost done and needs to be pushed to production quickly Big PRs create a deadly loop - when reviews take ages, developers make even bigger PRs to avoid multiple long waits Knowledge sharing is the hidden benefit - code reviews aren’t just gatekeeping, they spread understanding of how things work across the team One-day cycle time is possible - start work in the morning and merge by end of day with proper team culture and practices Split work vertically and horizontally - break features into small slices and layers that multiple people can implement in parallel Introduction In this episode, we discuss how to make code reviews fast and effective by keeping pull requests small. We explore why big PRs are problematic, what causes them, and practical strategies to create PRs that can be merged within a day. Instead of waiting days for reviews with 200 comments, we focus on techniques that help teams achieve smooth, fast review cycles where work flows quickly from code to production. Show Notes Feature flags - technique for merging code that’s not ready yet Mob programming - multiple people coding together with one driver Clean Architecture - helps with splitting features layer by layer Linters - automated tools for catching style issues instead of nitpicking in reviews Draft PRs - for getting early feedback on experimental work Shadow mode - running new code alongside old code to compare results The Phoenix Project - book mentioned about bottlenecks in teams Quotes If you have a couple developers that are independently working on some features, they are actually not a team. They are a set of individuals working on something. And at the end, there is no shared responsibility to deliver some features. Robert It’s a well-known phenomenon that if you have small PR, people will find three defects, but if you have huge PR, they will just write LGTM. Miłosz I like the phrase that five days of coding can save one day of planning. And I think it’s often true and it’s often a problem in multiple teams. Robert I sometimes see people mention that code reviews are a sign of lack of trust. As in, “You don’t trust your teammates because you need to review code they write.” I think it’s funny because I find it the other way. So I actually trust them to review my code because I don’t trust myself enough to know I don’t make any mistakes. Miłosz I think it’s one of the most important skills for product engineering, and I think it’s super rare as well. Because most developers, when they get a clear description of a problem from a stakeholder, will just say, okay, we can do it, because they want to be the hero. Miłosz I know that a lot of people are like, let’s skip planning, let’s just write that and ship that and it will be everything fine. And I know that I was also like that when I was younger and less experienced, but later you can kind of learn that, okay, it can give you some acceleration at the beginning, let’s say, but later it can be very problematic. Robert Timestamps 00:00:00 - Introduction 00:01:43 - Why review code 00:07:12 - Problems with big PRs 00:16:13 - Causes of big PRs 00:23:27 - Planning importance 00:35:17 - Breaking down changes 00:44:35 - Big refactoring strategies 00:52:43 - Code review culture 01:04:24 - Accelerating review cycles 01:12:04 - Q&A session Transcript Robert [0:00]: Ever waited days for Code Review only to get 200 comments and endless discussions in these PRs, so maybe you wonder what’s the reason of this kind of big PRs and why it’s happened that such big PRs are happening. Robert [0:18]: And what’s often the problem, it’s getting worse. I mean, when you are waiting ages for PRs to be merged, it’s creating some kind of dead loop, let’s say. Miłosz [0:33]: It gets even worse. Robert [0:35]: It gets even worse, because if you know that you wait ages for merging a PR, you are just making them bigger, because you know that if you prepare 10 smaller PRs, you will wait longer for those PRs. Today, in this live podcast, we’ll discuss what are the problems that are causing that PRs are big and some strategies that we used in our teams to make those PRs merged much faster. So by much faster, I mean how to create PRs that are merged within a day. So let’s imagine a scenario when you are starting to work on some task in the morning and it’s merged by the end of the day. Sounds like a perfect word that doesn’t exist, but our experience is a bit different. Miłosz [1:24]: Can’t be possible. This is our last live episode, so feel free to leave comments in the chat, or we’ll pick them up during the discussion, or at the end we’ll have a Q&A section. So let’s kick off with why we need to review code code at all. Robert [1:45]: Yes, I think it’s important to start with the fact that a lot of people are tracking code review as gatekeeping, so basically to ensure that changes that are not, expected are not merged. And I think it’s bad. It’s a bad sign if code review is kind of gatekeeping. So from my experience, if code review is gatekeeping, it usually means that you didn’t make a good job earlier on agreeing how things should be implemented. So it’s a matter of some good practices in the team, but it’s also a matter of designing things. So in other words, if you spend time earlier on planning that, it should be not a surprise what’s in the PR. Miłosz [2:37]: So if someone opens a PR and they start wondering what exactly it is about, then maybe it’s too late. Maybe they don’t have enough context and they want enough to review it in a reasonable pace. I sometimes heard of people starting PRs with checking out the code and running the applications, trying to understand what’s going on. I think we can agree with this. This should be an anti-pattern, unless something unexpected happens, like your team is busy with something else, I don’t know, you ask someone else from outside the team to review your code. But most of the time, you should know beforehand what the PRs will be about. Robert [3:30]: So, if it’s the case… Miłosz [3:32]: It’s a perfect word. Robert [3:34]: That can exist, but we’ll go into that a bit later. But if gatekeeping is not the reason why you are doing code review, it should be some reason why you are actually doing code review. And from my perspective, one of the most important reasons why you are doing code review is the knowledge sharing. And I think it’s not mentioned very often, and I think it’s some kind of hidden advantage that maybe we don’t think that often. But this knowledge sharing is important because when you’re doing review, you’re learning how something is implemented, and it’s helping to spread this knowledge how something is implemented in the team. And it’s pretty important because if you’re spreading this knowledge, it’s easier later to extend this functionality by more people in the team. Miłosz [4:25]: Because you know where to look for changes if you need to extend them or implement your own feature. Robert [4:31]: And while usually it’s not the best idea that one person has some area in the code and just this person is touching this code, well, I’ve seen it sometimes in my life. And okay, while in the short term it can be a good strategy because this person really knows how this code works, but at some point it may not scale if it will be more functional to implement there. And the problem starts to be a problem at this time. Miłosz [4:58]: Also depends how big your team is. For smaller teams it’s probably optimal because if you have a small PR, everyone can take a look and have a rough idea what you are doing. The bigger team is probably a bit more complex, because not everyone can see each PR in progress. Robert [5:22]: Definitely. But still, if you have a bigger team, we’re not talking about, for example, five people. Because I think if you have five people, it’s a bit easier to share this knowledge and be in the stage that everybody knows what’s done. But for example, if you have 15 people, it’s a bit harder and obviously it doesn’t make sense to not share for everybody. But I would say that if you have some subset there, that makes much more sense. Miłosz [5:49]: I sometimes see people mention and code reviews are a sign of lack of trust. You don’t trust your teammates because you need to review code they’ve write. I think it’s funny because I find it the other way. So I actually trust them to review my code because I don’t try to trust myself enough to know I don’t make any mistakes. So it’s good to have another pair of eyes. Just take a glance and very often they will have some context you missed. If we had no comments on PRs, we would just stop doing them. But they come useful. Robert [6:30]: In other words, you should not look on PRs like gatekeeping, but rather something positive that can help you to improve your stuff. I think it’s also important that it shouldn’t require from the reviewer to verify the changes, because I also heard that some teams were doing it like that, that somebody is doing or doing review and verifying if it’s worked properly. I think it’s probably more your responsibility to ensure that the functionality that you implemented is working. Miłosz [7:02]: Sounds a bit like QA, but anyway, you should be doing QA on your own code. so all right um Robert [7:12]: Okay i think let’s maybe start with why having big prs is a problem because i think if you are working with software most of people agree that it’s big elephant in the room that big prs are problem but i think a lot of people are not doing anything about that like yeah big prs are bad, but whatever. Miłosz [7:34]: Or they just say out loud, okay, guys, let’s not do it again. Robert [7:38]: Let’s not do big PRs. Miłosz [7:40]: But then it always happens. For me, the first one is the most obvious one. It’s just hard to review. It’s a well-known phenomenon that if you have small PR, people will find free defects, but if you have huge PR, they will just write LGTM. And, yeah, I think we know this paint well. It’s very easy to just glance at a huge change set and not go very deep into the details. Robert [8:17]: Obviously, we’re losing in this scenario the knowledge sharing that we mentioned at the beginning. If you have really huge PR, there is no way that you’ll read that and that you’ll be able to learn from that how it works. Miłosz [8:32]: Exactly. And then all the processes like reading, understanding code, comments, and whatever else goes on in the PR just takes longer. And no one likes long discussions especially asynchranos so if you take a huge PR and then on top of this you have 50 comments that will take ages to review and if someone spends weeks or longer working on this PR and then you drop 20 comments on it they are probably not eager to to listen to what you want to say, your suggestions. Robert [9:19]: You’re just thinking, I would like to get it out as soon as possible. Miłosz [9:23]: I just want to be done with it and push it through to prod. So, yeah, it’s probably not encouraging to change this. It becomes kind of like an artificial review process but doesn’t give you much. Robert [9:38]: Probably in this case, it wouldn’t change much if you would not have any cool review. The chance we can push to mine and… Yeah, exactly. Yeah, and another problem with big PRs is that, as you said, when you have the situation when, we are working on this PR for longer time, it’s a big chance that you receive 20 comments and okay, it depends on the team, but usually it may end up with a big rework of the job that you’ve made. So in other words, you will lose a lot of time on writing this code and you’ll lose a lot of time on rewriting it later. Miłosz [10:20]: And you are very late in the process because someone spent already a week or two writing it. And now if you want to rework it now, it probably will be a huge change. Robert [10:32]: And even worse, probably in the meantime, when you were working on that in your basement, people were doing other changes and you probably will have a lot of conflicts. It’s even more funny if other person in the team have PR like yours and this person merged it earlier. Okay, if you are not a lot of people in the team, you can always not review this person’s PR, so you are safe that it will not introduce conflicts. But you see the problem, the conflict of interest. I mean, it’s also problematic for you if the other person is merging some big PR, so you are not doing a review for this person because it’s too late for your PR. Miłosz [11:10]: Or you’re raised to do it first, so you don’t need to deal with conflicts. Robert [11:14]: Or you can add a lot of comments, so this person PR will be also blocked, so you can merge your PR first. Miłosz [11:25]: And then, when the conflicts happen, you get an approve and you think it’s ready, and then you just resolve the conflicts. And two hours later, you are finished, but it takes time to do something more productive. So just spend time on maintaining this PR instead of writing code. And this merging can also introduce new bugs or challenges that no one reviews, so it’s another risk. Miłosz [12:03]: There’s also a longer feedback loop, because communication is asynchronous in the comments, so someone gets sick or has days off, you need to wait for them, they come back, they lost some context, we need a separate discussion about all of this. And also that’s a paradox people might think that small PRs require context switching because you have to very often review code and review what you are doing but in reality big changes are even worse because of this because you need to every time you come back to it you need to refresh this huge context in your mind instead of something very small. So that’s another one. And also if you want to stack PRs, so you want to branch out of an existing branch because you need the changes in there, which is often useful. With small PRs, it’s quite easy to do, not risky, but with big PRs, you block everything. The big PR gets merged. Robert [13:20]: And later you have PR that depend on PR that depend on another PR. And if the first PR in the chain will need to be changed largely, it’s just an horror to keep it up to date. I have PTSD if I think about such PRs. Miłosz [13:40]: And then someone asks when this will be done and says, Yeah, we only need to wait for Robert’s PR to be merged. We’ve been waiting for two weeks now, so at the moment. Yeah, we’ve seen it too many times. And other ideas? Yeah, this lack of visibility is also risky because… If you don’t know when the feature will drop, because a PR takes us long to merge, you may be tempted to use estimates, which is also problematic, but this comes from the fact that no one knows when something will happen. You have this huge PR, which seems almost ready to merge, but you keep being added to it and discussed, and you just never know when it’s finished. And the worst of all, you can create a long-living branch. One that takes three months or so to merge. And it’s a completely different version of the software. Robert [15:03]: It keeps living in parallel. Miłosz [15:07]: If you ever work with this, you probably don’t want to do it again. Robert [15:12]: So, it all sounds bad. I mean, there are multiple reasons. Why we should not do big PRs, but from the other side, for some magical reason, from my experience, having big PRs is very often the problem in multiple teams. And there are a couple of things that I believe are creating big PRs. And again, I think it will be mentioned multiple times in this episode that, in my opinion, we very often have good intentions, but despite that, the problem still occurs. And the same is with big PRs. I think that’s why it’s important to be conscious about what can cause big PRs. I think one of the things that is coming to my mind at the beginning is that we have some kind of complete feature mindset. Robert [16:13]: We like to create a feature and have it work on production ASAP. So it makes sense that we’re working on something, and it’s just nice when you write that and it’s going directly to the production. And it sounds good, but unfortunately often features are not that simple that we can easily implement it within one PR. If we’ll try to do that, okay, maybe someone sometimes will estimate that, yeah, it’s just one day of work. But how often do you think that something is one day of work and it’s ending up with a week of work because there is some hidden complexity that you didn’t take into consideration at the beginning. Miłosz [16:54]: And splitting it seems like additional work. So you might be tempted to avoid it. You wouldn’t want to go. It should be easier, right? Robert [17:05]: Yeah. And I think it’s connected to, planning, but In many cases, when we’ll discuss about causes why we have big PRs, it’s connected to how we plan things, how we did some shortcuts maybe with planning, and I think often big PRs are connected to some surprises that we are creating. So what I mean by that is that… Miłosz [17:37]: So-called missed requirements? Robert [17:41]: Yes, yes. Miłosz [17:51]: If we don’t know some edge cases before we start the work, then you’re in progress of working on a PR, you discover something new, you have to stop, maybe discuss with someone else, Miłosz [18:05]: and then it starts to get tough to push through. Because you put some work, you started some writing some code already, it’s half done, but something blocks you. Robert [18:19]: Yeah, and obviously, if you include something like that in the PR, if somebody will be reviewing that, it will be kind of surprising. Even if you have some planning process at the beginning, if some person is reviewing your PR, this person might be surprised. I mean, okay, we’re planning this, but why does PR have this, this, that, and this? And this is triggering discussions. and discussions is triggering a slowdown of PRs, basically. Miłosz [18:50]: Yeah. Sometimes, if PRs get reviewed slowly in general in your team, you may think, like, no, it makes no sense to… What happened? Robert [19:08]: Oh our camera died. Miłosz [19:12]: Yeah, so let’s maybe take a minute break. Robert [19:20]: It’s tough. Yeah. So, this episode is special in a couple dimensions, because the first fact is that our room that we usually are using is used, so we were not able to use that. And our camera is overheating, so… Miłosz [20:01]: Great. Summer is there. Robert [20:02]: Summer is there. Cool. Miłosz [20:04]: So, as I said before… Robert [20:06]: What do you see? It’s live. if it’s not pre-recorded. Miłosz [20:10]: So if your PRs get reviewed slowly in general in your team, you might think like, okay, it makes no sense to quit additional PRs because I will wait for all of them for a long time to be reviewed. Maybe let’s just make a bigger run and review it once. Miłosz [20:30]: But again, the huge PR gets reviewed even slower. So it’s not a good solution really. Robert [20:40]: And I think it’s also one anti-pattern that I often see. It’s the teams that are not real teams. What I mean by that is that if you have a couple developers that are independently working on some features, they are actually not a team. They are a set of individuals working on something. And at the end, there is no shared responsibility to deliver some features. So, if you have no shared responsibility to deliver features, it’s hard to convince somebody to read your PRs, basically. Because nobody cares. Miłosz [21:16]: It’s not their job. Robert [21:18]: My job is to implement my features and save you. Miłosz [21:22]: I don’t want to spend time on your work, because I’m responsible for mine and there are stakeholders waiting for my work to be done. Robert [21:32]: Exactly, exactly. And I think it’s a big problem, because if you are not working as a team, but a set of individuals, well, there is no pressure or nobody feels obligated to review stuff. Miłosz [21:50]: Yeah, so in this case, review is not the team’s responsibility. Basically, no one’s responsibility. Just beg people to do it for you. That sounds like an issue. You can see the setup when there’s a feature mentioned and some stakeholders know to which developer to talk to about it. Because they are seen as owners or masters of this area. This is one of the indicators, I think, that this happens. Robert [22:33]: We already covered a couple reasons why having big PRs is a problem and how big PRs happen. I’m not sure if you agree with me, but I think one of the most important parts here and reasons for them is lack of planning, basically. Maybe I would say deep enough planning. I like the phrase that five days of coding can save one day of planning. And I think it’s often true and it’s often a problem in multiple teams. I mean, we’re sort of developers, engineers, we like to solve problems with the code. So if we hear about planning, meetings, no, no. Yeah, yeah. It’s always like, no, no. Yeah, yeah. Talking to people, especially non-developers. We are pretty allergic to that. Miłosz [23:27]: Especially after you hear about the future, you already have a database schema in your mind. You already know how to solve it. All you need to do is just sit down and write. Robert [23:40]: Yeah, and I kind of understand why people are allergic to deep learning, but from other sides, I think it’s also creating a lot of problems. And often it’s visible in CodeReview, because I think a lot of people are blaming the CodeReview process for many things like code review is such bad, it’s such a waste of time, it’s taking that much time. But I think in previous episodes, we also sometimes had this, let’s say, non-direct, visibility of some errors. So for example, some people are blaming DII for making complexity of the applications because dependency injection is so complicated, etc. Miłosz [24:30]: So it’s like the tip of the iceberg. Yeah. Everything is the issue, but underneath there’s some more complex issues. Robert [24:36]: And I think it’s a similar case with code review. A lot of people blame code review for a lot of bad things, but I think it’s just surfacing other problems that you have in the team. For example, if you have, I don’t know, 200 comments in the code PR that you submitted, Probably a good question to ask. Okay, why you have 200 questions there? Why you didn’t agree on that earlier before you started implementation? Because when it’s already implemented, reverting that, changing that, it will be super expensive. If you spent more time a bit earlier during planning and planning properly, probably you’ll waste much less time. And I think it’s not about always planning everything for 100% to not have an unexpected complexity, because it will always happen. And trying to plan that accurately, it will be super expensive in terms of time. But I think it’s, from our experience, we see that spending more time on planning, it can definitely… Miłosz [25:47]: It’s easy to change the whiteboard. you erase it and rewrite it. It’s much harder to change a 10,000 long 10,000 lines long PR with 100 comments. The earlier you discuss things and agree on how to do it, then the easier you iterate this. Robert [26:11]: I know that a lot of people are like, let’s skip planning, let’s just write that and ship that and it will be everything fine. And I know that I was also like that when I was younger and less experienced, but later you can kind of learn that, okay, it can give you some acceleration at the beginning, let’s say, but later it can be very problematic. Because, again, you lose many of the advantages that you will have from this code reveal. Knowledge sharing, or if you are doing a kind of more strict process of code reveal, it will be just blocking at some stage, because this stupid approach will be at some point, probably will go with some strange direction that you didn’t agree with somebody else, and it will be just surprising the PR, the surprise that we were mentioning a bit earlier. And again, PRs shouldn’t be surprising, so it should be implementation of what you agreed on high level earlier. Miłosz [27:16]: Yeah, so it’s another paradox. You save time on planning, but in the end, you spend even more time on the reviews. Just like with the paradox of context switching, when you think smaller PRs cause context switching, but actually the big PRs cause even worse. Robert [27:40]: And yeah, I think it’s interesting because on the first side, my thing that, yeah, splitting to multiple PRs, it will just add a lot of overhead and it sounds not logical to have multiple PRs instead of one big, but I think what I can recommend is trying to try using smaller PRs and see how different it will be. Obviously, it requires a team culture that supports that, because if you create 10 smaller PRs and nobody will review it kind of immediately, it will not help. And you cannot do much about that, and probably you will end up with big PRs at the end anyway, because it will just start to stack. So yeah, it wouldn’t make sense, but we’ll go into that, what you can do in the team to encourage teammates for a quick review. Robert [28:40]: So, yeah, we started to mention that missing enough planning obviously can help to create bigger PRs. But again, it’s also important to not go into extreme, because I think it’s also easy to go into really extreme and try to plan for everything, to go really deep into the planning. But it will be very expensive to do. I think it’s probably not a possible event because without touching the code it’s really hard sometimes to find all edge cases and complexities that you have. Miłosz [29:16]: The balance is in deciding what you want to agree on. For example, we skipped on most implementation details like exact database schema or naming of things. That was up to the implementer most of the time. High-level architecture, like what endpoints do you want, what services this impacts, and so on. Some very high-level picture of it, that’s super useful to have early. Robert [29:47]: I think it’s all about this non-surprising PR at the end. To think what you should plan earlier, to not surprise the reviewer when you are reviewing that. And if you were not working like that earlier, obviously, it will take some time to learn how to do that. But I think it’s a skill that you are just getting with time. You’re implementing more, you’re learning more how, okay, this is surprising somebody in the PR. Okay, maybe we should raise it during that. Miłosz [30:20]: Next time, let’s agree with what endpoints they will be exposed. This is a quick discussion, right? Of course, it’s better to have before any code is written. Like, do we want endpoint in this service, or do we want a message handler in this service? High-level, that can take 5 minutes. But after someone implements it, we can make one hour to revert and replace it. Or even worse, if the discussion is asynchronous. Robert [30:52]: Another thing that it’s worth doing is also including in the planning discussion some product person, let’s say. So it depends on each company how it looks like, but often you have some, maybe not stakeholder, but maybe some product owner or product manager or whatever. And it often may be helpful. So obviously it’s required not to have some discussions that are not super technical because this person will be just bored and this person will not understand anything. Miłosz [31:23]: So they will do it first. First understand the product. Robert [31:28]: But I think it’s nice because it also can help to cut scope a lot. So I remember that And often when we are doing it like that, we included product people during the planning phase. So when we have some planning meetings and we have this product person in the meetings, it sometimes helped a lot to simplify scope. Because we started to discuss some features, but we had like, okay, it will be very complicated because this, this, and this. And the person in the same room said, okay, so if it’s complicated, we can just keep that because it’s not that important. And sometimes you can lose hours as a developer to try to figure out some super complex problem, but at the end, nobody will care that much. Miłosz [32:11]: I think it’s one of the most important skills for product engineering, and I think it’s super rare as well. Because most developers, when they get a clear description of a problem from a stakeholder, will just say, okay, we can do it, because they want to be the hero and show, and we’ll be like this. Robert [32:32]: And we are problem solvers, so if we see problem, okay, we need to solve this problem, it will be so cool. Miłosz [32:38]: Especially if it’s well defined, if you have all the requirements and so on, you might be eager to just jump into it. You will tell them, okay, we can do it tomorrow, no problem. They will be like guys you are the best team in the company but then you get stuck on these complex cities and you need to talk to the stakeholders again and it takes a long time to read about everything So, I think it’s a super important skill to be able to pull them early to the discussion and see where you can cut the scope. Or, even if it’s not out of scope entirely, just find the V1 you want to deploy. And that’s super useful to have. Robert [33:28]: And I think we’re often kind of missing this, that we are implementers of some Robert [33:35]: external requirements and again, we’re trying to be heroes or. Miłosz [33:39]: I am just a developer tell me what to do and I will do it so that’s a common approach and E-definance because you can just trust people know what they want to help them figure out what exactly it is they want so basically, Robert [34:01]: One unexpected strategy to make your PR smaller, implement less things. Miłosz [34:08]: Yeah, or start with less. That’s maybe more precise. Try to start with a small scope. Robert [34:17]: But sometimes it’s not possible. I mean, sometimes we’ll ask product master or camera, right again. So, okay, can we simplify that? No, you can. So, what’s the strategy in this case? Miłosz [34:34]: I guess, breaking down changes, right? So, our ideal scenario would be having PRs that are implemented within one day. So, as you said at the start, someone posts a PR in the morning, it is merged on production in the afternoon. Something like this. This cycle time would be ideal. Miłosz [35:09]: And there are actually some strategies you can use. Miłosz [35:17]: Probably the simplest one is to merge code that’s not ready yet. behind the feature flag. But it’s critical that you’re able to manage feature flags in your project one way or another, even if it’s just some configuration. But it allows you to merge the PR, so it’s already on production, even if it’s used. But you avoid these discussions and conflicts, and so on. We just work on the code that’s already on the main branch. Even if not exposed yet. And you can have the tests running, you can enable it on some pre-production environment, you can show it to stakeholders in some very early form, some MVP form, and get some early feedback from them. So that would be my number one for breaking down changes and for small PRs. Robert [36:30]: Another thing that comes into my mind, it sounds pretty obvious, but basically to have some discussion to some RFC, ADR, some design document, and basically discuss it with the rest of your team and have some plan how we can lead it to some smaller parts. We already mentioned it multiple times. So, I think this magical value is one day that I know for so many people, it sounds like, no, it’s not possible to start implementation in the morning and merge something by the end of the day and not working until midnight. I mean, I know that for some people it sounds crazy, but it’s very possible. Miłosz [37:14]: Yeah, I remember having five minutes cycle time. If it’s small, you can just ping someone and then take a look and in 10 seconds you get an approve. Robert [37:26]: And the rest of the people is aligned because you spent some teamwork to discuss how it should be implemented. And it’s not surprising for people. So somebody can take a look quickly on that. See, ah, yeah, it’s what we discussed. Looks great. Merging. Miłosz [37:43]: And because it’s merged, for the next PR that’s built on top of this, Those changes are already out of context, so you don’t spend time discussing them, but that’s your dream. Robert [37:57]: Do you have maybe some tips from our experience, how you can split this work Robert [38:01]: to make it working nicely? Miłosz [38:05]: Yeah, so you can either split vertically or horizontally. And both kind of make sense. Probably vertically makes more sense at first, so instead of big chunks of the feature, you’ll deploy small parts of it. So for example, you can… Let’s say if you beat a dashboard or something like that, you can deploy just one chart or one button. And the endpoints behind it and so on. So it’s something you can deploy, for example, on staging environment or even in production. And show it to your stakeholders and ask them, is this what you want? It can be a very simple feature, part of the entire feature, but it’s already something you can ask them about. Robert [39:03]: Yeah, and I think you can go even one step further because, okay, you can have some vertical slice to prepare a chart, but often though vertical slices of entire functionality, it can be still pretty complex. You can also slice this vertical slice horizontally. For example, have separate tasks for SQL repository. So it works pretty nicely with clean architecture that we’ve been discussing in the previous episode, so I recommended to check it out. What Longstar showed is that it helps pretty nicely with splitting these features layer by layer. So it can be implemented by one person or by multiple people, and you can implement… Again, it’s good for more complex features, because for some features it doesn’t make sense to split that because it will just use a solar head. But if it’s a more complex feature, let’s say that it’s one week work for one person. You can always split it by layer. And for example, every person in the team can take one layer. And instead of implementing something for an entire week, you can have five people that are just separate slices with a grid API layer, and it can be even implemented within a day. And it’s cool, because you can show it to some stakeholders after this day and ask, okay, this is something but yeah? No? Okay, so let’s iterate on that. And it’s much, much, much faster. Miłosz [40:24]: Yeah, I think it’s another superpower that’s very rare in tech teams in general. It gave you to work in parallel on a single feature. More often you will find this approach of, okay, so this person is responsible for this feature, see you again in two weeks on the Spring demo. Robert [40:41]: And usually it’s not two weeks because there are some challenges or block errors. Miłosz [40:46]: So instead, if you can agree on the interfaces, then one person can implement the HTTP handler, the other one the message handler, the third one the database adapter, something like that. And it’s quite simple, because you agree on the interfaces first, and then you implement them. It’s really straightforward, and we know it works because we’ve seen it live. Robert [41:11]: And we have some comparisons. It was a process that we worked over a time, and we have comparison how it worked earlier, when we were not doing splitting, and later, when we did the splitting, and we were able to iterate that fast, and it was cool. Miłosz [41:27]: Except both vertical and horizontal, splitting is not that easy. Maybe the horizontal is easier because it’s a technical one, but the vertical split is related to product so you need some product sense to understand which features make sense at all and you need to discuss it with someone probably to understand better Oh no, Robert [41:50]: Talking to people. Miłosz [41:51]: I think it’s worth doing I’m just saying, I think it’s rare because it’s not trivial you need some effort from the engineers to do it Robert [42:01]: But after you learn that, and the team will learn that, it’s… Miłosz [42:05]: That is quite natural, yeah. For sure you can learn it. The entire teams can learn it. And yeah, it will be the best if, side note, there’s no single team lead doing this, but it’s encouraged so everyone can own this with time. Robert [42:25]: And I think it’s another, we again see the same pattern, that it’s some kind of overhead myth here. You may think that, okay, cutting it by horizontally and vertically, it’s that much overhead for that. But it’s not logical that in theory it will require more time. But from the other side, if you will not cut that and one person will implement that, for some magical reason, it will take longer anyway at the end, even if there is some overhead by cutting that. But if only one person is implementing that, this person may be bored, may have other interruptions, context switching. So at the end, there is also some hidden cost of one person doing all entire work for two weeks. And if you add all the complexities that we mentioned with the non-splitting PRs, but having one big PR instead, it’s starting to be much, much, much harder and take much more time. And again, at the first sight, it may sound like it’s requiring much more effort, but I recommend to check that because after you see that, you can see that in practice, it’s much, much faster. the planned things. Robert [43:51]: But you may think that, okay, it all sounds nice, but it’s not possible to split all functionalities that are maybe okay. Those functionalities are a bit easier. You can probably split most of them, but sometimes there may be some work that is not possible to split. And the first thing that I have in mind is some kind of big refactoring. So, if you’re doing some big refactoring in a project, sometimes it may be hard to split that. So, I like to compare it to Fatality from Mortal Kombat. So, you know, it’s like you’re getting backbone and, you know, it’s getting all the bones. Miłosz [44:35]: So, there are lots of unknowns. You just don’t know what you will find. Robert [44:39]: Yeah, but I also mean that sometimes touching one thing is creating some kind of domino effect. So you touch one thing, but to make it work, you need to adjust a lot of things in the project and need some kind of atomic change at the end. So if you touch that, you probably need to spend one week to just fix everything. And it’s, I think, also a pretty interesting case how to deal with that. Because, OK, this one person can do this big refactoring for a week or two weeks. Later, give a PR and, guys, can you review that? Because, you know, it’s important refactoring. If you are looking at, hmm, okay, two weeks of work, 2,000 files changed, 200,000 lines of code added, the same removed, okay, good luck reviewing that. So, one of the tactics here may be, okay, it just, okay, sounds good, let’s merge that. But I think it’s not the best, because usually you can end up with some brain changes in the code base, and everybody later will be, maybe not surprised in the PR, but surprised if somebody will be, because I’ll take something like that. Robert [45:48]: So, do you have some examples from our teams? What we did in such scenarios to avoid this? I think probably it happened to us multiple times to do this big PR and close eyes and okay, let’s merge that, whatever. But I think it maybe wasn’t a disaster, but it wasn’t the best strategy. Miłosz [46:11]: You can try mob programming. Robert [46:13]: Sounds like an app. Miłosz [46:14]: If you are brave. Robert [46:17]: So, what is mob-programming? Because I think a lot of people may not know mob-programming. So maybe some of you know what spur-programming is, but mob-programming, I think it’s much more exotic. Miłosz [46:30]: It’s spur-programming, but more than a pair. It’s basically a group of people coding together, and one person is the driver with the keyboard, and the rest comment and observe. It’s very similar to per-programming except there’s an entire team, if it’s small enough. Like people, I’d say. And yeah, and you take turns coding and implement the feature together. Robert [47:02]: Sounds like a big waste of time, isn’t it? Miłosz [47:06]: Yeah, like all meetings. I think we should all code alone, Robert [47:12]: Not discuss it in the basement. Miłosz [47:16]: Yeah, so for many, maybe not for many, but for some kinds of work, this can work well. Because of the knowledge sharing, primarily. So you don’t need to waste time reading PRs, especially if they are big, because we do some refactoring or scaffolding of the project and so on. Instead you do it together. So for sure it doesn’t make sense to do everything like this. If you implement an HTTP handler, there’s probably no need for four people to watch how you do it. But if it’s some critical domain code or some huge refactor, it’s probably quicker this way than if everyone reads the PR separately and comments. Robert [48:06]: I think it’s probably good to consider alternatives, if you are thinking that it’s a waste of time. Because, okay, what are alternatives? So you can probably just close eyes and merge these big refactoring PRs. It’s some solution, but usually it doesn’t end up well. Sometimes, okay, the person that created this big PR can arrange some meeting and show some highlights of how this feature or refactoring was done, it’s probably a bit better, but still, it’s no way that you can go over, entire thinking process that you did in one or two weeks of development. Miłosz [48:50]: So you need to consider if it makes sense. Robert [48:53]: But do you have maybe some tips how to run more programming sessions to keep it efficient? Because you have an entire team in the room, potentially, maybe on the call. I think it’s harder. I mean, I think it’s easier to do it in person, but… Miłosz [49:12]: Everything is harder on the call. So watch out. But yeah, if you do timebox that should help, so make sure there’s no one-person coding, but you switch seats. Robert [49:24]: Yeah, I think it’s very important because it’s easy to end up with a situation when one person, sometimes some team lead or whatever, is coding everything and everybody is looking on their phones to update the code because it’s… Miłosz [49:38]: So I need to keep everyone engaged. Don’t speak away. Watch out for CO2 levels in the room if you use a room make sure your team knows the process and everyone is aligned for example a big risk is if you let people drift into a topic and start discussing every line in the code that can end up terribly I think it works best if your team has been working together for a while. They know each other and I remember we had sessions with some new joiners to the team and that wasn’t going that well because they were not aware of some patterns we were using. They questioned everything. Robert [50:37]: Yeah, so I think it’s good to have some facilitator. Miłosz [50:41]: Exactly, you need some heavy facilitator. We can cut discussions. Robert [50:46]: I remember that what often works for us was that, okay, if we’re parking some discussions, sometimes we left a task for that, and this task could be done asynchronously. Because, again, running a session for an entire team is just expensive in terms of time, basically. And you need to be sure that you are using this time very efficiently. Miłosz [51:16]: So, sometimes you can also do it with a short session just to start a project. I remember when we started new services, we just use Mood Programming, where we just bootstrap the project or something like that. Robert [51:30]: Yeah, and it was pretty cool because we had really nice boosts at the beginning, so everybody was aligned. how it was implemented, and it was speeding up work at the beginning very, very much. Because, again, it’s also the theme that is going over and over during this episode, so knowledge sharing. So this very deep knowledge sharing that actually accelerates the work a lot. Because you may have multiple tasks to implement, everybody knows how everything else in the project works, so everybody can pick up the task and tasks are not blocked because, for example, this task can be unimplemented by somebody. Everybody can take that, everybody can review that, and it’s accelerating work a lot. That’s kind of hidden functionality of that, let’s say. But yeah, it can help a lot. And yeah, this is also the same with mob programming. So basically you can mob program the feature at the beginning of the project and and later iterate really, really fast. Miłosz [52:38]: Maybe we can talk about code review culture, going back to the topic. Robert [52:43]: Yeah, because I think it’s pretty important because a lot of ideas that we have here, it requires some nice culture in the team. Because again, you can do small TRs, but if nobody will review them, it’s pointless because you end up with waiting for ages for the reviews, so it will be not helpful. So what do you have in mind from our experience? What are the important traits of the team that have high review culture? Miłosz [53:19]: Maybe what you mentioned before, making the team responsibility, not individual responsibility. The team is responsible for delivering the entire feature product. Anyone should be able to review code for knowledge showing. Avoid the silos where people are closed in their own parts of the code and they know everything about it. No one else is able to suggest changes there. Robert [53:52]: And by everybody we mean that, okay, if you have 15 people team, or maybe not entire team, but you can have some… Okay, it’s really problematic if your team is 15 people. Miłosz [54:03]: I guess ideally we have two or three people working on a single feature at a time and they can review each underscore. Robert [54:11]: I would even say that maybe five. Five is also good because you can have some extra capacity for that. And it’s a number out of an hour. Miłosz [54:22]: Yeah, but I mean, it’s probably an anti-pattern to have one person responsible for all the reviews. So like the leader with the grey beard who knows everything and everyone just asks them to review their code. That’s probably not what you want, because this person becomes a bottleneck. Robert [54:44]: It’s also not good when you have your buddy, for example, that is doing your reviews. because it’s a bit better, because it’s not in the hands of one person. By the way, I think many of you may not know the Phoenix Project book. I don’t remember the name, but it was covering a similar situation of one person that was the person that known everything, but it was also a bottleneck at the end. So it’s a very similar situation. Miłosz [55:16]: I would usually just ask the person who has more experience in the area than I have. So, they can give you some insights on the way that or maybe Robert [55:25]: Just have a team channel and. Miłosz [55:28]: Oh yeah Robert [55:30]: And pick up it and yeah. Miłosz [55:32]: Yeah, that’s yeah, that’s the best thing times, I remember. What I mentioned about the 5-minute cycle time is when you just post a PR, post a link to some Slack channel and then you get an approval for comments 10 minutes later. That was my ideal working condition. Robert [55:56]: Yeah, it’s possible. It requires times and good culture. Miłosz [56:03]: I think the main concern is there is context switching. But once again, switching context to a 50-line long VR is something completely different than 500-line VR. Robert [56:16]: Or even 200 lines. I mean, it’s also something that you can review pretty quickly. Miłosz [56:21]: Yeah, I just mean the size in general. If it’s in tens of lines, hundreds of lines, probably still kind of fine, but if it’s thousands, then yeah, probably someone will open it and then they will go for a coffee break or back to their own code and we’ll just make a mental note to come back to it later. If it’s 50 lines or 100 lines, they can just read it in one minute and tell you what they think. Robert [56:54]: I have actually one thing in mind when we’re discussing that, that one of the problem with the discussion that we have now, it’s that having this so having everything working requires multiple things. For example, if one thing doesn’t work, it can break the good PR flows. For example, if, for example, it takes ages for people to pick up PRs, so everything else can be problematic. So it’s important that to not pick some parts of this, but try to, improve it from multiple angles. Okay, we’re going back to the strong code review culture. So other thing that it’s maybe also not obvious is actually how we are doing reviews. So I have two things in mind. Robert [57:46]: So the first is the feedback culture and feedback mindset. So you should not try this last. Robert [57:59]: You should not take this code review feedback personally, let’s say, because I think it’s often that people are looking like, oh, somebody commented on my code, I’m so stupid, and this person is also stupid because he doesn’t like me or whatever. So, try to avoid that and try to separate that thought. This is the cult, this is you, and this is a separate thing. It’s also always good to assume good intentions, because it’s easy to go in a direction like, ah, somebody is commenting on that, would like to make my life harder, and why he’s doing that. And I think it’s very rarely true, so most often it’s that actually somebody cares on the other side, and this person would like to create code that is maintainable and easy to work with. So I think it’s important to start with mindset by default that the person on the other side have good intentions. So this person cares and this person would like to just have nice code. Obviously, sometimes it’s going into extremes. Sometimes the person can have also some bad intentions, but it’s more about the default approach so I think by default and it will be probably true in 80% or 90% cases that people have good intentions unless. Miłosz [59:19]: They need to pick yeah Robert [59:23]: And I think it’s another problem so if you don’t know what nitpicking is so you know it’s all those comments like okay it’s not important but maybe you should use spaces here or, change how it’s formatted to a different… Miłosz [59:41]: Can you order the variables alphabetically? Robert [59:46]: Yeah, yeah. And all those comments that are… Miłosz [59:50]: Basically, I would like this code to look like this, but you do it differently. Robert [59:55]: So you can resolve this comment. I don’t mind. But, yeah. So this is nitpicking, long story short. And in my opinion, If I do a code review and I think about adding a nitpick, I don’t do that. Because if it’s not that important that it’s not required for merging, it doesn’t make sense actually to add any comment, spend other person’s time on resolving that or maybe addressing that, because it’s just not important. In my opinion, all those nitpicks should be covered by the Linter, basically. So if you have some styling requirements, just put Linter for that and not spend human’s time. Miłosz [1:00:46]: Because they are often cosmetics, so our Linter is good at catching that. Robert [1:00:50]: Exactly. And Linter have also one nice feature that it has a pretty quick feedback loop. Because if you are leaving a comment in the PR, some person will read it with some delay, some person needs to address that. It’s just a lot of waste of time. And with Linter, you’re just running Linter. You can see, in the worst case, a couple of seconds later. In the best case, in less than one second, that everything is fine or what should be fixed. You can format that and it’s fine. So I think this is how it should be solved with NitPix. So if something is that important enough to check for that, fine. Let’s put linter for that and linter will ensure. If it’s not important, why to do that? Miłosz [1:01:34]: Makes sense. Also, as the person submitting the PR, you can do some things to make it easier. Something I like to do is being my first reviewer. Just checking the trendset and often you will find some leftovers. Instead of someone commenting it for you, you can just do it yourself. Make their job a bit easier. and they can also include a short description, like why you did something. We sometimes comment on the onlines ourselves just like a heads up for the reviewer like, you know, I know this can look strange, but I did this because of something. Robert [1:02:21]: Or even better, leave some comments in the code that explains why it’s done in some way. And I think it’s important that you mention that to describe why not how. Because how it’s written, what’s written there, is visible in the code. I mean, it is the source of truth. But why? It’s not in the code. You can only see how it’s done. Miłosz [1:02:44]: You generate the how with AI. Summary, but it’s not really the point. The point is to give some extra context, not what is already there, but different words. Robert [1:02:54]: Exactly. Miłosz [1:02:56]: Maybe one other tactic is draft PRs. If you have some very rough idea, especially if it’s something experimental or something your team didn’t do before, I would create a draft PR and very quickly, in a sloppy way, share with everyone and ask them to give me some first impressions. Robert [1:03:20]: Especially if you discovered something unexpected in a PR. So remember, don’t surprise the reviewer, and if you feel that, okay, this may surprise somebody, yeah, good place for a draw. Miłosz [1:03:34]: This works only if you have this early feedback culture. You can get someone’s comments in 15 minutes. I think it’s way better to discuss it early, because if you go deeper into this and you have more tool you work, if you have some insights. And if you ask them early, it’s even better to have some quick meeting to align and agree and then implement. Robert [1:04:02]: It’s also connected with this idea that you are working as a team, not as a set of individuals. So, you collectively care about this draft. Miłosz [1:04:13]: And you have to see why we are in that. Yeah, but this needs to be in place because you will need to wait two days for our draft PR review. It doesn’t make zero sense. Robert [1:04:24]: Exactly. And I think it’s actually something pretty important to touch on. So, what are the tactics to accelerate review cycles? Because I think it’s a requirement for most of the things that we mentioned. So, if your review cycle is slow, it doesn’t make sense to split your PRs to smaller PRs, because if you are creating 10 smaller PRs and you are waiting for them for one week, it’s not a big difference between making one big PR… You’re just stuck PRs needlessly. Yeah, probably in this case it’s better to not split PRs and have big PRs that nobody will really review or create 200 comments. Probably it’s better than having 10 PRs like that. Unfortunately, so I think this accelerating review cycles is key here and keeping this cycle time more is also critical. So one of the most important things and mindset to learn is that making reviews should be. Robert [1:05:33]: We should prioritize making review, basically. So it’s pretty compatible with the lean Kanban approaches. Robert [1:05:42]: So basically the idea is that you should prioritize work to be done. So if you have something that is close to be done, PR, this is work that is almost done, let’s say, you should prioritize that and try to get it out of your board, let’s say, as soon as possible. And, It’s a simple idea, but, again, very, very important. Miłosz [1:06:06]: So, I guess, higher priority than full-in-progress, because it’s closer to production. Robert [1:06:11]: Exactly, exactly. So, obviously, it requires some context switching. So, if you’re working on some feature and you see that there is some PR to be reviewed, obviously, you probably shouldn’t, if you’re working on something, jump straight into that and review that. But maybe there is some person that finishes the job and can jump in this PR. Miłosz [1:06:37]: Yes, contact switching can be a problem, probably for some people more than for others, but you probably don’t work eight hours looking at the code anyway. I mean, writing. Robert [1:06:49]: You are not. We’ll need to talk about it after. Miłosz [1:06:52]: Maybe you’re supposed to. Robert [1:06:54]: We’ll talk after recording about that. But jokes aside, yeah. And I think it’s going back to the knowledge sharing. So we covered this knowledge sharing multiple times, and I know that some people are, why we need knowledge sharing? And this is actually a great example why it’s important, because if you don’t have only one person that is able to review your QR, but entire team can do that, you don’t need to wait for this one person. You can probably have one person that finished his work or her work and can look into this PR, and it’s making this acceleration working, basically. If you have only one person that can review your PR, obviously it’s making the cycles much longer, because only this person, multiple stars in the sky need to go into a special… Miłosz [1:07:48]: Into a line. Robert [1:07:50]: Yeah, yeah, yeah. Miłosz [1:07:54]: Yeah, and the end result is this cool flow of PRs going smooth from coach to production. Robert [1:08:02]: Yeah, so basically you can work on something in the morning and by the end of the day, and you’re not working for 18 hours, and it can be merged within this day. But again, it’s important mindset shift because I know it’s natural to focus on your job to be done. But again, try to focus on finishing work and work as a team, not set of individuals. Miłosz [1:08:35]: That’s a culture thing. Not easy to change, but helpful to have in place. There’s also this kind of controversial topic of metrics. There are many tools that you connect to your GitHub and that you track things like average PR size, cycle time, timing review and so on. And they can be useful. I think there’s a risk that someone uses them for measuring performance. That’s probably not a good idea, because people will start gaming them and that’s not what you want. Especially if you try to compare teams using them or something like that. I’m not recommended, but you can use it inside the team for insight or maybe to find some outliers. If you have one or two huge PRs in one iteration, you can ask yourself why this happened. Maybe it was okay, maybe we could improve something. Robert [1:09:45]: But I think it’s also important to set expectations. So we said that, okay, in this ideal work, you are starting in the morning, merging by the end of the day. But also, to set expectations. In our teams, it also wasn’t like that for every PR. So it was probably for 80%, sometimes 70% of PRs. But I think it’s enough. Miłosz [1:10:07]: So the target is not perfection, right? Yeah. Just that most work goes like this. Sometimes there will be some crazy refactor that you need to do at once. Maybe it’s fine. Robert [1:10:19]: Sometimes you probably should reflect that, okay, why this took longer, what we can do to improve that, and try to do some kind of continuous improvement. As we were already saying about fancy Japanese words, Aizen. Miłosz [1:10:34]: So it can be a good starting point for retrospectives in discussion. Robert [1:10:40]: Yeah, I remember that when we were really into optimizing that, we on the retrospectives were looking on these outliers and discussing, okay, why this PR took, for example, three days, for example, to be merged. And, okay, is this something that we can improve here? Okay, this PR was like that because, I don’t know, it was some unexpected complexity there and, okay, it happens. Probably couldn’t do much about that. It’s fine. Or maybe we can see sometimes, okay, we didn’t spend enough time on planning here. Maybe we can, for example, next time, plan more about this, because this, and just improve. And again, this is a skill. So if you work in such environment for a longer time, you’ll also learn what things are worth to do deeper planning and work when it’s just not productive. Oh, we have some comments. Hi guys, I came late, I have concern. How could you create small PRs if you are working on tickets? And maybe the ticket has complex tasks. How do you recommend doing small PRs in those cases? Miłosz [1:12:04]: Complex task yeah if you can follow up what you mean by complex but two things we mentioned before so one is splitting the ticket into smaller tasks but by vertically by the product features Robert [1:12:26]: Maybe it’s the situation when you have tasks from somebody up and you are basically implementing the tasks that you are receiving from some product owner somewhere that is planning your work. Miłosz [1:12:40]: Oh, yeah, right. I didn’t consider this. Because it’s going to be like this. Robert [1:12:48]: I think the root cause, if I understand it properly, the root cause is probably that some person is planning your work, as long as I get it properly. But I think it’s the root cause. I mean, it shouldn’t be a process like somebody’s giving you tasks that you should implement and deciding on how to state that. It should be probably the opposite. So you should discuss with this person, okay, what we have to do. So you should actually receive the problem that is to be solved, but not tasks that will be implemented, basically. And it should be some collective process to agree how to do that. Unfortunately, I know it’s not possible in all companies, and it’s sometimes impossible to change. And in this case, change companies. Miłosz [1:13:36]: If you can split it, try more programming. Robert [1:13:41]: Or maybe just split it internally, so you can have one task from somewhere else, do process yourself to split it to multiple smaller tasks, and just have smaller PR. So we already mentioned it a bit earlier, that you can merge code that is not functional yet, So it can work in the background, or maybe not. You may have some functions that are implemented but are not included in the production process, let’s say. In the last PRs, for example, you can have multiple PRs, and in the last PR you can integrate everything. In the meantime, if something is much more complicated, you can also do the… Miłosz [1:14:38]: It’s kind of what you mentioned before, Robert [1:14:41]: Right? Miłosz [1:14:44]: Just because a task is complex doesn’t mean you can’t split it into small PRs. You can have the scaffolding first and then use feature flags to deplace on that code to production before you start using it. I’m not sure if you’ve listened the entire episode, but we mentioned this before. Robert [1:15:06]: But yeah, the downside is that, again, it requires the entire team to kind of come into this approach. because if you have a long review cycle and you have many smaller PRs, it may be just blocked too long. You will not see the benefit of that. I also wanted to share that if you are splitting such tasks in more smaller tasks, if it’s more complex, you can also run shadow mode before the last merge. So you can run new code along the old code and compare if this code is creating the same result without storing it to the database or something like that. So sometimes you may be doing some bigger refactoring or something like that. So maybe you have some tests for that, that you see that, okay, it’s passing, but sometimes you would like to also compare on the production if it’s working as previously, but without breaking anything. So you can run two functions like this alongside within one service and just compare result at the end. Run it for a week, month for a complex one, and if you see that the results are consistent and it’s working nice, you can just swap that. But again, it’s for much more complex tasks. Robert [1:16:23]: Ah, okay, so I was right. The case is that tasks are assigned to me, and the tasks have goals. In some cases, I need to make changes in several repositories, and maybe I need to touch several files at once. So yeah, in this case, I’ll say that this is probably the pattern that tasks are assigned to you. In my opinion, it should not work like this. Miłosz [1:16:45]: So yeah, but what can you do in this situation? Robert [1:16:49]: Yeah, yeah. So, as I said, unfortunately, in many companies, you cannot re-change that. So, the thing that you can do is you can split these tasks internally. And, again, it depends on the team. I know that sometimes it may be hard to change the entire team culture to smaller PRs. Because, okay, maybe you can split your tasks into smaller tasks and try to have smaller PRs, but if nobody will look in your TRs and it will be no teamwork, because, okay, I assume if tasks are assigned to you, it may be problematic, because you are not collectively working on that. It’s your responsibility and it’s harder to actually integrate with that. Sometimes, unfortunately, it’s a bit hard to change the company culture. Miłosz [1:17:35]: Yeah, but we can still do just this on the technical side. Robert [1:17:40]: And I would say that, okay, sometimes it’s maybe hard, but not impossible to change that. I mean, we’ve seen sometimes in our life that the company culture was super hard to change, but if you push really hard, you can also do some positive changes. Miłosz [1:17:54]: But if you can, you can do it also on the technical side of things. You can focus on the logic first and then implementation details or the other way around. The reviewers don’t need to have the full context in their minds at the same time. They can do it in parts. It’s much easier. So there’s some impact you can make. Robert [1:18:20]: We recommend checking what we’ve mentioned a bit earlier. So we have multiple strategies and hopefully it will help. And again, try to change maybe something in your team. Maybe it will be possible. Good luck! But yeah, I think also don’t be demotivated that it’s so bad. Because from our experience also in the teams, when you need to do these changes and you need to push hard to do that, actually you can learn a lot. Because it’s not like just, oh, let’s do it like this. And everybody’s happy. Yeah, it’s too easy. When you are in the team that you need to re-spend a lot of time to convince your product owner, the rest of the team for that. It will be hard, but if you succeed, it will be much easier in the other teams and it’s a useful skill. Miłosz [1:19:11]: If you want to succeed, it’s a change of the job. Robert [1:19:15]: Yep, it’s the last resort, but it often works. And if you succeed in this team, it will be also something nice to get a better job earlier. I mean, if I heard such stories like, yeah, I worked in the team when tasks were assigned to me and I convinced product owner to spend time together to split tasks. I convinced my team to split work into smaller. It was always like, okay, nice. It’s so you know what to do and it’s also helping to join nicer teams later. It’s also a nice opportunity. You should try to see it in this way. All right. Robert [1:20:02]: Other alternative that I have in mind that you can, Solve problem of big PRs is you know what. What do you mean? I see big PR, I close my eyes and yeah, legit to me, that’s marriage death. Miłosz [1:20:18]: That is good, yeah. Robert [1:20:19]: I know that many teams are people who use this tactic and it can also work in some places. I mean, let’s imagine that we’re in a team, in some component that is fighting for survival and the long-term in this case of company is six months. And if they will fail, they will just fail and it will be closed. So obviously, you probably don’t need to care that much about long-term maintainability because long-term may not exist, basically. But if you are in a more mature company or you are not trying to survive, and you know that maybe you are building these features and it’s very likely that it’s not the final, phase of this feature and you will need to change them. It’s pretty likely if you are working in startup because very rarely the idea that you have at the beginning survives multiple iterations. So basically you need to be able to very quickly. In those cases, probably YOLO mode is not the best load. Miłosz [1:21:33]: It’s a bit like we discussed in the first episode, when to write low-quality code. Sometimes it makes sense, but you need to consider if you are in this place or not. Robert [1:21:45]: I think it’s also connected to using some basic episodes and using some good practices. Basically, to skip some good practices, you need to understand what this good practice is giving you and what’s skipping this good practice who gets all the gear. Miłosz [1:22:03]: If it’s worth the short term to drop some… I mean, sorry, if it’s long in the long term, drop some good practice in the short term. Robert [1:22:14]: Yeah. Like code reviews. Yeah, yeah. Because, okay, maybe just one person can work on some functionality for one month, and, okay, just rush that, and because there is some tight deadline or whatever, okay, fine, but it’s not for free. You will need to pay it back if you will need to extend this functionality, because the rest of the team will have no idea how it’s implemented, and if only one person, implemented that, probably it’s also not the best application in the world and it’s very similar in the case when you have YOLO code reviews. So you’re closing your eyes and yeah, legit to me and so again, it’s not like you may notice that the name of this podcast is No Silver Bullet so it’s not like you should always do code review. Miłosz [1:23:02]: If you work on a own, you can’t. Unless you use AI I think that’s a good closing point consider what makes sense in this particular scenario also what you mentioned before try to be aware of the perfection trap don’t go too far into don’t take the metric too seriously and don’t say something like we don’t accept PRs above 500 lines ever it’s like don’t make it a hard limit or something like this Robert [1:23:46]: But i think it’s also important that if it’s happening too often it’s also good to have some pushback like sorry mate but yeah this is, and maybe you can spend some effort on splitting that because often when you have big pr it could be split and it’s not that hard really so especially. Miłosz [1:24:10]: If you have something like you know have a feature and then you are a good scout, so you can just refactor this a bit. Robert [1:24:18]: Rename one function that is used in 2000 places in the project. Miłosz [1:24:22]: So this is a perfect scenario where you can split it to a separate PR. You don’t mix the context. Robert [1:24:28]: It’s probably five minutes to make a separate PR. Miłosz [1:24:32]: It will help a lot for the reviewers because they don’t mix the context of refactor with actual changes. Robert [1:24:39]: I think it’s actually what you said. It’s very important. If you’re doing refactoring, don’t mix it with implementing new functionality, because it’s very, very hard to actually distinguish that. Because if you’re doing refactoring and keeping the functionality working as earlier, it’s much easier to read, because you’re just focusing on nothing. And if you’re changing functionality and doing refactoring, it’s much, much harder to spot actually what is part of refactoring and what is part of new functionality. Miłosz [1:25:08]: It probably helps if you learn a bit of Git, so you are comfortable branching out of a branch, which in this case is helpful. It’s not super complex, but I know many people are intimidated by Git. Stacking PRs is sometimes a great help. Instead of one huge PR with many mixed changes sometimes it’s easier to create three separate PRs branching from each other which is a bit of work to do huge amount, but for the reviewers it will be much easier to review. Robert [1:25:50]: But also try to avoid that, because if you’re branching out of the branch, it means that you probably have problem with some basics, like you’re waiting for too long for the review. Miłosz [1:26:03]: Yeah, but sometimes even if you wait like, you know, four hours for review, so it’s the same day, it makes sense to branch out so you can work on something next. Robert [1:26:11]: It’s not that big, yeah, yeah, yeah. Miłosz [1:26:13]: But it’s also… Probably not the default for everything, but I think it’s… If you’re not comfortable doing it, then you will be trapped. Because you will wait for a review and you will think, okay, I can’t work. Robert [1:26:28]: But there is also one trap. If you try to start to work on the next functionality that is branched out of that, that you’ll receive with you, it probably may make the reverse cycle slower because you’ll need to contact switch back to that. Sometimes it’s maybe better to do and make a coffee, two or three, four hours. But again, as we said earlier… But again, as we said earlier, there are multiple things that you need to join to get this good, fast review cycle, and it’s making a much better effect at the end, when you, just have multiple good practices that we mentioned, and it really flows. Miłosz [1:27:20]: Yeah, this is a great place to be in when the world just goes straight to the production. Robert [1:27:26]: And again, if somebody is telling you that, you know, implementing functionality and merging it within one day is just in the books on the conferences, it’s not, really. It’s possible to do. It requires effort. It requires high confidence. Team code review culture, but it’s possible, and it’s very worth, and I really hope that everybody who is listening for that will be able to work in such a team. We did that, and it was just super cool to feel this work flowing and see how fast we’re able to implement things. Miłosz [1:28:06]: Yeah, I think it’s one of the top experiences you can have in a team. Robert [1:28:10]: Now we are actually two in our teams, so it’s a bit also easier because we can just… Miłosz [1:28:17]: Still applies, though. Robert [1:28:18]: Yeah, yeah, yeah. So obviously it’s a bit easier to do when we’re two. Miłosz [1:28:24]: But it’s more tempting to build big PR, so we need to watch out. Robert [1:28:31]: Yep, we also do it from time to time. So don’t be perfectionists. I would say 80% so try to aim for 80% of PRs to be merged within the same day. It’s possible to do. But also, don’t probably 100%, it will be hard. But try to learn from those PRs that are not merged within this timebox. Alright, so I think we are out of the topics so it will be time for Q&A so if you have any questions please leave them on the chat and we’ll be more than happy to ask for them. Also, one small announcement. So, if you are longer with us, and if you are in our newsletter that we highly recommend, we mentioned that we’ll be dropping the… Live from our podcast. So in other words, we’ll pre-record episodes because, it’s a bit hard sometimes to organize everything. Miłosz [1:29:30]: As we can see today, we have our overheating camera. Robert [1:29:33]: Yeah, yeah, yeah, exactly. And as also you may notice one week ago, when we were not able to be live because of some sick leaves, etc. And also, it’s not super easy to actually record it live because we need to think really quickly. It’s probably too quick for us, and it’s not that easy to keep this slow. And for our podcasting platform, we are doing edits, and it’s easier to listen for that. But in the YouTube, the version that is staying there is the live version, basically. So there are some pauses, so it’s a bit… Miłosz [1:30:10]: But it was a fun experiment. We did 10 episodes. Robert [1:30:14]: Definitely. Miłosz [1:30:15]: I think we want to come back with live format, but probably something more interesting than this talking. Maybe some coding or something like that. Robert [1:30:26]: If we have time. Yeah. So we cannot promise, but we’ll also run some AMA sessions from time to time, so the next one in the next week. We should put soon to the description or maybe I will do it now. So I will put to the a link to the form for AMA questions. So I will put it to this episode’s description now. We’ll also cover some questions that we have in the comments on YouTube. Yes. I will just put it to the description now because I may forget that. Can I edit it? What else? Miłosz [1:31:25]: Yeah, that’s it. Thank you for joining the last live episode, for now at least. And thank you for the comments. Robert [1:31:34]: Yeah, we’ll see each other next week in the AMA episode. And later we are doing some short summer break because it was actually pretty intensive now or a bit longer. And yeah, we should be back after summer with some other updated podcast. So we’ll have some more surprises, but we’ll see after vacations. Miłosz [1:32:13]: Cool. Robert [1:32:14]: All right, cool. So I think we are done for today. So remember to leave some comments if you have under the video, leave thumbs up so the Deadly Algorithm will like us more. Subscribe to our podcast on Spotify, Apple Podcasts, and every application that you are using. Yeah. Miłosz [1:32:42]: Thank you, everyone. Robert [1:32:43]: Thank you. And see you next time in a week. Bye.

Quick takeaways Event-driven architecture (EDA) is powerful but tricky – it’s great for scaling and decoupling, but has many hidden traps. Observability is essential – debugging async systems without tracing, logs, and correlation IDs is almost impossible. Use the outbox pattern – it’s the safest way to publish events without losing data. Design events carefully – large, generic events can lead to tight coupling and painful refactors. Avoid over-engineering – sometimes synchronous systems or simple monoliths are just better. Start with sync if unsure – it’s easier to migrate from a well-structured synchronous system to async later than the other way around. Introduction In this episode of No Silver Bullet, we dive deep into the real-world challenges of working with event-driven architecture. We share hard-learned lessons from building distributed systems with events, covering pitfalls like dropped messages, debugging, eventual consistency, and designing events. We talk about the trade-offs, share practical advice, and help you decide when EDA is actually worth the complexity. Show Notes Go Event-Driven Training Go in One Evening Training Watermill Duplicator Middleware Blog posts Component tests Server-Sent Events Wild Workouts example project pq - poison queue CLI watermill-sql Outbox example Debezium RabbitMQ Kafka Quotes If you have a big event that 10 teams depend on, you can’t just say, okay guys, we will replace this event with another one or with 10 other ones. Miłosz If you feel like you need distributed transactions between three services, maybe you should just merge them into one. Robert The good news here is that message brokers are usually designed so that you don’t lose the message, because it’s quite critical for you if you use EDA. Miłosz I heard horror stories about sending 10 millions of SMSs to one person within 10 minutes. Robert The point here is that we don’t accept inconsistency, we just accept that it happens a bit later. And it can be also difficult to explain to your product peers or your boss or whoever. Miłosz Unfortunately, you cannot test every edge case that you can have in your application. I mean, you can try, but in reality, it’s never enough time, as long as you’re not building really critical services. Robert Timestamps 00:00:00 - Introduction 00:02:53 - Events and messages 00:05:30 - Debugging and observability 00:11:11 - Missing events 00:19:40 - Eventual consistency 00:27:28 - Outbox pattern 00:31:37 - Designing events 00:39:14 - At-least-once delivery 00:47:47 - Dead letter queues and alerting 00:59:39 - Distributed transactions and sagas 01:08:49 - When not to use EDA 01:18:48 - Q&A Transcript Miłosz [0:00]: Event-driven architecture sounds great. You get better scaling, loose coupling, resilient systems. Miłosz [0:06]: But it’s difficult to get right and can turn into over-engineering. Today we talk about the tough parts that make you wonder if going async was actually a good idea. I am Ior. Robert [0:18]: And I’m Robert, and this is No Silver Bullet Live Podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. And we learned that but following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow into principal engineer level. Miłosz [0:45]: If you have any questions or comments, you can leave them on the chat. We’ll pick them up during the discussion or at the end we will also have a Q&A Miłosz [0:55]: session where we can discuss all of it. Robert [0:57]: Exactly. And today’s episode is kind of a continuation of the previous episode, so if you didn’t have a chance to listen to that, it’s highly recommended, but it’s not required. So, in the previous episode, we discussed the differences between async and sync architecture. Today, we’ll focus on one implementation of architecture that is async, so event-driven architecture. So this is some form of event-driven architecture. And you will take a look on challenges of applying event-driven architectures to your applications. Because we had a chance to work with multiple event-driven applications. So we know when it works nicely, when it doesn’t. So it will be helpful for you to decide if event-driven architecture is for you. Miłosz [1:45]: In the previous episode, we assumed synchronous architecture is the default approach in many projects. Then we focused on some tips when moving to async is the idea. And today we focus on the challenges, because there are many with EDA. So maybe I’ll start with a very quick TLDR of what event-driven architecture is, which won’t be a complete definition, but just so we are on the same page. So, EventDriven architecture is based on events, obviously, and an event is some state of change in your system. And the idea is that your system or systems publish events after something happens, something changes in the system, and then other services or systems react to it. And can publish their own events, and this is how they communicate, in contrast to calling each other by some more direct ways. And this architecture is often built on top of messages, and that’s what we talk about today. Robert [3:00]: It’s probably also worth mentioning that you have a message, and message is some kind of way how you are transporting events. So basically, in most cases, event is payload inside of the event. So you can think if you are, for example, in writing Go, so message is something like htRequest and event is part of the body, let’s say. So it’s the analogy here. Miłosz [3:26]: So event is a kind of a message, very specific message that says that something already happened, some facts happened and then whoever listens is free to react as they see fit exactly, Robert [3:42]: Okay, but let’s go to the challenges because I think every nice technique has some challenges so yeah, this is the NoCyverBluet podcast and. Miłosz [3:56]: There are definitely many challenges when it comes to even driven architecture The first one can be debugging issues and the fear of losing events. And I actually have a quote that comes from Reddit comment we received from under the previous episode. And the commenter said, Stuff comes in, doesn’t come out. Things appear that you cannot explain where they came from. The amount of telemetry you have to add on your own to make those things sane is absolutely ridiculous in 2025. Robert [4:31]: Yeah, probably it sounds like a person that maybe has seen some people misusing this kind of architecture. Miłosz [4:37]: So you clearly see someone that struggled with this kind of approach before. But I would also say this critique is kind of fair. This can definitely happen. So it’s good to be ready for it and prepare. Robert [4:55]: And I think it’s a common theme if you were discussing multiple different techniques in previous episodes. It was a common theme that people were always complaining about some techniques. Also in places when this was misused. Because using some technique for some approaches may be just too sophisticated, let’s say. And if you are not solving a real problem with any technique, you will just have more problems at the end. Miłosz [5:27]: But even if you pick the right tool for the job, still can have this issue. Like, if you just call an HTTP endpoint, it will end up with some status code. If the other end doesn’t support the method, you will get 404. You will right away know what’s going on. Events, this is the hard part. If something goes wrong and you publish an event, and then the other end just doesn’t react to it. That’s the hard part, because you don’t know what happened. Robert [6:01]: It’s kind of similar to using CI and tests. I mean, if you’re using CI and tests, you may complain that you have a problem that you need to write tests, they’re flaky, and you maintain them. And yeah, it’s true. But maybe the question is, if you have right tests, maybe you should remove some of them in this case. But it’s all about cost versus return of investment, basically. Miłosz [6:25]: Yeah. The good news here is that message brokers are usually designed so that you don’t lose the message, because it’s quite critical for you if you use EDA. So I guess you could compare it to databases as well, but you don’t see people worried about SQL database losing the data, because it was designed to keep it, right? Robert [6:55]: And usually, if somebody is complaining that a message broker is losing messages, it’s probably not a problem with the message broker. Let’s say that it’s similar to many people complaining like, oh, I think it’s a kernel bug or a programming language bug. in 99% of cases it’s not. Sometimes it is, but I would not start with assuming that it’s the case. Miłosz [7:19]: This reminds me of this chapter in the Pragmatic Programmer about select just works. It’s exactly about this issue. So yeah, if you use the message broker the right way you shouldn’t have this issue. Of course it’s not that trivial sometimes to configure everything. But the good news is It is designed to help you not to lose any messages. But still, some weird stuff happens. I have quite recent example of this, where we have been running services on staging. And we got some events missing, just randomly. And it was super frustrating because we had proper telemetry and tracing and event log. And we just couldn’t figure out why this single event doesn’t arrive at the destination. And yeah, the intuition here is to blame the software, right? We were usingbiu RabitMQ at the time and we started wondering how can it be that it just randomly rejects some messages. Miłosz [8:34]: And of course, this wasn’t the reason. The reason was our frontend team were running our services locally. And they just used the staging RabitMQ address to run it. So unknowingly, they were consuming our messages and the handlers were doing changes in the local database instead of the staging database. Robert [8:57]: So let’s learn having access to non-development environment from local environment is always a great idea. Miłosz [9:05]: Yeah, that’s one lesson learned for sure. And the other is, RabbitMQ probably works. So if you see some very weird issue with some event arriving. Probably look into your code, or in this case, who uses your code. Still, it doesn’t need to be super complex to have this observability in place. If you have a tracing setup, probably the best, because you can see the entire flow of the system. But I would say even a simple correlation ID in logs can help a lot. You can just trace which services receive the event you published. Robert [9:53]: And I assume that if you are thinking about using event-driven architecture, you probably should have some kind of distributed architecture under a hood. And I assume that if you have, you should have already some observability for that in place. Some tracing or at least supporting correlation ID properly. If you are supporting it already for other transports like HTTP or gRPC, adding it for messages, it shouldn’t be hard. Miłosz [10:28]: It’s also often skipped. Observability is not a super interesting topic. Sometimes if you build a product, it’s often an afterthought. And then you already have trouble if you have no tooling in place and you start using events and then you need to debug something. That would be a terrible idea. So, definitely start with this. Robert [10:57]: There is actually a question. How you build on the chat? So just a reminder. So if you have any questions, please drop them on the chat. If not now, we’ll answer all of them at the end. But I think that this one is relevant. How you build your testing to catch dropped events like this? Robert [11:15]: Basically, in this case, I would say that in most cases, what is helpful for us is using component tests. So there are a bit like end-to-end tests, but just on the level of your service. And there pretty useful for that. So in other words, you are calling some endpoint, you are assuming that something is processed, and at the end, it should be some output going from the system that is showing that something was processed. The bad thing is that usually when you are losing some events, it’s some small chance that it will happen. So it’s because of some misconfiguration or some edge case or some lost connection and it’s a bit harder to do that. So in this case, what’s useful is using some library that already have tests for that and is ensuring that no events are missed. Of course, you still can have some misconfiguration, but in most cases, unfortunately, it’s like you might notice something on the production and you need to fix it afterwards. But, for example, in Watermill, so the library that we’re developing, we have a lot of tests that are checking managed cases like disconnecting or doing some strange negative acknowledgement to ensure that no message is lost. And so far it works because, We just have a lot of tests for that. Miłosz [12:42]: Regarding component tests, it’s important to use the same configuration you use in production. The exact same setup. Because it may be tempting to have something much simpler in tests, so it’s easier to just start. But this way you don’t test this entire topology, which can be sometimes quite difficult. Especially if you have something like a single event topic that routes events to some data lake or event log, something like this. It can be useful to do it for all services, but you should do it the same way in the tests, otherwise you’re not sure if you are testing the right thing. Robert [13:30]: But in general, if you are using, for example, some libraries like Watermill, it should abstract it from you. And it’s hard to lose this message, basically, as long as you will not do things like something ever happened, and you will think, okay, everything is all right, I’m acknowledging the messaging going forward. But I would say that it’s just about being sane where you acknowledge some message that you are processing. Miłosz [13:51]: Yeah, but testing is also not everything. You need this tooling to debug these production issues, because this issue of another team consuming your events by mistake, there’s no way you can do a test for it locally or in the CI. Robert [14:08]: I think it’s probably not only the case for an asynchronous architecture, because it can also happen for anything basically. If someone is interacting with your staging, there’s no way that you can test that, and this is the reality. Miłosz [14:21]: Yeah, but I think it’s easier to see the HTTP requests, right? Because you usually have logs on both sides. And in this case, you see the event is published, but you don’t see the other end at all. So that might be a bit more difficult to test. Robert [14:41]: That’s true, that’s true. But yeah, TLDR, unfortunately, you cannot test every edge case that you can have in your application. I mean, you can try, but in reality, it’s never had enough time, as long as you’re not building really critical services. So, in this case, you can spend insane amount of time on ensuring that. But, yeah, it’s… Miłosz [15:05]: Yeah, so to reiterate use observability from the start. Add some simple correlation ID in logs, tracing if you can. That will help a lot in debugging. And also don’t reinvent the wheel when it comes to using brokers. So for sure don’t implement your own queue from scratch. Probably not a good idea. Similar to not developing your own database. Use proven patterns, brokers and libraries that are tested and they work on production. Robert [15:47]: And if you were mentioning patterns, we’ll today also mention the outbox pattern. And I think that in this situation, this is probably the most important one because this is a super common issue that people are not using the outbox pattern. And why it’s probably one of the simplest ways to miss events. But we’ll go into that. Miłosz [16:06]: A good design of events can also help here. The more complex the workflow you have, the more difficult it will be to debug if anything wrong happens. So if you can keep your design simple and easy to grasp, it would be also much easier to debug any issues. So let’s say, if you have a single event that’s being consumed by tens of subscribers, instead of smaller events maybe that have fewer consumers, it might be more difficult to understand why some issues are happening. Robert [16:55]: And you also mentioned that some people are kind of afraid of this losing message. And I think it’s also important to, you know, if you are building an asynchronous architecture, it doesn’t need to be event-driven, it can be also message-driven. You should just trust it very much. I mean, you should trust that if you are publishing some event, this event is later consumed. Because I heard multiple times that people were kind of not trusting in their infrastructure that they have for that. They feel that, okay, I’m metting some event. I’m not really sure if it will be later constant or whatever. So I would say that this is a red flag. I mean, it should be something that it should be as stable as your database. So when you are doing SQL query, you’re assuming that this database basically works. And this should be also the case for your event driven or message event applications. Miłosz [17:50]: I wouldn’t like to work with a system that no one trusts the job done Robert [17:57]: So, the next thing that I would say is also a pretty big challenge is maybe changing the mindset of how you are building your system. Because if you are building your system in a synchronous way, as somebody mentioned on the chat, it’s also kind of more straightforward. So you are doing some operation, you are storing something in the database, some external service is called, and you know that it’s done. And when you’re building an event-driven application, for example, there is some trade. Robert [18:32]: There’s a use case that we discussed in the previous slide. In this use case, for example, we are reducing user and we are sending some data to some external system. But user registration is usually the critical part of the application. And if we are sending some data to external system, we cannot always trust that the system will be up, especially if you have out of traffic and we have some promotions. And let’s imagine that we have 1,000 users registering in one minute, for example, or even more, and it doesn’t work because some third-party system cannot handle that. It can happen, even if it’s not super big scale. But from the other side, it may be also important to send data to this external system. So, a nice solution, maybe using Event-Driven architecture here, because you can send this data asynchronously. But the downside is that the data that will go to the system may be not always consistent. So, it may happen sometimes that there is some bug, and instead of storing this data synchronously to the system, you’ll need to wait for a while before it will be stored there. Miłosz [19:39]: And at first it seems like a big issue because how can I live without transactions? What to do now. Robert [19:46]: But I would say that it’s important to think about the trade-offs here, because what would be the trade-off here? So instead of doing it within the request, if it’s delayed, it probably means that something is wrong. Robert [19:57]: And if something is wrong, so this request will probably just not work. So it’s much better to have this request working, so your customers will not experience any degradation of your service, everybody will be happy and data will be just delayed this is a trade-off and this is also something that we often, heard when we’re discussing it with some business people that if you ask them that okay does this data need to be consistent and i’m sure that in 90 of cases the person will say yeah yes data need to be always consistent because why it shouldn’t be consistent what will happen if it will be delayed yeah. Miłosz [20:35]: Yeah it’s like asking if it should be high quality Robert [20:38]: Of course Yes. Miłosz [20:39]: It needs to be. Robert [20:40]: Yeah, boss, can we make bucks? Miłosz [20:42]: What am I paying for? Robert [20:44]: And, yeah, obviously, like you said, with quality, we can always write highest quality code. But is this worth? I mean, writing this highest quality code, it will cost more time. Maybe we don’t have this time. Maybe it’s fine to cut some corners and they vary it faster. And this is the same with consistency. So sometimes it’s better to be delayed if you compare it with something not happening at all. Miłosz [21:11]: So this is the eventual consistency concept from EventDriven architecture. The point here is that we don’t accept inconsistency, we just accept that it happens a bit later. Robert [21:25]: Eventually. Miłosz [21:26]: Eventually, yeah. So it’s an important point and it can be also difficult to explain to your product peers or your boss or whoever. Because we normally this inconsistency we know is super small smaller than a second but sometimes it can be longer longer What’s surprising is, very often, it’s more acceptable than you would think. You could have inconsistent systems in reality, sometimes for hours even, and it’s completely fine. So this is more a product decision than purely a technical decision. Something you have to discuss with your stakeholders, basically. Robert [22:15]: Yeah and i think it’s again always good to show trade-offs so okay it can be delayed but what if for example the system is down what we prefer maybe it’s fine that it will be a bit delayed but it will not stop entire entire users journey you can also for example say how it’s work on bigger social media platform for example so if you are posting a post to x facebook instagram whatever so you see that some post was posted but probably it’s taking a couple seconds before it will be shown in different people’s wall and it’s just because of adding it everywhere synchronously will be just super slow because probably it needs to happen in multiple regions and doing it synchronously it will just sometimes fail and probably it’s super rare that you are adding some posts somewhere and you see the error maybe it’s happening under the hood so for example maybe in some region, it’s added after five minutes, maybe after some hours even, you’d never know. But you don’t see it in the UI, because it’s just eventually consistent. Miłosz [23:21]: It’s similar when changing your avatar or something like this. You can often see that many components on the website will be updated, others don’t. But also not in the technical sense, you can see it in real life, in how some systems work. Even bank transfers which seem like something that’s always consistent, right? Because you have the ledger. The traditional transfers are not immediate. So it’s a very similar concept. Robert [24:00]: Yeah, but it’s important to keep in mind. So if you are discussing it with some product owner, some product person, it’s important to show that, okay, this is the trade-off and the UX implication of that. Because if you are adding something to your system, it may be not visible immediately for users. So it’s important to keep it in mind. Sometimes it’s worth asking a question if it’s really needed. If it’s needed to have it eventually consistent. We already mentioned it in the previous episode, but we are big fans of server sent events. So this is really cool technology for handling this kind of updates. Also, I think a link in the episode materials, our blog post showing how to implement it in production-grade environments. Miłosz [24:52]: So it’s maybe another challenge here that you have to adjust your UI for this kind of errors. Something we also mentioned in the previous episode. Robert [25:03]: The good thing is that there is also some midway here, because let’s imagine that we are creating some system that is adding some data to your system, and maybe it needs to synchronize some data with external systems. But there is fortunately a mid-air ground that often people are missing, I think. I have an idea why, because to do it properly you need to know a couple of some techniques, but from the other side, it’s not rocket science, especially if you do it already once. So, because if you are doing some changes, you can still store some changes to your database locally in the transaction and also emit an event. It has one challenge compared to that, but it’s nice from the UX perspective, because you can write your data to your source of truth database, that is, for example, database of your service, and you can also emit an event. And it’s nice because if operations exceeded, you can show it to the user immediately and you don’t need to do any magic with servers and events. It will be just there. Robert [26:05]: But there is one trouble with this approach because, as I said, it assumes that we are storing something in the database and we are emitting an event. But what if something will happen in the middle? So, for example, we stored our data in the database, and our service died before sending the event. And before somebody saying that, oh no, the chance for that is super small, trust me, even for low-scale systems, it’s happening more often than you can think. It’s enough. And you may also say, oh, we have graceful shutdowns. Yes, as long as, for example, your service will not go out of memory, or you have some bug that will not make it shut down gracefully. Miłosz [26:49]: The earnest network dice, which is super common. Robert [26:53]: And you cannot do much about that. So it’s actually an interesting challenge to think. Okay, so in this case, we should store the data first to the database and emit our event to Message Broker, or vice versa. Maybe we should emit our event first and store data to the database. Miłosz [27:10]: Or call the publisher inside the transaction. Robert [27:14]: Or maybe in Thread and try to synchronize that. Miłosz [27:18]: Yeah, so basically there is no good Almost no good way out here. Robert [27:23]: In every situation, we’ll end up with inconsistency, basically. Miłosz [27:28]: Yeah, but there’s an outbox pattern that you mentioned before, which is a nice way out of this. Robert [27:35]: Yeah, and it’s also pretty cool in the simplicity. So basically, the idea of an outbox pattern is that the event that we would like to emit, you should store it in the same database as you are storing your data. For example, let’s imagine that your source of truth database is Postgres, So within one transaction, you are storing your event to Postgres, and you are also storing the data that you would like to send to your Postgres, and you are storing the event that you would like to event to this database within one transaction. And later, you are just streaming those events from this table with events to your method broker. And that’s it, basically. Miłosz [28:18]: And the streaming happens in the background. So it’s essentially the main transaction. Robert [28:22]: So it’s still eventually consistent, but well, like everything with event driven architecture, everything is eventually consistent. So it’s obviously a bit slower because there is one operation in the middle. So you need to stream it from your Postgres to your message broker. But well, trade-offs. Miłosz [28:43]: The difficult part is streaming from databases is not the easiest thing to do. And there are some patterns how to do it, but still, it’s easy to make a mistake and, for example, Watermill SQL supports it, but took us three major versions to do it right and avoid some bugs. So it’s also another thing you probably don’t want to write from scratch. Probably better to use something that’s proven. Robert [29:17]: If you are using Go, use Watermill for that, because Watermill supports Outbook’s implementation out of the box. Miłosz [29:23]: There’s also a tool called Debezium, we used some time ago. It’s also quite difficult to configure, so there’s no easy answers here. Robert [29:34]: You put it in a nice words. It was a nightmare. Miłosz [29:41]: So once again, it’s something you don’t want to reinvent, probably, unless you have a lot of time and can write lots of tests and test it on production properly. Robert [29:53]: And there are two alternative approaches here that I have in mind. The first one is that in some cases, actually, you can store data within transactions and later emit an event, but it’s only in a situation when it’s not a problem to lose some data in the event. For example, we are storing something in, let’s imagine, some telemetry or some continuous data. Miłosz [30:18]: Yeah, why you don’t care about losing the message? Robert [30:21]: Yeah, so in worst case, you may lose one data point, and it’s fine because a couple of seconds later, you’ll have another one. So this is fine. And maybe in this case, if you care about performance, it’s totally fine. The second interesting approach may be also if you’d like to avoid the streaming from PubSub to from your database to PubSub, you can just use your database as PubSub. So for example, in Watermill we have PubSub implementations for SQL, MySQL, Postgres, Firestore. So it can be also done, but it’s also a kind of specific implementation. I would say that by default you should use Outbox and stream it to your message broker. And I think we can probably link the source code of Watermill SQL in the episode notes. If you’re not writing in Go, you can take an inspiration. Ask AI, please rewrite this code in a way that it will not be copyrighted. Miłosz [31:29]: If you don’t get scared of these SQL queries. Robert [31:33]: Yeah, but don’t do it yourself. So there are a couple of queries there. Miłosz [31:38]: Okay, let’s move to the next challenge, which is designing events. Miłosz [31:44]: This is something that looks quite easy, or seems easy. Because it’s more soft than implementing some technical stuff. But it’s also quite difficult to get right. And especially because there’s no good universal answers here. For example, we often get asked if what’s better, should I have one big event or several or smaller events. Like the difference between user updated and having something like user password changed, user email changed and so on. And there is no good answer to this, because it has many trade-offs either way you go. So small events versus big events or generic versus specific and this kind of discussion there’s just no We’ve got one solution here. Robert [32:45]: But I think it’s probably one case when it’s a pretty bad idea. So having an event when you’re just putting an entire entity from the database there. We’ve seen it in one system. It was actually pretty interesting archaeology. Because when we joined the component, the event was already like 50 fields. It was crazy big, and it was super hard to mock, and it was super hard to change that. But actually, a couple years earlier, when I checked in Git history, it was a pretty small event. It would have probably four fields or something like that. But it was the same model as it was stored in the database. It was growing, growing, growing, growing, growing. Like a bowling frog, basically. Miłosz [33:25]: So you can imagine something like user updated with 50 fields in the event with all the details of the user. Robert [33:33]: And even if you feel that at the beginning it’s not a big issue, maybe. I mean, also you should watch out too, you know, not overcomplicate days from day one. Miłosz [33:41]: Exactly, that’s the hard part, right? Because, you know, say this is an anti-pattern, but let’s say someone has a small project they do, they write themselves. So then having even like this, maybe it’s good enough. Robert [33:56]: Yeah, but later, let’s imagine that the company is still there and you have like 10 teams, 50 people, and you have no idea who’s using that, or you would like to remove some part of these fields, almost impossible. And again, it just started with four fields and putting database model directly to event. So again, try to find some balance, but watch out for some extremes that it can… Miłosz [34:25]: So this is what I mentioned before regarding debugging and production. So we have this one huge event that everyone consumes. Maybe it’s more difficult to trace it and debug it, in contrast to smaller events. But this is just one heuristic. So there’s a question on the chat, what do you prefer? Granular or more composed? I’m not sure I can give one answer here. Robert [34:57]: Yeah, but I think probably we can risk assessment, but that usually we have more problems with the bigger one, less granular. I think it’s probably easier to join events later than split them, basically. Miłosz [35:18]: Yeah, because it’s easier to get tight coupling with bigger events, because many systems start depending on it, and then you can just… Yeah, you talk about refactor, and it’s not always that easy. Exactly because of this reason. If you have a big event that 10 teams depend on, You can’t just say, okay guys, we will replace this event with another one or with 10 other ones. Robert [35:51]: And it’s even worse if one of the teams that depends on that is data science team. If one of those teams is data science, it’s impossible. Sorry. Miłosz [36:02]: Yeah, refactor is easy if this is an internal event your service uses. That’s cool. But if it becomes a contract with other teams, yeah, that’s much more difficult. So, yeah, probably if you have more teams, it makes sense to start smaller, so at least you have this small contract. Maybe similar to API design of HTTP, probably you want to have a smaller method than one huge update method. It also depends on the API, but in terms of contract, it’s probably easier to track who is using each method rather than how people use this one golden method that does everything. Robert [36:52]: And I think a good heuristic to see if your events are too granular are cases when you need to do some kind of aggregation. So, basically, to do something, you need to listen to two events and do some magic to, OK, I finally received two events and I can do operations. So, it may be the sign that, OK, maybe it’s too granular. So that’s the challenge. Miłosz [37:19]: I would say, this lack of good universal answers here. Also, it’s not something very technical. You can just get better at it by reading a book. Most often you want to have discussions or event-storming sessions with other people in the team or in the company, with your stakeholders, to understand how to model it. So that also can be challenging. Robert [37:48]: Also, watch out for not discuss it for a couple weeks, because I think at some point, probably it would be good to record an episode about some kind of trust to engineering things. But it’s another extreme that doesn’t help to gain trust in engineering things when you’re spending weeks or discussing something. Such stuff. Miłosz [38:10]: That’s also a good point. So it’s easy to become a purist here as well and just decide we are not moving forward until we decide how exactly this event should be made it’s probably also an extreme you want to avoid Robert [38:29]: To summarize I think one of the biggest challenges here is decoupling so decoupling is downside here because everything is that decoupled with event-driven architecture that sometimes it’s hard to really change something downstream. Miłosz [38:46]: Or check how it’s used. Robert [38:48]: Yeah. Miłosz [38:49]: Especially if it’s… This is what you mentioned before about this, in contrast with HTTP, where you see the logs and, you know, who called it? If you just use events, it might be more difficult if you don’t have the tooling in place. Robert [39:06]: Agreed. Okay so the next very common challenge that you can have when you are building event systems especially when you didn’t work with them is at least once deliver and this is, connected again to how most of the message brokers are built but tldr is that, most of the message brokers are offering you at least once deliver semantics so it means that you can receive every message at least once. It’s not possible that you will not receive this message, but you can basically receive it multiple times. This is connected to the problem that we were discussing a bit earlier with Outbox. What will happen if your consumer will lose network connection in the middle of processing the message? They are processing the message, you lose the network connection, somebody cut the fiber cable between data Or DNS. Miłosz [40:13]: It’s always a DNS. Robert [40:16]: And what message broker should do? He doesn’t know if you reprocess that, so you stored the data and committed the transaction already, or maybe not yet. And the assumption here is, in most cases, that message broker is assuming that you didn’t process that, and it’s red-delivered. Miłosz [40:35]: Remember, message brokers are designed so you don’t lose the message, which is actually a good thing. Robert [40:39]: Yeah, the downside is that you need to keep it in mind. So basically, if messages can be redelivered, you need to handle the situation when it’s delivered. The good thing is that it’s rocket science. So for example, if you’re a creating user, you’re basically checking if the user that you are creating already exists. It’s good to have some identity that you can match and deduplicate by that. Or sometimes you can have some duplication ID, but you need to keep it in mind. Miłosz [41:10]: Maybe just reiterate why this is an issue. The event that create the user gets duplicated, let’s say it’s executed twice, right? So the first message will create the user, and the second one, or the second delivery of the same message, tries to insert the user into the database and it ends up with some kind of duplicate ID error on the SQL side. And this error will cause the message to go back to the queue, and it effectively blocks your queue. Robert [41:47]: And it’s probably the best situation, because you can notice this error and fix that just maybe by adding to your queries like on WK do nothing and it’s fine. Miłosz [41:58]: Yeah, sometimes it can have worse consequences. If, for example, this email will be delivered over and over again, and it’s usually within milliseconds. So if the handler does something like sends a message to the user via notification or text message or something like this, and for some reason it’s not stopped. You can do it thousands of times. Robert [42:26]: Yeah, I heard horror stories about sending 10 millions of SMSs to one person within 10 minutes. Miłosz [42:35]: Yeah. So that can be tricky. But more often you have an error. You can detect and fix. Robert [42:44]: I think there is also one case when it’s worse and it’s hard to detect. So, for example, when you have some counters or something. So, when you have a counter and a message is re-delivered, it’s a pretty subtle error. So, it’s hard to detect because, for example, as we had the question on the chat earlier, how to detect some things and when. When, for example, this counter is not implemented properly, it may be hard to detect that because probably, maybe at the end, someone will notice that I’m, for example, charged much more than I should have. Miłosz [43:20]: So this is a worse scenario where you don’t know that an issue occurred. You’ll discover it much later when it’s too late. Robert [43:29]: But for that, actually, we have a pretty good tactic on how to avoid that. So when you have systems who are prone for this kind of issues, you can use, for example, in Watermill, duplication middleware. So basically, this is the middleware that is delivering every message twice. So basically, every message that you receive in your local environment will be delivered twice. And it’s fine when your system is not super heavy and you are able to run tests with that, because you are just 100% sure that everything is processed properly, because this is basically how it should work. You should be able to receive every message twice. Your system should work exactly the same as this message will be delivered. Miłosz [44:10]: It’s a bit like chaos engineering, where you introduce some issues and purpose to the system. Robert [44:18]: The downside is that if your operations are heavier, you probably need to have more runners in your CI or more powerful laptops. Miłosz [44:27]: Or you could just do it in the tests. Robert [44:31]: Or just for some handlers, this is also the case. But it’s not worth doing it for everything. I mean, it depends on the system, of course, but doing it for everything, it may be probably an overkill. For example, if you’re working with some financial domain, like we worked, for example, we just have this middleware enabled and we’re just handling every message twice. If you’re crazy, you can even do it with your production. Miłosz [44:57]: Yeah, it should be safe to do it, actually. So the solution are idempotent handlers. So the handlers that you can safely execute twice or more times with the same input, and they won’t fail. In this case, in this example of the user being added, you just add onConflict, do nothing. Or you just somehow ignore the error. That’s a good application. And I know many people when they hear about it the first time are try very hard to figure out another solution. Because the idea of writing handlers like this is… I don’t know, annoying, maybe? They don’t like to do it. But the truth is, it’s not that complicated most of the time, right? Robert [45:51]: Yeah, so if you’re inserting stuff just like that, it’s a bit harder with counters, because sometimes you need to have some table with the application ID, so you’re storing some message ID or operation ID. they are deduplicating it by that. One scenario when it’s harder is when you are calling some external systems. So for example, when you are charging customers, but in most cases, when you are using some same systems, they should have some idempotency key or something like that. And basically, in this case, you are just sending this idempotency key and it’s deduplicated on the other side. I think it’s showing nicely like, okay, you’re calling some external system that it’s not really even driven, but often this kind of system offers some kind of application because it’s not really event-driven, but it’s a very similar situation. Miłosz [46:39]: Yeah, payment systems try to work like this most often. Robert [46:43]: Yeah, because this is the same situation as you have with message brokers. So you have HTTP connection, it was interrupted before the HTTP status code was sent to you, and the system on the other side doesn’t know what to do. Especially when it’s important in cases when you, for example, are charging people. Nobody would like to be charged twice. As long as you are not the person that is charging. Miłosz [47:08]: You can also see this idea in API design sometimes. If you have an HTTP handler that returns 200 or 201. So you can differentiate either by the response if the entity has been created or just everything is fine, nothing to do. It can also be helpful to handle this. Robert [47:33]: That’s true. Miłosz [47:38]: Okay. A bit related challenge here is messages that can’t be processed, so similar to this error. Because of the replicated message, you can have some message that has been published by mistake or can’t be unmartialed because of some issue. Basically it’s broken and whatever you do, it won’t be processed. And most often it will block your queue or this part of it. Robert [48:15]: And I think it’s showing nicely the trade-off that we are making here with asynchronous or even Jumann architecture. So in normal HTTP endpoints, it would be not an issue. So it will just return 500, and it’s not a problem. But you are losing the resilience. But in this case, well, resilience is important, but you need to do something about this message. Miłosz [48:37]: Yeah. And most often you want to move it out of the queue, so acknowledge it. But you probably also don’t want to lose it entirely, because you might have some important details or you don’t want to lose this information. So then you can move it to this separate queue, sometimes called the data queue or poison queue, or it just sits and waits for investigation. It sounds simple, but it’s also not trivial sometimes. I mean publishing to a separate queue is very easy, but you need to have some tooling to review those messages and move them back to the main queue if needed. And I think there’s no universal tooling here, depends on the PubSub. Some PubSUps have some admin interface, like RabbitMQ lets you pick messages. That’s one way to do it. But it can be also tricky. You can also provide your own tooling for it. We recently added a CLI like this for Watermill, for the SQL queue. So it can be useful to have some even simple tooling that lets you just review what’s there and has actions like delete or move to the main queue. Robert [50:04]: It’s also worth mentioning, because we’ve been mentioning Watermill a couple of times, but if you are with us for the first time, you might not know, it’s not our product, it’s an open source library that we are just sharing for free and there is no enterprise additions. Everything that we mentioned is just free in open source. There was one episode, I think two episodes ago, we’ve been sharing more details about Watermill. So worth checking definitely if you are thinking about creating your own open source library, because Watermill is pretty successful in the Go community. So to give you some number, it has more than 8000 stars on GitHub and a lot of projects using that. So again, if you’re thinking about creating your own over-the-source project that is useful for other people, definitely worth checking. Miłosz [50:55]: Yeah, and it can help you not reinvent the wheel. Like this CLI, for example. It’s much easier to use something that’s ready to use tend to figure out your own… I mean, maybe you can wipe code one, but I wouldn’t like to wipe code anything message-related. It’s kind of a risky area. If you lose a message, it might be permanent. Robert [51:23]: Fortunately, we have a lot of tests, so it may be a pretty good case for that. So maybe some agent that is running our tests in the background, but it will have a lot of logs to process, so it will be expensive. Miłosz [51:38]: Gabriel mentioned that it’s important to regularly check deadletter queue and analyze the reason of the issues. One more challenge here to have some alerting in place. You might have the deadletter queue in place, but if you don’t know the messages arrive there. Robert [51:59]: I would actually say that I don’t fully agree, because I think you shouldn’t be checking your metrics. This metrics should let you know, because it’s just hard to check hundreds of metrics at the end. Miłosz [52:13]: I mean, alerting, right? Robert [52:16]: No, I mean, you should have alerting, but it’s making you free from checking the metrics. Miłosz [52:26]: So, what you don’t agree with? Robert [52:29]: I would say that you shouldn’t check metrics. I mean, alerting should notify you about… Oh. Miłosz [52:35]: Yeah, yeah. Yeah, sure. Basically, you need a way to know that there’s something in the letter Q. Robert [52:42]: Yeah, so it’s more like you shouldn’t be looking at your metrics like, oh, everything looks… Right, because at some point you just may have too much metrics. Miłosz [52:52]: Ah, okay. I see now. You meant the regular check part. Yeah, yeah, okay. I see. Robert [52:59]: Because probably someday it may come that, okay, not look on those metrics and something may happen. So it’s better to have something that will notify. But I assume that, yeah, Gabriel had this in mind. But I think it’s important to distinguish because I’ve seen some components that had metrics, but didn’t have alerting. And it’s better than nothing, but still, it’s At some point, you just may miss something. Miłosz [53:25]: Unless you have someone looking at dashboards. Robert [53:28]: You’re paying somebody just to, you know, his head is fixed and he’s looking all the day on the metrics. Miłosz [53:34]: I know one company like that. Robert [53:37]: What’s this company? Tell me later. All right. But I think that there is one thing that can help us with this deadletter queue and poison queues. Because, for example, I’m personally not that big fan of poison queues, that letter queues, because you need to move these messages back, and it’s a bit problematic sometimes. And one solution here may be proper message ordering. So basically, if you’re doing proper segregation of the messages, So in other words, a bug for one entity will not block all other entities. You don’t need to put this message to Poison Queue because it’s isolated and it will just spin for maybe one person, for one customer, for some small part of the system. But if you’re ordering all the messages in one big queue, actually one spinning event can block the entire system. And this is a pretty bad situation. In this case, you should have some kind of poison queue. But the problem with ordering is that it’s also the science of having good bias. So you shouldn’t have one queue for everything, because throughput will be small. Robert [54:59]: Because one message will block everything. But from the other side, the good thing is that if you just have one big queue, you don’t need to care about proper order of events. So you just processing each event one by one and you don’t care about ordering. If you have everything unordered, it would be cool because it will not block anything else. But from other side, if you have events like addUser, removeUser and, this will come out of order, your read models that you have may be not right. Miłosz [55:30]: Or in some special design to make it work. Robert [55:33]: Yeah. Miłosz [55:33]: Like versioning or whatever. Robert [55:37]: Yeah. Again, we also were covering it more in the previous episode, but in my opinion, the most important here is to balance and trying to have as narrow ordering as you can. So not having super big queue, but also not have everything unordered. And it’s hard to give one hint how to do that, but probably you need to find the smallest unit that should be ordered and just have ordering with that. I think it’s also nicely compatible with the aggregate idea from Domain Driven Design. I think probably at some point we’ll also do an episode about that, but basically aggregate pattern from domain-driven design is all about that. How to find the smallest boundary of transactional boundary, but it’s kind of connected with that. Miłosz [56:33]: One challenge here is also to realize you need ordering. Because if you’re just starting out, you might not think about it at all. And you might just assume the order in which your publish messages is the same as you receive, which can sometimes be the case. Okay, the next topic here is transactions again, about distributed transactions. Robert [57:03]: Our favorite ones. Miłosz [57:05]: So, yeah, this thing is like, you know, many people when they start with distributed systems or even different architecture and they split the operations into a few services, maybe they initially use events, but then realize they actually need this to be in a transaction. And the solution is obvious. We need the transaction to span multiple services. So you just have to orchestrate those handlers to work together, and if one fails, because you have to roll back the rest of the pipeline. Robert [57:49]: Sounds like a great smart plan. What can go wrong? Miłosz [57:52]: Exactly. If you are a smart engineer, you should be eager to implement this right away. It’s a sign you are a good programmer. But most of the time you should probably reconsider because the complexity will be tragic in the long run. I don’t even know what to start with, but maybe the easiest case to consider is, let’s say you have three services that communicate through events and you create this kind of distributed transaction or saga on top of them. And then the third one fails, so you roll back everything that happened before but then the rollback fails and what do you do now? Robert [58:43]: I have PTSD already. Miłosz [58:44]: So you’re stuck with this inconsistency in the system that It can be fixed. You have to go manually to the database or whatever storage you have and fix it somehow. Robert [58:59]: And later implement compensating transactions for everything. You spend an enormous amount of time on it. Miłosz [59:07]: So this is something that can be useful in some very specific cases, but most of the time you probably don’t want to deal with it. Robert [59:15]: Very specific, probably it means super, super complicated system and many, many big teams. Miłosz [59:22]: Probably it would be easier to just merge your three services into one and run a database transaction on top of it, rather than this thing. Or just embrace eventual consistency, like we said before, and just accept that the transaction doesn’t need to be here. Yeah. Robert [59:42]: And again, it’s a different case when, for example, it’s some very complicated business project that is orchestrating between, for example, three teams where every team is 20 people. In this case, okay, it totally makes sense, but it’s a complicated process that probably each service in this case is very complex and you have some orchestration at a very high level. But if your team is 5 people, 10 people, and you are doing such a complicated transaction that is distributed, you can maybe ask yourself a question. Maybe it can be just one service. And I know that it’s not sexy to have less microservices, but at least it will save a lot of time on complexity. Robert [1:00:27]: And, for example, we have a pretty nice exercise about that in our training, Go Event Driven. So, basically, when we’re introducing sagas, we’re not starting with implementing sagas. We’re actually giving exercise with services with saga implementation, and task is to remove saga, just to show how much it can simplify everything when you are removing saga, and how much complexity is added there just because somebody decided to do it as a distributed transaction. So I think it’s important when you’re using this kind of tool like Saga that can create a lot of complexity to actually know how you can hurt yourself with that and after that implementing that. Not the opposite way because I know it’s very easy to start to use that but if you would like to implement it properly it takes a lot of time and it needs to pay back. Miłosz [1:01:25]: Yeah, so watch out for too complex architecture. Robert [1:01:33]: We’ve been already covering a bit the topic of testing, so obviously it’s a bit different than testing synchronous endpoints, so we already mentioned that you need to take in, consideration, eventual consistency, you need to keep in mind that messages might be delivered more than once. The good thing is that you already know that you can have, for example, duplicate middleware that is duplicating everything, but it’s also making it a bit harder to test that. So it’s probably more similar to problems that you have on frontend. So you are clicking something and you need to check in some intervals if something happened. In this case, it’s similar. So you’re calling some endpoint, you see some that, for example, accepted, and you need to poll for some data that is happening under the hood. Maybe sometimes you can listen for events, but it’s also listening for some events and waiting after it will happen. Obviously, it’s adding some complexity. Rather, it’s not rocket science. I mean, if you did it once, you can… Miłosz [1:02:39]: Yeah, experience helps. And also, if you create some helper, test framework or test library that makes it easy to publish and subscribe to events. It’s much more fun to work with. Robert [1:02:54]: From our experience, component tests are nicely compatible with this approach. We’ll link how you can do component testing. We have an article about that, Robert [1:03:04]: so we’ll link it in the materials of this episode. Because we know that a lot of people don’t know that something like component tests exists. DLDR, it’s nothing between end-to-end and integration tests, and it’s fitting this case pretty nicely. Miłosz [1:03:19]: And lets you test the broker in the same way you run it on production, which is important? Sometimes you also don’t have access to the same exact broker you run. That can be challenging. Like if you use cloud-based Pub-Sub, you just can’t run it locally, right? You can use emulators, which often are good enough, but for some edge cases, they might be just not supported. And that’s the hard part here. Robert [1:03:52]: But maybe you don’t need these cases from the other side. Miłosz [1:03:55]: Yeah, if not, then it’s fine. But in general, it’s one more moving part into the local and CI environment, so it can be a bit more tricky. On the other hand, message brokers are usually not super heavyweight, maybe except for Kafka, which is often fun to set up in Docker. Robert [1:04:20]: And you need a lot of memory for Heap or JVM. Miłosz [1:04:24]: Yeah, but if you use something more lightweight, Postgres maybe, or Gravity Split Lightweight too, Redis is super lightweight. So it doesn’t have to be painful. Robert [1:04:39]: Also, when you are creating this kind of tests, unfortunately, it’s easier to create a flaky test, because something is not happening immediately. And if you’re polling for something, there is more, bigger chance of flaky tests, basically. And in this case, it’s important to have good observability in place. So the thing that we were discussing at the beginning about tracing, logging is super important, and ensuring that correlation IDs are propagated properly. It’s also worth logging correlation ID in tests. So it’s easier to correlate the test failure to the logs that you have from Docker containers. Miłosz [1:05:24]: Oh, yeah. Debugging those tests can be fun as well. Robert [1:05:28]: But if you have logs, and there are good logs and correlation IDs are propagated properly, it’s much, much easier. LiveTip sometimes can also work to just get all those logs, put it to some AI and ask. Sometimes it can really help, basically, when you have a super big wall of text. Miłosz [1:05:47]: Yeah, we need to integrate it as an agent. Robert [1:05:51]: Enter Text… Miłosz [1:05:54]: You don’t see the logs on the output, but just a single summary. It will cost you much. Robert [1:06:00]: I run me off. Miłosz [1:06:04]: Coming back to observability, maybe one more thing is we talk about alerting a bit of the data queue, but also a super important thing to monitor and have alerts on is the number of unacknowledged messages or the queue length. Because sometimes you might just quietly keep spinning on one handler and you don’t see the impact anyhow in the system. And then later you look at the logs and, oh my god, there’s 200,000 messages waiting for to be processed. Robert [1:06:42]: And also Gabriel put the follow-up comment that he hates systems which are like a fridge. Miłosz [1:06:49]: I need this on a t-shirt. For context, this is about system that you need to open to check it. So it makes a lot of sense. But I like the end result. I hate systems which are like a fridge. I hate systems which are cold inside. Cool. Should we move on to the summary? Robert [1:07:27]: Yes, but maybe one more thing also. Miłosz [1:07:31]: There are a Robert [1:07:34]: Couple of challenges that we’ve covered there. Like every technique, like we mentioned, most of the techniques require some effort and extra cost to implement them. Yeah, and it’s not different for event-driven architecture. It requires some special techniques, and it’s important to use the right tool for the right problem. If you use event-driven architecture for systems that doesn’t require that, it will be just pure over-engineering. You’ll pay all the challenges that we just mentioned for nothing. You’ll just have one more problem in your Sometimes. Miłosz [1:08:21]: Your system is just synchronous by nature, and that’s fine. Robert [1:08:25]: Yeah, so your team is not super big, like for example mentioned with sagas. You don’t have super big scalability requirements. You are not integrated with an external system. You don’t have super big traffic spikes. In this case, maybe doing systems synchronously, it will be just enough. And if you’re in doubt, you can always start with Synchronous and migrate to a Synchronous one later. Like we mentioned in the previous episode, if you’re using some kind of abstraction over interactions to your system like clean architecture, it’s much easier because you have some layer that is allowing you to interact with your system and it doesn’t matter if it’s from HTTP request, when it’s a message. It’s making the migration much, much easier. And it’s also a common thing to ask your questions if you are doing some decisions. How you can reverse this decision? If it’s hard, ask yourself, okay, maybe there is some way how I can make this decision easier to be reversed. It’s worth, of course, and it can be helpful with tough decisions. Miłosz [1:09:36]: Yeah, but if you are not sure, you can just start with sync approach and later migrate. Robert [1:09:46]: Many traps that we mentioned today. The good thing is that it’s not that many traps for other sites. So I think it’s important to know about those traps and not reinventing the wheel. Because all the challenges that you may have with event-driven or asynchronous architectures, they are solved problems. And obviously you can try to solve them yourself. But I’m quite sure that you will not do it better than how programmers figured it out within the last 50 years. Miłosz [1:10:21]: So it’s the hardest if you just start out. You don’t know what you don’t know, so you might try some naive approaches or just doing it yourself. You read about event-driven architecture and then you decide, okay, I will use events and I will send them via HTTP or I will have a long-living an HTTP connection between services or something like this. So, sure, this could work, but maybe just use a message broker, which was designed for exactly this thing. And thousands of companies used it on production for 10 years. So it’s very likely it will work out for you, if you will. Robert [1:11:03]: And if you think you can do it better, I would recommend you to start learning what’s already there. And maybe you can build something better, but I think to do that, you need to actually know the state of art that is currently there. And I think it’s possible to still improve things, but again, I think it’s important to have a very strong base. But it’s only in case when you have some idea how to improve some stuff. When you’re normally working with some product, probably don’t have time for renovating the oil and improving stuff. What’s already there should be more than enough. Miłosz [1:11:41]: It’s the cool part. That’s the difficult thing to do. It’s exactly like with sagas. You can convince yourself that it’s a good idea to create a distributed transaction framework for your systems. But maybe what you need to do is improve your product a bit to create something people want to buy. Which might be not as interesting in a technical sense. It’s probably the right thing to do for your company. Robert [1:12:12]: But from another side, I think it’s also kind of the trap that you might think that doing some complex stuff like SAG or whatever may be interesting. Maybe at the beginning, but later you’ll need to maintain that. And I’m pretty sure, don’t ask me from where, from experience, that later you’ll need to maintain that. And it will be not interesting. Miłosz [1:12:32]: Yeah, you know what, you need to build a framework and then leave the company. Robert [1:12:37]: And it will be no longer a problem. Miłosz [1:12:39]: Problem solved. Robert [1:12:40]: Unfortunately, it’s often the case. But trust me, it’s much more fun to work with a system that is fairly simple, but you can iterate quickly and add features quickly there. And again, from our experience, using some boring, already well-known techniques and tools, gives a lot of fun at the end, because maintenance of that is much, much easier, and you can also do a lot of stuff. Miłosz [1:13:08]: Moving fast is fun. Robert [1:13:11]: For sure. Yeah, seeing how people are using your product and are happy about that, it’s also pretty cool. And I know that it’s not super technical from one side, but trust me, try this and it’s much. Miłosz [1:13:22]: Much high-stay. Robert [1:13:23]: And I think it’s also nice later when you’re looking for a job and you’re going to interview, and the interviewer will see that, okay, you’re a pragmatic person because you were not doing CV-driven development, but instead you were using the right tool for the right problem. So it’s, I think, often a really nice skill, and when we’re interviewing people, it was a big bonus point to see that. Okay, even if a person was just using very simple tech, but in proper context, and have big knowledge of the simple tools, it was always great to see it. Miłosz [1:13:59]: There’s no silver bullet. Robert [1:14:02]: Yeah. Sure. So, probably at the end we should say, is it worth using Evangelion architecture? And, well, unfortunately, it’s not that easy. So, as I said, there is no silver bullet, and there are many parameters for that. So, we believe that it’s most important to give you a lot of data points so you can decide on your own when using Evangelion architecture is good for you or not, to be prepared, the price that you need to pay, and to see if it’s fitting properly and if it will pay back, basically. Miłosz [1:14:42]: Yeah, there are probably no shortcuts here, no easy answers, since you consider what you work with. You should have no all those challenges before you start, so you are prepared for things like we mentioned about at least once delivery or eventual consistency. It’s just a bit different mindsets than working with Synchronous approach. So if you have the basic theory in mind, and you spend some time working with the projects in a Synchronous way, you will be ready for handling it. Robert [1:15:24]: I think the only silver bullet here is the fact that it’s not like it never works or it always works. I think it’s important to know what tools you have So not be a worker that only knows how to use Hammer, because this is not the best worker. So you should just know what tools are available and be prepared to use them in a proper context. Because using Synchronous Architecture Everywhere, it will not work. Using Asynchronous Architecture Everywhere, it will also not work. It’s just about finding the right tools for the right problem, because Opposite is not the best. Miłosz [1:16:07]: Yeah, so if you want to learn more about Event Driven architecture in general, or in Go specifically, we have our training of Event Driven on sale for one more week. So this is a place where you learn hands-on with exercises, and we do it for the first time in a cohort-based way, so everyone learns together. You can still join us for one more week and then we will start an intensive month of learning together. Robert [1:16:40]: Next one probably in half year or so. So, if you’re just listening to that and you’re listening for this now, we are super lucky. If you’re listening for this episode later, sorry. You can sign up to the waitlist. Miłosz [1:16:58]: Okay. Before we move to the Q&A, it’s probably time you hit that subscribe button and leave a comment or a review, if you listen to the Minna Podcast application. And as Robert mentioned, you can subscribe to our newsletter on our blog, and we will let you know about the next episodes. Actually, we don’t have a topic for the next episode this time. Robert [1:17:21]: It’s a secret. Miłosz [1:17:22]: It’s a surprise. Robert [1:17:24]: Yeah, but the one thing that we already know is that we’ll be changing the form of this live podcast a bit. So basically, we’ll do two more live episodes and later we’ll switch to sometimes doing live episodes. Most of the time, it will be pre-recorded because of a couple of technical challenges of doing it live. Like some workers that are super loud. Fortunately, our mics are pretty good and noise-canceling, but it’s super loud here. Miłosz [1:17:57]: It’s 15 minutes before the start. They started drilling, so it’s super fun. We also want to make the sessions a bit more interactive. If you do live sessions, we will try to figure out something more interesting for you to follow. Robert [1:18:09]: I think we’ll be running some Q&A sessions, definitely, between pre-recorded sessions. And we’ve also seen that most of you are listening to this after recording, so we’ll try to balance it a bit. And I think at the end also the content will be a bit better, because trust us, it’s not easiest to go over the scenario of the episode live, because we need to just keep flowing. But again, we’ll still have live sessions, so no worries, it will be just a slight change. But again, we’ll have two more live episodes, so don’t forget to join us. Miłosz [1:18:44]: And the short summer break and then we’re back. Robert [1:18:50]: Correct. Alright, so time for Q&A, I think. Miłosz [1:18:58]: I think we discussed most of the comments before because they were relevant to what we discussed at the moment. There’s a recent question from Pedro. It’s about starting a monolith and breaking down into smaller surfaces and then adding events. Robert [1:19:23]: AKA monolith first. Miłosz [1:19:25]: And this is something we did once, more than once. Once specifically. So we had this project when we started with a quite small team, but then there was a perspective that it will be much bigger in the future. didn’t happen. Robert [1:19:46]: At least we didn’t lose time for doing microservices. Miłosz [1:19:49]: We wanted to be ready to scale but we also didn’t want to create distributed transactions at the beginning. So we started with our monoleaf and quite simple setup where we have modules that are completely separated and connected with code, basically, with interfaces between each other. So you can think like, you know, imagine microservices that expose HTTP API. So it’s the same idea, but instead you have modules that expose interfaces in code. And that’s, as you can see, that’s pretty much the same thing, just different transport. In this case, there is no transport, because we don’t need it. But we try to do it in way that, if needed, we could take any module out and it would be quite simple to deploy separately. Robert [1:20:47]: But from the other side also, if your team is not that big, so like 10-15 peoples, maybe 20, it shouldn’t be that big problem, so you should be able to split it from anything. Okay, it depends on time frame basically. Miłosz [1:21:04]: But we like to think of Monolith versus Microservices question as deployment, not about project structure. Robert [1:21:15]: And also about some kind of social concept or concept of team. So basically, when you need microservices, you should probably need them because of how big is your team. So basically, you have one project and you have 50 people, for example, committing to one repository and deploying it. Miłosz [1:21:34]: Yeah, maybe they block each other, maybe the CA becomes complicated. Robert [1:21:39]: Somebody is introducing some flaky tests and 50 people are not able to deploy for an entire day. So yeah, this is a place where you need to split. But if your team is five, it’s not an issue. But anyway, if you’d like to see the implementation of the project that Miłosz mentioned, how we did it, we have blog posts about that. So we’ll include it in the episode materials tomorrow. And also, maybe one interesting thing. We also used to do the opposite one. We had microservices, and we did the microservices. We actually connected multiple microservices into one monolith, because it was just too granular, and it was too complicated to communicate between them. And having one service with everything, so similar situation to what we had with Saga, it would just make everything much easier. Miłosz [1:22:34]: And Gabriel mentioned that onion architecture can help when you use the DD from the one. So onion or clean or hexagonal or whatever, layered architecture in general can help. For bigger projects especially. Also not silver bullets. A separate episode about this. So in this setup I mentioned this is exactly this idea that following hexagonal terminology let’s say the modules communicate via adapters and that’s it it’s very similar to communication with any RPC just with non-network connection and Robert [1:23:18]: Also worth mentioning you don’t need to use DDD for on your architecture, you don’t need to always use DomainLayer. And it’s also totally fine. Because also often people join this and sometimes it’s creating some monster and people say, oh, architecture is overcomplicating. But often it’s because of that, because you don’t need to always introduce DomainLayer. Miłosz [1:23:42]: Yeah, and each module can have a separate structure. One can use CleanArch, one can use DD and some can be one file. Robert [1:23:51]: Or about that in the episode about clean architecture. Miłosz [1:23:56]: Yeah, so Gabriel mentioned you can move faster with M&Oleaf. If there’s a small team, Robert [1:24:02]: Then definitely. Miłosz [1:24:03]: If there’s a small team, a single developer especially, if you’re a single developer and you have microservices, something is wrong. Robert [1:24:12]: It’s not your fun project after hours. Miłosz [1:24:16]: Yeah. We love, but we also worked with teams where there were more microservices than people on the team. And it’s not fun at all. Robert [1:24:29]: Probably it was fun at the beginning, but later it was the opposite. Miłosz [1:24:35]: At the end, DDD is the best way to go, says Pedro. It depends, as always. Robert [1:24:43]: I’d say that probably depends which part of Domain-Driven Design, because many people are actually kind of not splitting Domain-Driven Design in two important parts. Domain-Driven Design are strategic and tactical patterns. Most people are talking about the tactical one that are in the code, But there is also a very important part of Domain Driven Design named strategic patterns. And actually, it’s totally not connected to the code. So it’s more about gathering requirements, talking with business, ensuring that the problem that you are solving is the right problem. And probably you should do it always, even if you are not naming it Domain Driven Design. Domain Driven Design is just giving you some tools for that. So if you are talking about strategic patterns, totally, you should do that, even if you are not naming it Domain Driven Design. What tactical patterns? Maybe. Sometimes. Miłosz [1:25:36]: It all depends on the service or the module. What does it do? There’s a service that lets you upload your avatar. You probably don’t need to involve your stakeholders to discuss what file formats are supported or something like that. Robert [1:25:53]: And have a separate service for that. We’ve seen it once. Miłosz [1:25:57]: So, yeah. Once again, balance is important. And if you have this modular monolith with separate modules that are independent, it helps, because you can have one module that is super technical and you don’t care about the design here, and another that’s pure business rules and you spend a lot of time on design. So, modularization is the way to go. Probably not in the form of microservices, but some kind of separation is great to have. Robert [1:26:39]: It’s also, in general, starting with microservices, there are cases, but probably there are cases when you are starting a pretty big development team, or for some reason you know that your development team will be big. And it’s not like yeah, yeah, totally, our, theme will be big and product will be super scalable every startup thinks this and probably five percent is really ending up there so and. Miłosz [1:27:07]: It’s also not that easy to design a system that would be Robert [1:27:11]: And be profitable 90 of them are spending time on setting up kubernetes and burning all the money and they are failing later five other five percent is failing because product is bad, and other 5% is lucky. And didn’t spend most of the time on setting up Kubernetes and Istio. Miłosz [1:27:33]: You mentioned this in the first episode as well. It’s very hard to design a system that will be ready for all the future challenges. It’s better to evolve over time. Because you just can’t foresee how things will change. Robert [1:27:48]: Easier to say than do, but again, check first episode. We have a couple nice ideas how to do there. Miłosz [1:27:56]: I think there are no more questions. Pedro says we are doing a great job. Thank you very much. Robert [1:28:03]: We are doing our best. Miłosz [1:28:04]: Thanks for joining us. Thank you for all the questions. We will see you in the next episode about a secret topic. Robert [1:28:14]: Exactly. In two weeks. So we’ll let you know in our newsletter. Just a reminder so go event driven you’ll be just able to join for another week next time or I’ll be healthier or so. Miłosz [1:28:28]: If you are listening later you can join the waitlist Robert [1:28:33]: Or go in an evening if you don’t go don’t know go yet because maybe a lot of you doesn’t know go and yeah we’ll. Miłosz [1:28:42]: Leave links in the description yep oh right Thank you everyone for joining us for the questions. See you in two weeks. Robert [1:28:52]: Thank you, Milosz. Thank you, everyone. Miłosz [1:28:54]: Bye.

Quick takeaways Start with synchronous architecture by default - it’s simpler to understand, debug, and maintain for most use cases Async architecture improves scalability and resilience - message queues and events help handle traffic spikes and failures Design matters more than the technology choice - tight coupling creates the same problems in both sync and async approaches Consider team experience - async architecture require more experienced teams and better tooling to handle new challenges Adjust as your system grows - external APIs, heavy operations, or the need to handle failures gracefully are good use cases Hybrid approach - use both sync and async where they fit best, rather than forcing one over the other Introduction In this episode, we discuss when to choose synchronous versus asynchronous architecture for backend systems. We talk about the trade-offs between simple, predictable sync communication and the complexity but resilience of async approaches using message queues and event-driven architecture. Instead of picking one approach over another, we focus on understanding when each makes sense and how to avoid common pitfalls like distributed monoliths and over-engineering. Show Notes Go Event-Driven training Watermill - our open-source Go library for working with message streams Event Storming - a design technique with a great unfinished ebook by Alberto Brandolini CQRS (Command Query Responsibility Segregation) - a pattern that works well with both sync and async approaches Killing the legacy and other CQRS stories - Miłosz’s talk on this topic Our CQRS article and Server-Sent Events post Message brokers mentioned: RabbitMQ, Kafka, NATS, Google Cloud Pub/Sub Event schemas: Protobuf, Avro, CloudEvents Event Sourcing - a pattern mentioned in context of recreating state from events Clean Architecture/Hexagonal Architecture - architectural patterns mentioned for making sync/async migration easier Quotes What’s the key difference here is the fact that when you’re publishing some message, you basically don’t care how it will be processed, by whom it will be processed. And it’s nice because it’s decoupling you from the consumers. Robert The bigger the system, the worse it becomes. For example, if many services depend on this one, which is a bottleneck, it can kill your entire platform. Miłosz Decoupling is not always good. It’s similar like with Don’t Repeat Yourself. Often, don’t repeat yourself may be a good idea, but not always. If you go into extreme, it may be problematic. Robert The worst case here is when you ignore the issue. You assume the database commit never fails. And it might be true most of the time, but usually if something can happen, it will happen on production sooner or later. Miłosz Event is information about the fact that happened. So it’s not something that will happen, may happen. No, it’s information about something that happened and it cannot fail. Robert It’s just counterintuitive, because you think that splitting things makes things decoupled. But it’s not really the case. It depends on how you communicate, really. Miłosz Timestamps 00:00:00 - Introduction 00:01:00 - Synchronous architecture 00:12:00 - Asynchronous architecture 00:21:35 - Error handling and resilience 00:28:03 - Challenges with async 00:37:53 - Common anti-patterns 00:47:29 - Distributed monoliths 00:57:24 - Message ordering 01:05:54 - How to choose 01:17:05 - Getting started and migration strategies 01:21:45 - Q&A session Transcript Miłosz [0:00]: Should we make this async? It’s one of the questions that seems simple, Miłosz [0:04]: but can make a huge difference in the long run. If you get it wrong, you’ll spend months fixing the mess. How to decide? You should not just guess. I am Iwos. Robert [0:13]: And I’m Robert. And this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. Learn that following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow you into principal engineer level. Miłosz [0:38]: And if you have any follow-up questions, you can leave them in the chat. We can pick them up during the discussion if they are relevant. Otherwise, we will have a Q&A session at the end. So we will go through all the questions. Robert [0:54]: And to be on the same page, today we’ll discuss async in context of architecture, not async in context of your code. So we’ll not discuss threads, we’ll not discuss goroutines, we’re not discussing code level, async, we’ll discuss the higher level architecture, so related to pubs, queues, and similar systems. Miłosz [1:21]: Yeah, maybe the context would be we have two services, microservices or programs in general, two or more, and we want them to talk to each other over the network in some way. That’s what we want to focus on today. And usually, it works like one of the services asks for something, makes a request, and then waits for the result. And you have to handle the error in some way, if it happens, or basically check the status code, which is the main difference between sync and async. Miłosz [2:08]: And yeah maybe one specific case is where you fire and forget so you make the request and then you don’t care about what happens later in the other service but I think we can agree not to talk about this case today because it’s pretty rare but you don’t care about what happened on the other end if you lose the status code, the response, you don’t know if we can continue, or should you let the user know what happened? Let’s skip this one for now. Robert [2:55]: If you are talking about synchronous architecture, as contrast, in most cases we are talking about our gRPC, HTTP architecture, so in other words, you’re calling from one place, you’re calling one service, and within the request context, some operation is done. So maybe some database, something is written to the database, maybe some transaction is done, maybe some money is moved, or something like that, so basically… Miłosz [3:27]: Yeah, transferred to another account. Something happens in the other service. Robert [3:35]: Yeah, and basically you know that if this request returned you the some 200-like HTTP code or gRPC status code that it succeeded, you know that it succeeded. And it’s nice because it’s the typical architecture that we are building. So if you are learning how to code, usually you are using synchronous architecture. You are not using event-driven architecture or as a broker, because it’s probably something more complex for somebody who’s learning. I guess, Milos, when you were learning how to code, you started with building some synchronous APIs, probably. Miłosz [4:16]: Yeah, I started with no APIs at the very beginning. Robert [4:20]: When you were learning programming, there were no APIs yet. Miłosz [4:24]: Well, I didn’t learn microservices first, but it’s for sure the first method of communication you can see while learning. It’s probably the most natural. That’s one of the upsides of using synchronous architecture, that it’s quite predictable in the flow. It’s kind of like the method call goes over the network. and when you read the code, you don’t really need to know what happens underneath. I think there are some frameworks that try to do it for you, even that you can distinguish between… Like the RPC calls basically look like normal function calls. That’s a separate topic. But the point is, that’s one good thing about SYNC. That it’s pretty easy to understand what’s going on in your function. Robert [5:25]: And it’s natural for you, so the entry point for that is much smaller. Miłosz [5:30]: If an error happens, or an exception is thrown, or whatever, you know it will exit the function, just like any other function call. That’s one of the pros. Robert [5:44]: If you were working as a software engineer for longer, As usual, you may prefer simple approaches, because simple approach doesn’t require any additional complexity. But, well, as usual, it doesn’t cover all the possible use cases. So, unfortunately, there are some cases when the simple approach of synchronous communication is not enough. So, for example, you are calling some external service, it can be other team, it can be other company, And for some reason this is working slowly or even doesn’t work. It’s not the best situation, but this is where we are often. So we need to call some other team services or some external services. Doing it synchronously sometimes may be hard. I remember when we were working with some, for example, invoicing software that was outsourced and it was sometimes working, sometimes doesn’t work, but we were not able to change that because it was some contract and we were being obligated to use that. Miłosz [6:47]: It might be even an external API completely out of your company. There’s some third-party service you use, and you need to use it, but you have to deal with the API, whatever it takes. Robert [7:02]: Yeah, and sometimes, for example, also some real-world examples, sometimes you may be integrated with some external marketing systems. And for some unknown reason that I don’t know why, but often those systems are not that stable. And sometimes it’s part of customer registration flow, but from some other side, you need to send some data when you’re registering customer. And, well, you cannot risk that basically it will break customer registration Robert [7:30]: flow because it’s part of the critical flow. So it’s also pretty problematic and in synchronous way, it will be hard to handle. Miłosz [7:41]: Yeah, it’s one of the limits. I think it’s often a question people ask us about designing the architecture, so They recognize that they have a database transaction that does something like saves the user, let’s say, in the database. And does a few other things. And they also need to call this, let’s say, marketing software via API. And they are confused how to model it, because the API call, should it be inside the transaction, should it be outside the transaction, how to do it. And basically, there’s no good way to handle this in the synchronous approach way. Robert [8:24]: Yeah, you can choose basically what’s less important. For example, okay, we’ve failed to send some marketing information, integration to some marketing system. Well, or maybe we should, it’s something critical, and maybe we should interrupt our registration function flow. It depends, unfortunately. But, well, when you are doing it in a synchronous way, you need to do those trade-offs. Miłosz [8:48]: The worst case here is when you ignore the issue. Like, you assume, okay, but database commit never fails. Robert [8:56]: Or the system will always work. Miłosz [8:58]: Yeah, and it might be true most of the time, but usually if something can happen, it will happen on production sooner or later. And what I really don’t like about ignoring this challenge is that you end up with inconsistency in the system. And fixing this is super hard. First of all, you need to figure out if you have inconsistency. And if you have no proper monitoring in place, you might never know. You can see it weeks later or months later. Robert [9:37]: Or even worse, let’s imagine that something on the other side, we assume that it’s stable, it’s always working, this external system of other company, and you have some big promotion, big marketing, and you have hundreds of people trying to register, and the system is down. And I guess that a lot of people may be pretty upset because somebody spent some horrible amount of money on marketing, preparing that. And because of this assumption, it’s not working. And often, you know, it’s important integration that if you disable it, you’ll find out that, okay, it’s hard to track what’s happening with customers and it’s a super big problem for everybody. Miłosz [10:17]: Yeah, so it can easily become a bottleneck. Robert [10:20]: Yeah, and if you are talking about bottlenecks, the other problem that often you have with synchronous architecture is problem with scalability. So often you have some operations that are heavy and they are working fine when you don’t have super big traffic, but hopefully your product that you’re working on at some point may have some bigger, spikes of traffic, and you may have more customers. And, well, if it’s not designed in a proper way, and it’s just done everything asynchronously, at some point, people will just start to see errors, because the operations that they are trying to do cannot be done, because you just don’t have enough resources. And if you’re doing it synchronously, you cannot do much about that, if it’s a critical part of the functionality. Miłosz [11:08]: And the bigger the system, the worse it becomes. For example, if many services depend on this one, which is a bottleneck, it can kill your entire platform. Robert [11:24]: I’ve seen that many systems are trying to do some workarounds for that, like runjobs or or even worse, database triggers. It’s some kind of workaround for that, but from our experience, it doesn’t scale at the end. Not only in terms of resources, but also in terms of maintenance. It’s a mess at some point of complexity, basically. Miłosz [11:56]: So, the other approach is asynchronous architecture, and we don’t talk about the code level. So, stuff like callbacks, or promises, or director model, whatever you use in your code, it’s out of scope for now. Coming back to the context of two services that talk to each other, I would say that the goal of a synchronous architecture in this context is that we want to execute this in the background, this request, whatever it is, but also we don’t want to lose the information about the result. And even if the result is empty, maybe there is no payload. But the status message or the fact if an error happened or not is an important fact. So we want to handle it sometime in the future, but we don’t want to wait for it to complete in the current function. Robert [13:00]: And again, some poor man’s approach may be putting it to the table and doing some batch jobs, but while it can work, it’s not a perfect approach, because Because at some point, if you have more jobs to process, you may end up with just taking more time than the entire day have. So paralyzing of that may be a challenge. And it’s often also delaying when it’s processed, basically. Yeah. Miłosz [13:33]: So Gabriel mentioned in the chat that it’s easier to debug. I think this was about a synchronous approach. Yeah, that’s a super important point. If you have a classical function that goes one after the other of the steps, it’s quite easy to understand what’s wrong and debug. With icing it’s a bit more complicated when you get into that. So, what solutions do we have in terms of the architecture? I think maybe what many people may be familiar with is task queues, which is usually a simple FIFO first in first out queue of tasks that some worker processes pick up from. And it’s a pretty easy solution and can work well for simple use cases when you just want something to be done in the background and forget about it. The downside might be that if you have a more complex setup, it’s probably not enough to handle it well, but it can work. Robert [14:50]: The two cases that I would have in mind when queues may be not helpful, compared to some event-driven architecture, is a case when you have more teams, so it doesn’t help you with organizational scalability. So basically, you have one queue, and it’s kind of similar to sharing database. So if multiple teams are sending messages to queue, and one message is spinning, it’s blocking other teams. So it’s not best experience. And the second thing is that if it’s order queue, even if you have multiple queues and you have some high priority or low priority queues, it’s also pretty limited in terms of performance. Miłosz [15:36]: Yeah, but for a simpler architecture, it might be a pretty good solution. For sure, better than this, hoping it won’t break in a synchronous way. But a step forward in this are messages in general, and often using a message broker in the middle, like or a PubSub. So this is a similar approach, but instead of having workers, pulling tasks and doing one thing. We connect the services using messages. And there’s usually many ways you can build this topology, like who receives the message, and it depends also on the infrastructure you use. You might be familiar with message brokers like RobidMQ or Kafka, or the cloud-based ones, and they all have a bit different configuration. But the point is, it allows you to communicate between processes using those messages. Robert [16:45]: Yeah, and what’s the key difference here is the fact that when you’re publishing some message, you basically don’t care how it will be processed, by whom it will be processed. Basically, emit some thing, usually event, and this event can be processed, maybe not processed, it may be spinning for a while, you don’t care. And it’s nice because it’s decoupling you from the consumers. Robert [17:13]: And it’s adding the thing that I mentioned in terms of queues that it’s not handled by queues. So it’s creating some kind of way that helps with scaling multiple teams. So, if you are decoupled from customers of your teams, basically tens of, hundreds of teams can listen to your events and you don’t care if it will fail or not. Miłosz [17:40]: So, this is if you use events, right? So, an event is kind of a message and it states a fact that happened already. This is a big, I think, more like a mindset shift rather than a technical thing. Because you can easily use messages, but if you send, for example, requests like in RPC as other messages, doesn’t give you the same decoupling benefits. Robert [18:16]: Yeah, I think it’s nicely how it works out of the box with events. I mean, how it fits this model nicely. Events is information about the fact that happened. So it’s not something that will happen, maybe happen. No, it’s information about something happened and it cannot fail because it already happened. So you can’t say, oh no no it didn’t happen it happened and it’s nice because it’s in some way immutable so everybody can listen to that they cannot say it didn’t happen yeah and. Miłosz [18:48]: They can’t not agree yeah it i think it simplifies the design often right because let’s say you have this process after i don’t know someone placing an order and instead of this one process figuring out tell other things that need to happen in the entire system, like send a receipt, send an email, notify the sales team, whatever. Miłosz [19:16]: Can we manage your guess like that? You can just send a single order place event from this system, and then other systems react to this. Robert [19:27]: We’ll go to anti-patterns a bit later, but I think one of very often seen anti-patterns is passive aggressive events on the pattern. So usually when you are creating an event, it should be verb in past tense. So, order placed, but it shouldn’t be event like order should be placed. Because in this case, it’s more like a command, and commands are more fitting the queue system. So you’re sending this queue, and one consumer is subscribing to that, and maybe order will be placed. But maybe it will be not placed, because you don’t have enough items in an inventory, for example. Miłosz [20:11]: It’s a funny thing, because it’s mostly about naming things. There’s nothing much different about an event than a regular message. The difference is just the payload and how you call it. But if you pick the right names, the design becomes much easier to follow. Robert [20:29]: I think it’s interesting because many people are looking at that like, ah, naming doesn’t matter, whatever, name it XYZ and it will be fine. But I will not agree after seeing how it can help, or it can not help if it’s not properly designed. Miłosz [20:46]: Yeah, it’s about mental load, pretty much. How easy it is to understand the design. And then it’s also easier to debug issues or extend it. Robert [20:57]: And I remember that some people asked about that in Watermill, so the library that we’re maintaining in Go. And what’s the difference between command and event? because basically API is very similar, but there are some really small reasons. And yeah, basically you can send command as event, but again, it’s just practical to follow this because it will be easier later. Miłosz [21:25]: Maybe let’s come back a bit to messages in general, not just events. And I think one thing that’s pretty different in how you approach asynchronous way is error handling. Robert [21:40]: Yeah, and I think it’s a cool thing that you get out of the box, we can say, with event-driven or many asynchronous approaches that is basic or un-message broker. Because if you are doing operations in the background, for example, instead of emitting an event or sending a message, when you are just doing it within a process, you never know if this process will not die. Because, for example, Kubernetes will say, you should be that. You never know. It can happen. And I know that in some smaller systems that are not mission critical, it doesn’t matter. You lose some operation, maybe somebody from operations team will be upset, but that’s it. But if you’re building some systems when this consistency really matters, and I would say that in most cases it is, if you are not storing things that you would like to process in Message Broker, in some cases it will just happen that your process will die and some operation will be not processed. And when you’re sending this event or message to your message broker, it’s up to the message broker to ensure that it’s processed. And only if you will confirm that it was processed, the message will be deleted. Robert [23:07]: And it’s especially useful when you have some temporal outage. So imagine that your some external API was down for a couple minutes, your database was down for a couple minutes. In this case, when you are subscribing to the messages in MessageBroker, it’s not an issue. So messages will accumulate over the time, and it will just auto-resolve after the time. And I think it’s pretty cool when we work with some systems, when we’ve seen that, okay, some outages of external systems was happening, but it was not a big deal. So we sometimes receive non-critical alert like, oh, we have some messages that are stuck, but it was just stuck for a couple of minutes and later you can see how the chart is going down. Always pretty possible to see that. Miłosz [23:52]: It can also be a bug in your code. You can mess something up and then the direct requests fail. So in synchronous approach all services calling your service this broken service don’t have much choice what to do like best they can do is probably return the error up the stack to the previous caller and then up to the user show on the UI right but if you use messages for this then yeah this this thing won’t be done in the moment, but then you can deploy a fix for the code issue, and as I said, all the messages can be processed again. Auto-healing is pretty cool to have. I think it’s also another factor that simplifies the mental model, because you don’t need to care about error handling in each scenario, how to do it, how to make this nice for the user to report or… Get over with. Instead, you know that you can just retry, and most of the time it will work fine. Robert [25:14]: And another similar situation is when you are receiving bigger traffic because of many reasons. So many people went to your website or you have some attacks sometimes. It’s also kind of similar situation because normally you would not be able to process all those requests, but if you just receive a lot of messages, you can accumulate them and process them in order that you can have. And it’s also helping you to scale it with that, because if you have some way of autoscaling your application, you can basically configure it in multiple ways. But when this autoscaler will detect that, I know you are using a lot of CPUs or there are more messages to be processed, you can just autoscale more, process those messages and downscale your application back. And it’s kind of almost out of the box, we can say. Miłosz [26:08]: Yeah, not always ideal. It depends if you need this thing done right away or not. But if you can wait a while, it’s much simpler than scaling up to handle all incoming requests. Robert [26:22]: Yeah, and again, I think many people are mentioning event-driven or in-general asynchronous architectures in terms of technical and performance scalability, but we have also the perspective of organizational scalability. So you can basically integrate more teams over events, and thanks to the decoupling, you can add more teams and they can just work more independently. Miłosz [26:53]: If you design them well that’s a prerequisite obviously because there are also lots of downsides of AC Cloud architecture maybe not downsides but challenges you need to be aware of right We mentioned a couple of times already that the design is important, so that’s one thing. It’s more complex in general and you need to understand the rules well. Ideally, the entire team would be familiar with what the stuff works. So the more experienced team, the better for this kind of architecture. Robert [27:35]: Yeah, but as usual, so you have more complex problems to solve, it’s better to have more experienced team to handle that. Miłosz [27:44]: Yeah, because for some platforms you can just use only a synchronous approach and be done with it. So at some point you just need to move to at least parts of async. And there’s lots of new issues you need to deal with, like eventual consistency or phrase conditions or debugging, for example. Local development can be a bit more complex. You need new infrastructure, the message broker, which also is something that you need to be familiar with and you need to maintain. Robert [28:25]: Yeah, also testing is a bit more challenging because it’s no longer working like you’re doing HTTP requests and after request is done, you know that operation is done. No longer. You need to do some way of polling. So if you work with frontend a bit, it will be familiar for you because like in frontend, things are not that synchronous and sometimes you need to just poll for some operation that you expect that will happen and it will, eventually happen, but maybe it will not. Miłosz [28:59]: Yes, but speaking of front-end, the UI also needs to be adjusted sometimes. Robert [29:03]: Yeah, I think even Gabriel mentioned that it can be more difficult to handle errors in UI in async system. So you need to either use WebSocket or ServerSendEvents to push async errors, like in case of failed sync operation. Yeah, we totally agree that this is the challenging part. Miłosz [29:25]: Exactly, yeah. This is it. So with synchronous approach, you can just return the error of the stack until the UI and then just show a big 500 error and something like try again or contact network support. With async approaches, the error is there somewhere in the system and the user doesn’t know if something happened or not, if it was successful or not. Let’s say this extra setup to show the status as recurrence. Robert [30:04]: In other words, trade-offs. It really depends on basically what the system is doing under the hood. Because again, if you’re storing stuff to the database and it’s your database and it’s stable, So it probably wouldn’t be a problem. But if you’re doing some stuff in the background, it’s starting to be tricky. And it’s also easy to create a poor UX design that will be just misleading for Robert [30:34]: people that are using that. Miłosz [30:36]: Which is also counterintuitive because it seems like something specific to backend. And here you need to consider it in the entire flow, even in the UI. Robert [30:47]: Yeah, and again, the thing that Gabriel mentioned, we are also big advocates of servers and events. If you are interested how to implement it properly, go. So we have a pretty nice article about that, that Miros wrote on our blog with super nice example. So we’ll link it in the podcast materials if you are interested. Miłosz [31:13]: Any other downsides? what about observability? Robert [31:18]: Yeah, it’s also a bit tough topic because when you are building your REST API, you can basically monitor your HTTP load balancer. Miłosz [31:31]: You can open the console in the browser and most of the time you will see some error when it happened. Robert [31:39]: Yeah, or just check your load balancer and you have 500s on your load balancer. You’ll see that something is wrong, we need to do about that. With an asynchronous approach, it’s not that easy because probably you should monitor how long is the queue, how long, what is the age of oldest message. It’s also not simple, because it depends on queue or topic or subscription. Sometimes having event dot is there or queue for five minutes is fine. But sometimes, if it’s delay 30 seconds, it’s maybe a problem. Sometimes having 10 messages spinning is fine, sometimes it may be a critical issue. So I’m not saying that it’s not possible to implement, but it requires more thinking and it requires more of what you said at the beginning, that you need to spend a lot of time on designing and understanding. Okay, maybe not a lot of time, but basically you need to have it in the back of your head. Miłosz [32:49]: Have some tooling in place, like tracing at least, to see what’s going on, to understand where the issue is. Robert [32:57]: Yeah, and some metrics that I know that not every team has, but in this kind of architecture, you should have that. And I know that also not all message brokers are exposing the metrics about, for example, oldest message or how many of the messages are spinning. So sometimes, unfortunately, you need to write it by hand and export it by hand. So, obviously, it’s some cost. For the other side, it’s also not an unsolved problem. I mean, it’s a solved problem if you find a well-known solution for that. It’s not rocket science at the end of the day. But it’s just important to keep it in mind, and keep in mind that every operation is also different. Robert [33:45]: And there’s also one challenge that I think not a lot of people are mentioning, because it may be a bit counter-intuitive, but it’s a problem of decoupling. And why it’s interesting? Because usually we’re talking about decoupling in context of good things, but also decoupling may be challenging when it’s, let’s say, extreme decoupling. Like with events. So let’s imagine that you’re emitting some event, like order was placed, And you may have no idea, actually, who is listening to this event. So when you have your API, you basically see that somebody is calling this API. You probably can have some tracing and by that correlate from where it comes. But if somebody is listening to your event and the service doesn’t have good observability, you have no way to understand who is doing that. In worst case, even you have no idea if anybody is doing that. And it’s starting to be problematic when you need to change something. We’ll get into that, but it may be challenging. Robert [34:46]: Especially if you have a bigger system with more teams, it may take months to deprecate some events. Miłosz [34:53]: Yeah, and you want to get rid of some code that produces some old events, but you don’t know if a team is using it. But it can be in larger organizations that can be challenging. Robert [35:05]: So yeah, in other words, decoupling is not always good. But I think it’s also similar like with Don’t Repeat Yourself. Often, don’t repeat yourself may be a good idea, but not always. And it’s a similar situation, because don’t repeat yourself. It’s trying to kill the coupling, but also, if you go into extreme, it may be problematic. It’s, I think, interesting how you can apply some patterns and anti-patterns from level of code to level of architecture in some way. Miłosz [35:41]: Yeah, because it may promote teams that are very focused on what they do. They just publish some events and then they don’t care about whoever is doing with those events. Yeah, it’s harder to see than with API calls. I think that there are some tools for this. I’m not sure if we used any of them, but probably also depends on the on the approach you take, because there’s no one standard for how to define events. I mean, there are some, but probably depends on the company and message broker and technology use. Robert [36:24]: Yeah, but I think you reminded me about this case and it’s pretty interesting that I remember in that in one place we had an event that we’re integrating, we have integration between two teams, but indeed, we have some data quality issues there. And what we ended at the end was getting rid of this event and instead having a synchronous endpoint that the other team needed to call. And we were able to basically do validation. Miłosz [36:52]: Oh, that’s interesting. Robert [36:53]: Yeah. And in this case, I would say that this extreme decoupling was bad because, again, there was problem of responsibility because somebody could emit an event that was totally not valid, and it was hard to enforce, because if this event was emitted, we’ve been notifying it after a while in our systems that we are starting to receive some events. It was like, oh, we don’t care. Miłosz [37:23]: We did our job, we published the event. Yeah, that’s an interesting thing. It’s like on the boundaries of a team, when you integrate two teams, two services that belong to different teams. And the API becomes super important. And yeah, maybe sometimes synchronous is better in this case. Yep, that’s a good take. Miłosz [37:50]: Let’s move to common anti-patterns. Robert [37:54]: So you say that it’s not perfect and it has some pitfalls and something can go wrong with asynchronous architectures. Miłosz [38:03]: Looks like there’s no super bullet. So I would start with two naive takes on background processing. So I’ve heard of systems, or seen some, where people use something like a cron job to process the background tasks. It’s similar to task queue, but more like in-house. And again, similar to task queue, it can probably work well for some cases, except you reinvent the wheel in this case so it’s a bit worse I would say and the main issue is if you have any more complex scenario then simple please do this later then it breaks it won’t help you much you don’t get all those guarantees of pub subs and so on and Robert [39:05]: It’s also often processed at some magic hour so it’s creating delay basically, if you are doing bad processing with your current job. Miłosz [39:15]: Yeah. Robert [39:17]: But I would even say that it’s still better than database triggers because, I mean, it’s choosing between two evils, but at least I would say that it’s probably a bit easier to obtain. Miłosz [39:32]: Yeah, logic in the database is a controversial topic. I wouldn’t want to work with a system that does it. And of course there is probably this one specific case where it makes sense. I’m sure there is, but in general it’s tough to update and manage this. Robert [39:52]: And test and debug. Miłosz [39:55]: Yeah. If I had to use a cron job like this, I would just do it in the code. And for example, in Go it’s trivial to spawn a new Go routine. So that’s a perfect scenario. Have anything in the loop, pull the sleep and do something every, I don’t know, minute or something. Robert [40:14]: It can be some good starting points that you can migrate a bit later to a more proper solution. Miłosz [40:21]: My main issue with this is that you reinvent something that has been solved. And many people put years of work into figuring out edge cases and everything. So maybe it’s better to do something that exists. Robert [40:38]: Especially that, for example, with Watermill in Go, you can pretty easily use Postgres or MySQL as PubSub. So basically, you don’t need to add an extra architecture there. And you can start with something simple, and without many changes, change it to some real PubSub later. So I would say, I know that it’s not available in all languages, but in general, it may be also some nice starting point. and it’s just less to migrate later when it will start to be problematic. Miłosz [41:10]: There’s one worse kind of cron I’ve heard about, is that you write a cron to check for inconsistencies in the system. Robert [41:22]: Oh, no. Miłosz [41:24]: So let’s take this example of calling this marketing API after a user signs up. So you realize this can sometimes fail and you write a cron job that does a check between the database and the marketing API and see if anything is missing and Robert [41:46]: Basically you need to implement everything twice. Miłosz [41:49]: Yeah and my my worst worry here is that you kind of promote ignoring issues And whatever happens, whatever issue like this you have in the future, a challenge like that, you will just write another consistency check. And this is another code you have to maintain. And basically you just sweep the issue under the rug instead of fixing the root cause. So yeah, I think that’s too naive to take to work in a large-scale system. Robert [42:30]: I can remind one case when it makes sense, but I would say it’s rather an exception. When you are working with some, let’s say, critical systems, you are quite sure that everything is fine, but you would like to do some double-check. Miłosz [42:51]: Checks in general, consistency checks in general, I wouldn’t say that are a bad thing, but if you are doing them because you can design a better API, that’s the issue. You anticipate them to happen. I would create a consistency check in places where I’m sure they won’t happen, but I want to be extra careful anyway because it’s so critical. Robert [43:19]: I think the big difference is if you are implementing some compensating operation in this checking. For example, if you are working in some financial system, it’s typical to do some reconciliation reports in some intervals, but it’s making reports. It’s not doing the compensating operations. I mean, your logic should be valid and you should just do validation at the end to ensure. But again, It’s just for critical systems that you need to be sure. For example, if your company or whatever, $100 came in, $100 should come at the end. But again, it’s not about doing compensation. It’s all about double-checking the logic that you have. Miłosz [44:06]: Right. Especially if you have a few of those in the most critical areas, that should be fine. But if in every API call you need to do this check and compressing logic then it won’t be maintainable after a while you will go crazy just trying to cover all these cases and then any API change or whatever will also break this so you have more code to maintain again I would fix the root cause if you know it can happen. Robert [44:42]: Agreed Thank you. Miłosz [44:44]: And one more regarding this naive approach is, you know, when you use messaging and use kind of events, like you said, those passive-aggressive events, or just something that’s more like technical, not in the domain or business sense. Miłosz [45:03]: It can seem like smart design, but not always the case. It doesn’t automatically give you good results. I worked once with a system where there were two applications, like an old one and a new one, and they shared some entities. And there was a system that, for every change in the entity, published an event that it changed, the other system consumed that and updated the entity, and the other way around as well. So, you know, if a naive setup like this, you ignore many edge cases that can happen. And there are some variants on the entities, they have some rules, but if you just do a naive update like this, even if it’s asynchronous, it doesn’t give you anything and can be even worse because there can be some inconsistencies and you won’t even know. Because maybe some message arrives out of order, or is overwritten by the other one. Miłosz [46:16]: So this comes back again to the event design topic, right? This needs to be a proper design. You need to spend some time and understand what you are doing. Robert [46:30]: And I think it’s probably time to give a big shout to event storming. That was usually pretty useful for designing some event-driven systems. But not only, because it may be also suggesting that event storming is for designing event-driven architectures or async architectures, but it’s not. I mean, event storming is good for basically designing any kind of system, but it’s pretty useful that you have notation that can be mapped to your software really directly, so I recommend it to check event storming. If you’re looking for some materials how to do that, there is an e-book that is probably unfinished for the last 5 or 10 years. But no worries. Miłosz [47:14]: It’s great anyway. Robert [47:15]: I think 5 or 10 years earlier it was already very good. So Alberto, really good job. Robert [47:29]: Another downside or trap related to asynchronous architectures is creating distributed monolids. Because, like many useful techniques, they are helping with many things, but often they are also opening gate of hell. Miłosz [47:44]: It’s the new normal. Instead of monoliths, we have distributed monoliths. Robert [47:51]: Let’s wait for distributed monorepos. Miłosz [47:54]: We’ve seen it too many times. Robert [47:57]: Yeah, but the usual scenario is like, okay, microservices are great, we need to be scalable, etc. And the team is starting to build microservices, and you start to have more microservices than people, and at some point you’ll notice that communication between them is hard because if one microservice in the middle will not work for a while because our Kubernetes cluster is not that stable yet, all our requests in the platform are failing because of cascading failures, we need to do something about that. And unfortunately, often a reversing decision of going into microservices is not on the plate, because come on. Miłosz [48:41]: Too late for that. Robert [48:42]: Yeah, we invested that much in many stuff that we don’t need. So, often the solution is, well, let’s use event-driven architecture. And, well, it’s making all the problems that the team had earlier, it’s making it deeper and deeper, because debugging everything is harder, understanding everything is harder, everything works slower. Or, okay, the one upside may be that, okay, at least it’s kind of more stable, with caveat that usually there is the thing that Gabriel mentioned about UX, that, well, if something is processing in the background, it’s a bit harder to show it in the UI. So compared to showing error directly to user like earlier, now something is happening and user have no idea what is happening. And yeah, it’s a pretty problematic thing that we’ve seen many times. So it’s important that asynchronous architecture can solve, in this case, some problems. But again, don’t be afraid to do the thing that we like to recommend, and we’ve did multiple times and it’s the microservicization. So basically getting your services and making less services. Again, having more microservices than people in a team, it’s terrible. Miłosz [50:04]: But the interesting part about this pattern or anti-pattern is that it comes from good intentions. Miłosz [50:12]: It starts with this idea that we have a small team now but we will grow and we need to to be able to separate some code easily. And then you have those separate services and you figure out, okay, let’s now just connect them like we did with function calls before. And then it breaks, but it comes with this good intention of separating the constants. Robert [50:44]: Oh, I think it’s problematic with good intentions, because I think all the bad systems that we worked with was built on the good intentions. Because, you know, it’s rarely the case that somebody had a bad intention. Like, let’s build the worst systems that we can. Yay, hooray, it will be so great idea. Usually, the intention is good. But I also understand that sometimes it’s also hard to reverse this decision. But from the other side, it’s also sometimes leading to systems that are totally unmaintainable. And, well, even I can say that rewriting is the only way to redo something. Miłosz [51:27]: It’s just counterintuitive, because you think that splitting things makes things decoupled. But it’s not really the case. It depends on how you communicate, really. Robert [51:38]: Yeah, so basically it’s all about splitting things that are coherent. So if you are splitting things that are coherent, you are just ending up with a lot of accidental complexity. And it’s pretty bad. I think it’s also partially the reason from where some people see asynchronous or even given architecture as anti-pattern and over-engineering. Because if you’ve seen many or a couple projects using some patterns and you’ve always seen that is overcomplicated. Well, it’s logical that next time when somebody will suggest you let’s do it asynchronously or let’s use evangelical architecture will be pessimistic because you’ve seen that, okay, it was always an over-engineered problem. But again, it’s probably, it wasn’t a problem of evangelical architecture. It was probably a problem of applying it in the wrong project. Robert [52:30]: It’s important to keep it in mind. So if you’ve seen some technique or tool in many places and it was always overcomplicated, it’s maybe not a problem of this tool. Maybe it’s a problem of a person using this tool, or this person maybe didn’t understand this properly. It’s a thing that we’ve seen in many places, and sometimes it’s requiring a lot of effort to convince people that no, no, it’s not because of this tool, it’s because of applying it in the hard place. But again, if you are not convincing somebody, it’s not our issue at the end. Robert [53:09]: I mean… Miłosz [53:12]: There’s also not really a strict definition of all those techniques. And depending on the language you use and other technology, it will be a bit different. That’s why it’s hard to judge. Wrapping up distributed monoliths, the coupling is the issue here and… Synchronous is not really decoupled by default, because events are also a form of coupling, in what we mentioned before, right, so… On the one hand, you have this, you public an event and you don’t care what happens later, On the other hand, it’s also a kind of contract between you and the consumers. Let’s say you have this distributed monolith and you now replace the RPC calls with messages. Is it decoupled? No, not really. It’s the same issue, just with different transport. So it’s more about the design of communication, not really the async or sync here. Robert [54:27]: Because, yeah, basically you can implement RPC over queue and it will be basically the same. I mean, it doesn’t change much if you are coding something directly over HTTP, if you are expecting to have a response from your operation. Miłosz [54:44]: Yeah, I think Gabriel mentioned this before, that there is one interesting pattern is used by NATS. The call seems to be synchronous, but it’s asynchronous under the hood. I’m not sure about this exact one in NATS. Robert [55:01]: But yeah, I guess that it may be about something that Celery supports. So basically, it’s basically doing RPC over queue. Miłosz [55:11]: Yeah, this is a weird hybrid approach when you can do synchronous over message broker. It kind of works. Robert [55:20]: It has some use cases, obviously, but it’s not really asynchronous. We also have each in Watermill, so we have the Request-Reply component. But again, it’s not really fully asynchronous. So it’s nice because you can have some response to know if something was successful, and if you are no longer listening, it will be processed anyway. Not like with Request, because if your Request data is originating, let’s say originating request, and it’s calling everything later, and this request will be cancelled. So if you are handling context cancellation properly, everything else should be killed, basically. So it should be kind of rolled back in all requests. But if you are doing RPC over cube, it will be done everywhere anyway. There are some cases when it makes sense, but… Miłosz [56:07]: Maybe one promise here is that you use one platform for all communication. So it sounds good on paper. We don’t need RPC because you can do everything over messages. Maybe it’s a good idea. I don’t know. Robert [56:22]: But it’s tricky. If you have a request and you would like to, for example, insert something in the database, your request was cancelled. And this cancellation is not propagating. because the translation is not propagated to the queue, so it can lead to some strange inconsistencies. Because you killed the request, so expect that it should be not written, but it’s somewhere later in going through the queue, so it will be done anyway. And it can lead to some strange state where you don’t know what really happened. Miłosz [56:54]: Yeah, I would summarize it as messages behave differently from requests most of the time. So, if you use one as the other, it can work, but you can have some weird edge cases down the line, because it’s just a different method of communication. So I will be careful with this. Okay, and any more anti-patterns? Robert [57:24]: So I have two connected anti-patterns here. So the first one is using no message ordering. So it’s maybe not anti-pattern, but it’s something to watch out. Because in many systems, you need to have some way of ordering your messages. So let’s imagine that you have event for subscriber subscribed and subscriber unsubscribed. And the problem is, if you not order them properly and build some read model based on them and they will be received out of the order, you may unsubscribe a person that was not subscribed yet, if the unsubscribe event will arrive first, because it can if it’s not ordered, and later you will have a subscribed event that should be… Miłosz [58:13]: Yeah, that’s one of the things you learn when you first start using messages. Those wish behaviors that seem simple at first. Robert [58:24]: Yeah, and the simple solution may be, let’s just order every message and just have everything ordered. And yeah, it sounds like a pretty good idea, but sometimes it may work. But it has also one big downside, at least. I would say two big downsides. So the first one is performance. So, basically, performance of processing those messages will be limited to how fast you are often to process them. And basically, they will be processed one by one. And if operations will be slower, it will just start to pile and you will be not able to process them. Miłosz [59:01]: Or block all processing of all messages. Robert [59:05]: And the second thing that I believe is worse is that if any of messages will spin, it will block all other messages. And it’s pretty bad because it’s killing all the advantages of asynchronous architecture in terms of stability. Miłosz [59:26]: Yeah, on one hand it can help you not care about ordering, which seems like a nice idea. Yeah, but if you make it too wide, then it can be also an issue. There’s a lot of things to consider here, so it’s hard to give someone a solution, right? So much depends on the domain and the system design. Sometimes you can get away with no ordering, but you can use something like a version. I remember one system where we didn’t have ordering at all, because all events updated different parts of a read module. Robert [1:00:12]: And I think it was also the problem where we’re using Google Cloud PubSub and it didn’t support ordering at this point also. Miłosz [1:00:20]: Yeah, we can kind of get away with it, but I remember we actually used times for solving some conflicts, so it’s also not ideal. Depending on timestamps is controversial. Robert [1:00:33]: It’s hard to synchronize. I think in some places where it was possible, we’ve been using versions. So basically every event was that within aggregate, let’s say that, within the, let’s say, transactional boundary, so where we need to keep ordering, had an incremental version. So if we received an event that was out of order, we’ve been rejecting that or in some cases we’ve seen that, okay, we received newer price, for example. Sorry, we received older price and we already stored the newer, so we just can ignore older. But the downside is that you need to support it in custom way for everything. Miłosz [1:01:17]: Not super complex, but something to consider. Robert [1:01:20]: But nice to have it out of the box. For example, Google and PubSub have ordering keys. Kafka has partitioning. So there are method workers that just have it out of the box. And you don’t need to think about that later, basically. Miłosz [1:01:33]: Remember, you can talk a bit more about this in the next episode. Robert [1:01:39]: Yeah, I think ordering is probably something that can be just an entire episode about that. because this is pretty… Maybe not complex, but it has many different use cases that it’s good to know about. Because in many cases, you should approach it differently. Often, you don’t need to care that much. You can choose something simple and it’s fine. But it depends a lot on scalability requirements or and, let’s say, resilience. Miłosz [1:02:10]: The balance is difficult. It’s easy to do. No ordering or ordering by every message something in the middle you have to consider manufacturers Robert [1:02:20]: Yeah but tldr i would say it’s try to do as narrow ordering as you can so for example try to do some subset of events for one customer maybe or you need to basically analyze where this ordering matters and based on that, do really narrow ordering and it will work, basically. Miłosz [1:02:48]: There’s one thing that’s similar between async and sync. I think it’s API design. So with Synchron’s approach, you have some kind of contract in how your API works. If it’s a REST API, you can have some open API document with gRPC, probably protobuf schema. Robert [1:03:14]: Yes, I see that Gabriel asked about that. What we are usually using is protobuf, because you can marshal it later to at least two formats. Usually we are using JSON as a transport format, because it’s easier to debug. But if you care about performance and the size of message, you can use protobuf, so it will be a bit smaller and faster. And you can also convert it to multiple programming languages, and also it’s nicely supported the versioning of fields. Miłosz [1:03:51]: There’s the cloud events schema, I think. We didn’t use that yet. Protobov usually does the job. The point is you need to follow similar ideas for both sync and isync here. So your event schema or your API schema is your contract. So you have to make sure there are no breaking changes. And yeah, the API is stable. And probably do some versioning if you need to do breaking changes. We sometimes have events with version delay, like v1, just to anticipate it. Not always needed, but sometimes useful. Robert [1:04:44]: Yeah, and it’s also worth mentioning that many message brokers are supporting some kind of schema registry out of the box. So those message brokers can validate the schema for you. So, for example, Kafka has it. I think Google and PubSub have it. I think it’s also worth mentioning that some companies are using Avro for that. I think we didn’t use that, but I know that it’s often used in some companies, but I think I guess that it’s more in Java world. I think in Go world, more gRPC is used, at least from our… Protobuf. Protobuf, yeah. Protobuf is more used in our bubble. Miłosz [1:05:29]: It’s pretty nice. And as a plus, generated code is It’s a quite popular choice and it works pretty well with Protobuf as well. Miłosz [1:05:49]: Shall we move to the final question? What to choose? Robert [1:05:54]: Unfortunately, it’s as you heard. There are really many factors to consider. Yes. Miłosz [1:06:05]: And the balance is the hardest part, as always. There is no silver bullet, so you have to figure out what balance means for you. It’s a bit like the CAP theorem in databases, right? So you have consistency versus availability, and you have to pick one. Robert [1:06:26]: Yeah, so in async system you basically are sacrificing consistency, but you have better availability. Miłosz [1:06:36]: You have eventual consistency, but that’s it. Robert [1:06:40]: But it’s eventual. Cool. Miłosz [1:06:43]: So what should we choose, do you think? Robert [1:06:48]: I think to give you some simple algorithm, maybe. If you’re in-depth, you can always start with synchronous. I would say that in many cases, it will be a good starting point. In many cases, using async, by default, maybe, Maybe not over-engineering, but maybe not needed at the beginning. And if you have infrastructure already in place, switching something from async to sync, may be not that big difference, as long as you didn’t do any strange thing. And it’s especially simple if you are using clean architecture, because you should have some application layer entry point to your application. So basically calling you from RPC or event handler or something, it can be a pretty simple change. Of course, it depends a lot on the use case, but again, it’s if you’re in that. In some cases, you may be just sure, like you’re calling some external system and you know that it’s not stable, do it synchronously. You have some critical flow that you know that it needs to be working, you need to send some external data, do it asynchronously. Again i’m saying more about cases that you are not sure. Miłosz [1:08:10]: Sometimes it’s a product decision as well if something can happen in the background and sometimes it’s tricky because in the background can mean a second like later or you know 10 seconds later so it so it’s also it takes some skill to to be able to talk to you know whoever your stakeholder is whoever is responsible for the product to understand if it’s fine to do it in an asynchronous way. Which means most often it will show up in the UI right after, but sometimes it will not after a few minutes maybe. That’s also counter-intuitive. Why would you talk with your product owner about sync versus async architecture, sometimes it makes sense. Miłosz [1:09:07]: Probably it’s good to watch out for the extremes, as always, right? So, not everything can be handled by synchronous architecture, as you mentioned. There are some hard limits, where more complex systems just won’t be able to do it. But as you said, sometimes doing everything over messages can be also over-engineering. So if you ever consider doing everything one way, probably you need some balance. It’s usually a clear sign. Robert [1:09:45]: And if you maybe didn’t work in any system that required building asynchronous architecture, it’s fine. But anyway, I would recommend to check out how you can do that, because it’s just know what tools you have available. So if you have a problem that will require asynchronous architecture approach, you’ll just know, okay, I have tool for that, it fits here, and I can use that. So you will not need to learn it when you have this problem, because it might be just too late. Miłosz [1:10:17]: Sometimes a hybrid approach may be fine. One is what we mentioned before, the RPC of our messages. A bit unusual, maybe if it fits some use cases. Something else that comes to my mind is CQRS here. So, TLDR is a pattern where you split your controller functions, let’s say, into commands that change the system and queries that fetch the data. Super short definition. Not to dive too deep into it, but the interesting thing about it is that it gives you this extra structure in your code. Commands and queries are separate. And then it can make mixing them and how you call them a bit more intuitive. Because queries are usually synchronous. So someone is waiting for the data, that makes sense. and commands can be either. So it can give you this mental framework to decide how to mix sync and async in one project. Robert [1:11:33]: I think it’s interesting, the CQRS example, that it’s a good example of technique that people really misunderstood because for many people if they hear CQRS it’s immediately asynchronous but no. So it’s totally misunderstanding of this pattern. Again, we’ll add to materials our article about SQLRS, because we’ve been covering that. Milos also did a really nice presentation about that, so I think we can also put it to the materials. It’s important to mention it all in our SQLRS. It’s not about doing things asynchronously. It’s some implementation. it’s worked nicely, but you can do skewers with just doing everything synchronously. And it’s nice because if you are using skewers in a proper array, so when commands are not really returning stuff, you can convert it to asynchronous architecture really easily, because it should not really depend that much on having the feedback from the command. Miłosz [1:12:40]: It helps if you don’t have these mixed functions that change the state and also return data. It can be more tricky to decide what to do with them. Robert [1:12:49]: And it also doesn’t require some super big overhead because this is also something that we heard from many people, like, oh, security requires so much effort. But no, it’s just a matter of, instead of having some super big application services with everything, just have multiple little services with commands instead of arguments and commands that doesn’t return stuff. And that’s it. And you have the basic secure implementation in your product. You can put it into your CV. Miłosz [1:13:24]: Um… Robert [1:13:27]: So yeah, I already mentioned also the migration. So SecureOS helps with that. But you can also get some hybrid approach, because often at the beginning of the project some endpoint can be synchronous, but with time, more and more stuff is added to that. Robert [1:13:45]: We have actually a good example in our trainings platform. So we have webhooks from our payment provider about information that somebody bought our training. And at the beginning it was pretty simple, just adding, ah, somebody has training. But now it has a lot of stuff and we need to actually convert it to asynchronous to using events. But now it’s probably doing five things. And sometimes some of those things are failing because it’s just doing a lot of that. But, again, at the beginning, it didn’t make sense to do it asynchronously because it was pretty lightweight, but with time, you can convert that. And you can even start with still having a synchronous endpoint and with emitting some internal event or command and just cutting it part by part. If it’s some internal logic, you can, at the end, maybe just remove this endpoint and say the team that is calling this endpoint, you can now just emit an event or the opposite. it. So, for example, if you have a problem with data quality and other team is not taking responsibility of that, introduce endpoint instead of event and say, if you will not respond that this is valid, sorry, you need to provide this valid event and just agree what data should be in this endpoint to make everybody happy. Miłosz [1:15:04]: And speaking of code structure, maybe one more thing that can help is layered architecture or cleanarchitecture or hexagonal or whatever you want to call it. If you place the logic in some isolated logic, place in your code, and then have two entry points that are either messages or RPC or HTTP. It’s super easy to switch between how you call the handler, because you have just two entry points that do nothing else, just just map the request to your internal structure. And it makes it super easy to migrate as well. Robert [1:15:53]: And it’s happening, I would say, because it’s often argument in clean architecture, like, okay, in clean architecture you’re separating database, you can switch it, or whatever. It doesn’t happen often. But changing things from synchronous and asynchronous actually can happen quite often. Miłosz [1:16:11]: Or just both at the same time. You can have some logic that can be executed synchronously or asynchronously by message. And if you use the layered approach, it’s super easy to do. Robert [1:16:22]: Yeah, and it happens pretty often because database, again, you are not changing it very often, but changing things from synchronous to asynchronous or vice versa, the first I think it’s happening more often, but it’s happening often. Because, again, just some operations are starting to be bigger and bigger. You start to have more and more teams. Your product is growing. This is, hopefully, what’s happening. And in this case, it’s just making your job easier and saving all of your time. So it’s, I think, a great investment that is not mentioned that often. Miłosz [1:16:56]: Yeah. Okay, maybe as a finishing point, we can talk about reducing this accidental complexity that comes with starting with messaging. First, once you start, but there’s a lot of topics to grasp. We talked about some today, but it’s not really an episode about event-driven architecture, only the comparison of Async and Sync. So one idea might be to start with some PubSub you already have, which is probably SQL database. I probably have some Postgres or MySQL running around, and you can actually have a pretty simple PubSub running on it. That should be good enough for most use cases. Or you can use a cloud-based PubSub. It means you don’t need to maintain the architecture or infrastructure, which can also be a complex task sometimes with high availability in place and all this. Robert [1:18:10]: I think Postgres is a great starting point and it can handle much more than you think. I actually have some idea about one article that maybe we can show how far away we can push Postgres-based paths up, because I’m pretty sure that we can push it pretty far away, but let’s see. Miłosz [1:18:29]: Yeah, unless you want some complex topology. For starting out, it’s a great way to start. Another way, maybe using a high-level library that abstracts away some of the concepts for you. So for example, Robert mentioned earlier Watermill, which we maintain in Go. I’m not sure about other languages. I think it’s something there’s something similar in Python. Probably in every other language there should be something like that. Because you probably don’t want to deal with all the protocol, the infrastructure you use, all the low-level details, so you can focus on the high-level publishing and subscribing to messages. Robert [1:19:21]: It’s pretty easy to mess something with that. So lose message or make some sub-toolbox. We’re creating WaterMeal already for a couple years, and you’ll be surprised how many little quirks you can have in every PubSub. It’s very hard to debug at the end, because you may just be losing one message for one million messages, and it’s very problematic in systems. It’s good if somebody already did it for you. If, for example, you don’t use Go, We’ll recommend to use Go, but if you can use Go, you can always even look on our SQL implementation and see and inspire by our queries. Miłosz [1:20:06]: Just use Go. Robert [1:20:07]: It’s a best idea. Miłosz [1:20:11]: It’s not that complex in the end. But there are also some quirks, so it’s better if you don’t reinvent the wheel. Because it took us some time and I think four major versions to arrive at where the SQL implementation is right now. So I’m going to manage cases. Miłosz [1:20:33]: So, if you’d like to learn more about async architecture and event-driven architecture in particular, we have an event-driven training that’s sales open for the next two weeks, starting from today. So you can learn those concepts in a more hands-on way, solving exercises. Robert [1:21:00]: If you are interested in learning, Doug, you are actually pretty lucky, because we’re running this training by Sayer, so if you are listening to it later, it may be already not open, but again, it will be open next year. Miłosz [1:21:17]: If you listen to it later, you can join the waiting list and wait for the next sale. We run it as a cohort, so everyone learns together. And yeah, you can just Google Go event-driven or we will put a link in the notes. Robert [1:21:32]: We have pretty good SEO, so it should be on the top of the list when you will Google Go event-driven. Miłosz [1:21:40]: Okay. Before we move to the Q&A. Robert [1:21:45]: Yeah, definitely. Okay, so Dima mentioned that naming always matters. Yeah, so definitely it matters more often than you will think, but… Miłosz [1:22:07]: Okay, so Gabriel shared more about that events should be facts that already happened in the system, what we talked about. Provides many advantages as states can be recreated from events. Yeah, that’s one of the things we didn’t mention. It’s super useful so you can have this event log somewhere in your system and then reply those events to do whatever you want. Robert [1:22:31]: And in other words, you have event sourcing, but event sourcing is not asynchronous. I mean, event sourcing can be asynchronous, but it can be synchronous as well. So I would say that it’s synchronous by default. And then it’s probably a nice topic for other episodes. So we mentioned already financial systems in a couple places, and event sourcing is actually great for this kind of system. When you need to have audit by the law or you need to have audit because you would like to understand how you ended up with some state. Miłosz [1:23:10]: Gabriel also mentioned this dependency matrix of events based on types could be helping an organization. I think I said something like, I wonder if there is a ready software for this. I think event catalog or something like that, but we didn’t use anything that we could use across teams and I mean place where you can just go and see which team uses what because it is to integrate with your code I guess ideally if you don’t want to keep it up to date over time which can be a manual effort so it sounds like a challenge to do it for any language you use and if we find something like that we’ll let you know Robert [1:24:15]: So Gabriel mentioned that there is a nice chapter about event storming and learning domain driven design from Orly. I didn’t read this one, I’m not sure if you had a chance. Miłosz [1:24:25]: I have it on my shelf. Still waiting. Robert [1:24:29]: TBD. Maybe this summer. Miłosz [1:24:32]: But everything about event storming, I think you can recommend right away. Robert [1:24:38]: Yep, definitely. I think event storming was one of the biggest game changers for working with some more complex projects and collaborating with some multiple stakeholders. Miłosz [1:24:51]: Also not super complex to learn, but you have to be willing to talk to people. Can be controversial in tech teams, sadly. Ordering is probably not needed if you have a good state machine, which guarantees the state of aggregates. It sounds to me connected to event sourcing again. Robert [1:25:17]: Or the thing that we have with versioning. That’s true. You can implement that. But from other side, if you also do it properly from a message broker perspective, it can simplify also a lot. Because you can just listen to event and don’t care about that. We’ve been able to compare migrating to that and ordering, at least in our case really help to simplify it a bit. Because I think it’s good because if it’s done properly, it works out of the box. So kind of out of the box, let’s say. And it’s harder to make mistake when you need to remember about something. But you can also implement universal state machine that can handle that. But universal things that are implemented by you often may have some chances. Miłosz [1:26:07]: But you can see how many approaches there are to ordering. It’s a pretty long topic. You have to consider many approaches. So thanks for mentioning this one. Robert [1:26:21]: So Gabriel will have Monopoly on our chat today. Miłosz [1:26:25]: Thank you for contributing. You are the third author today. Robert [1:26:31]: I think at some point we should try to also share the link to join with us. That would be cool. Risky but cool. Gabriel also recommended Reactive Manifesto, which is a nice definition of Reactive Systems. I remember reading that probably a long, long time ago. Miłosz [1:26:52]: Yeah, same. Robert [1:26:53]: Recommend checking, yeah? Miłosz [1:26:55]: I feel like an ignorantat the moment, sorry. Robert [1:27:00]: In Scala, there is a Lagon framework, which was nice, Akka is under the hood. Maybe. We are not Scala guys. Miłosz [1:27:12]: Yeah, so it’s good that there is similar software, so you don’t need to understand everything, what’s going on, which can also be tricky because sometimes you may lose some you may not understand something correctly, the configuration of some PubSub. But to start out, it’s super important to have this A-level concepts in place. You just publish a message, subscribe to subscribe to a message and don’t care about what the bytes do underneath. Robert [1:27:41]: And also Gabriel mentioned that it’s under the hood, so it would be actually interesting to see how actor model is going up. I mean, if it’s visible or not. Personally, I’m not a big fan of actor model. I mean, I don’t like the mental model of actor model. So even driven fits in my head much better because you can get to even storming and it maps one-to-one in some way and actor model for me is a bit more like magic mental mapping that you need to do and, I don’t know what’s your opinion on that but. Miłosz [1:28:13]: Yeah I didn’t use it much except for playing a bit with elixir so yeah I can’t say much maybe you can do another episode sometimes about comparison Robert [1:28:29]: Fortunately Gabriel said that but I love Go much more. Miłosz [1:28:34]: Way to go! Robert [1:28:43]: Okay. Cool, so a couple more seconds for more questions. Miłosz [1:28:54]: While we wait, it’s the time you should hit the subscribe button and rate our show Robert [1:29:02]: Let’s wait Please click this button You didn’t click yet I see your screen Okay, thank you So, also now 5 star review, Let’s write something nice. We’ll wait here. We have time. Miłosz [1:29:23]: Yeah, and instead of the YouTube subscribe, there’s an even better subscribe button on our website, where you can join our newsletter, so then you won’t miss the updates from us. Robert [1:29:34]: Plus, you will also receive notifications about the episode notes. There is an entire transcript. We’re linking to all sources that we’re mentioning here, like WaterMule, like ebook about event storming, and about a couple more things that we mentioned and I don’t remember now, but it will be on all-in materials for this episode. Plus, also if you release any new articles on the blog, you’ll be notified. So I think it’s pretty cool, because I mentioned, for example, article about SSC that we have on our blog. So we’re trying to share a lot of quality stuff that will be useful for you. Something more, Miroz. Miłosz [1:30:19]: I think we have no more questions. Thank you, everyone. Robert [1:30:22]: Just a reminder. So, Go Event Driven, available for next two weeks. Later, you need to wait for half year. So, don’t wait for the last hour, because… Miłosz [1:30:34]: Better to join now. Robert [1:30:35]: We are starting, and you will need to wait. Cool. Miłosz [1:30:41]: Thank you, everyone, for joining us. Thank you, Robert, for joining me. Robert [1:30:46]: Thank you, everyone. Thank you, Miłosz. And see you in two weeks. And we’ll talk about… Miłosz [1:30:52]: Event-driven architecture more in-depth than today. Robert [1:30:55]: Sounds like a good plan. So, see you in two weeks. Thank you very much. Miłosz [1:30:58]: Bye-bye. Robert [1:30:59]: Bye.

Quick takeaways Solve real problems first - successful open source projects start by addressing actual needs, not by looking for problems to fit a solution Keep breaking changes minimal - Watermill stayed on v1 for 6 years with no breaking changes in the core library, building trust with users Examples and documentation are crucial - provide real-world examples with automated tests, not just simple “hello world” demos Promotion matters - creating a great library isn’t enough; you need to actively share it through conferences, blog posts, and communities Be patient with growth - Watermill took 7 years to reach 8,000 stars; overnight success in open source is rare Introduction In this episode, we share the story of how Watermill, our event-driven library for Go, grew from a side project to a popular open source library with over 8,000 GitHub stars and 100+ contributors. We discuss the key decisions and strategies that helped make Watermill successful, from focusing on solving real problems to maintaining backward compatibility and building a community around the project. Show notes Watermill documentation — our open source event-driven architecture library for Go Watermill repository Go Event-Driven training that complements Watermill Real World Examples Quotes My idea was that it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial reads, network failures. Robert It’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this. Miłosz If you would like to create a library that people like to use, create something that people need. I know that it sounds obvious, but I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us. Robert If someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea. Miłosz If you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero. Robert Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life. Miłosz Timestamps 00:00:00 - Introduction 00:01:06 - What is Watermill? 00:02:54 - Why we created Watermill 00:08:23 - The long journey 00:10:33 - Why create open-source projects? 00:13:42 - Create something people need 00:18:20 - Early feedback 00:20:00 - Documentation and examples 00:25:14 - Libraries vs frameworks 00:26:17 - Public API design 00:28:33 - Testing strategy 00:34:28 - Breaking changes policy 00:41:46 - Maintenance challenges 00:43:02 - Marketing is necessary 00:50:18 - Commercial synergy 00:55:20 - Q&A session Transcript Robert [0:00]: Did you ever wonder what’s the real key to build a successful open-source project? Today, we are going to share what helped us to grow Watermill from a small sideproject to a popular open-source library with over 8,000 stars on GitHub. What have we learned along the way? I’m Robert. Miłosz [0:19]: And I’m Miłosz, and this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We have spent almost 20 years working together in different projects and teams. We learned that following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices and grow into the principal engineer level. Robert [0:46]: So if you have any questions, leave them on the chat, so we’ll surely address them at the end of the live. or maybe during, if it will be also relevant to any place that we’ll cover during the discussion. Miłosz [1:01]: Right, so let’s start with what is Watermill for anyone who is not aware. Watermill is a code library we maintain for building event-driven and message-oriented applications, and building it the easy way. And it has currently over 8,000 stars on GitHub, more than 100 contributors. Robert [1:25]: Probably somebody’s using that. So yeah, obviously GitHub stars is some kind of proxy, probably to understand how popular the library is. But, well, we’re not buying them, so probably it means something. Miłosz [1:37]: Yeah, and GitHub also shows, I think, above 1,000 repressories that use it. Some of them are also quite popular. So this is kind of a social proof, I guess, the best we can have that it is in fact used. Robert [1:51]: The outside is that we don’t really know what companies are using that. So we know some, but probably most we don’t know because we have no way of backtracking who is using Watermill. So if you are using Watermill in your company, let us know, because it will be cool to know. Miłosz [2:07]: Sometimes we hear from people on conferences or just randomly on the internet that But yeah, we use Otremio, we love it. But it happens after someone uses it for a few years, let’s say. So it’s not always the best way to know. Robert [2:21]: So it’s always cool to hear from you that it’s useful. So we always appreciate that. Miłosz [2:27]: Yeah, so Watermill supports 12 officially supported Pub/Subs like Kafka, RabbitMQ, NATS, but also Postgres or MySQL. And we have SQLite support coming soon. Robert [2:40]: Yep, exactly. Miłosz [2:42]: So whatever infrastructure you use, you might find it useful. Miłosz [2:49]: Before we go further, maybe let’s focus on what Watermill solves. Why did we create it? Robert [2:56]: Yes, probably it will be easiest to start with actually how Watermill started. Because I think that this is interesting and it may be also useful for you to try to create some library. And, oh, it was like probably many libraries that are there. So we’re working on some project where we had one challenge. So in this project, Event-Driven Architecture would be a perfect fit because of resilience requirements and some tasks that we could do in the background. Upside was that I was already familiar with Event-Driven Architecture, but most of the team wasn’t at this point. Miłosz [3:35]: Yeah, and I remember the meeting when we sat in the room and you draw the Event-Driven Architecture on the whiteboard and you were very confident it would work. And the rest of us were like, I don’t know, it seems complicated. And we never used those pops-ups, Kafka is scary, what if we mess something up? Exactly. What would be useful is to have something easy to use, to abstract away those complicated details. Robert [4:06]: Yeah, and I would say that the bar was set pretty high because my idea was that, it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP handlers, HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial leads, network failures. Miłosz [4:37]: Reconnect. Robert [4:38]: Yeah, it’s a lot of things that can happen in the background or it’s happening all the time, But you are not thinking about this normally. You are defining, I would like to have handler at this URL with this method, and that’s all. And it’s really high level, and you don’t need to think about all the details. And, well, Watermill is a similar idea. So you should be able to create your event-driven services, but with really high-level API. And adding support, for example, for one event should be super simple. You don’t need to really think about topics, Kafka, or any message broker that you are using under the hood. It’s not an only thing, because there are a couple more things that Watermill is solving. The next one is handing for you a marshalling and un-marshalling of messages, because usually you are using some transport. Robert [5:36]: So JSON, protobuf, whatever, and just doing it over and over, it’s just boring and a waste of time. Another thing that also is cool and Watermill can support is allowing you to swap PubSub implementation pretty quickly. So if you have seen some presentation that I was doing about Watermill over the time, I often show an example when I was using, for example, Kafka, and I wanted to switch it to Google Cloud PubSub or vice versa. And I was able to do it during live coding, and it took like one minute or less to do that. And it’s cool about Watermill because it’s putting high-level abstraction on this PubSub. And thanks to that, you can switch it pretty easily, like with ORM, basically, with databases. basis. Miłosz [6:27]: Yeah, so those are the basics we started with. And then we built more features on top of them. Some of them are quite complex, but with a high-level API, so you can use them easily, like CQRS pattern with commands and queries. Sorry, not queries, but commands and events. Or Poison Queue or the transactional outbox pattern, which is super useful and very common. are commonly needed, not everyone uses them, but probably more people should. And the important part of those features are that they should be easy to use for everyone in the team, even if they don’t know how it exactly works under the hood, which is something we would need in that project back then, because we, not all of us were experienced with even the architecture, but it would be a nice approach for the project. So if we had Watermill back then, it would be much easier to start. Robert [7:30]: Yeah, and probably today we also don’t want to go that much into details about some Watermill specifics and things like what’s the difference between PubSub and Q, because I know that for many people it’s not clear what’s the difference, what are use cases for PubSub, what are for queue. It’s something that we’ll cover more in the episode that will be in two weeks. Today, I would like to more focus on what we believe that made Watermill successful. Because, again, it’s probably not the most popular library of all times in Go, but it’s probably the most popular library for building event-driven applications in Go. So I think it’s a pretty big achievement. And there are a couple of things that we believe that helped us with that. And I think it’s also worth mentioning that it wasn’t overnight success. And as we mentioned, so it was after some project that we started after work. And we started to spend more and more time on that. And it took us time basically to make it. Miłosz [8:37]: And the first comment was seven years ago? Robert [8:39]: Yeah, I think it was seven years ago and seven days, something like that. So yeah, super long time ago. and it took us half year to issue version 0.1 so not 1.0 0.1 so it was some kind of alpha that we were able to raise to people so and i think i didn’t really write down how many comments it was but it was a lot of work to at least prepare this 0.1 version i remember i’m actually not sure but I think before version 1.0, we spent 500 days to release that. And what also helped us during that time is that we didn’t have a daily job, because at this point we, for the first time, tried to go full-time on our own. That time it didn’t succeed, but at least we released Watermill 1.0. But it took a lot of effort to do that. Miłosz [9:41]: It was just going over time, steadily, instead of this huge success right away. Robert [9:49]: Like in previous episodes, we’re not giving you any magical solution that will make your library super popular in one month. Because probably it’s often about a lot of luck because you can create a great library, that for some reason didn’t take out because there are, for example, 10 other libraries that are doing the same. So there is just a lot of competition. And if you are not lucky, well… Miłosz [10:23]: It’s difficult with regular products and open source is more specific than that. It’s probably worth asking why someone would create a open source project it’s probably not to make money you can see many companies and projects try to monetize on open source, it usually doesn’t end well all the issues with licensing and so on Robert [10:50]: I think it’s not fully fair when you’re starting an open source project and making it closed source after time after more people are using that but unfortunately it’s often happening. I guess that often it’s some pressure from investors that invested in some company and it kind of makes sense. You can expect that if you are using VC-baked open source that someday investors will come and ask where is my money back. Miłosz [11:21]: I can count on sponsors as such but it’s probably not the idea to do it just for this. We found a way to connect it with our commercial projects, but we will touch on this later. But so far, I think we can summarize that we learned a lot doing it. That’s probably the primary reason I would try again to build an open source library. Just to learn something new and publish it. And also gives you some credentials if it succeeds. Like if you’re looking for a new job or just want to have some proof of your competence. That’s a good way to do it. Robert [12:06]: I think it’s interesting sometimes to hear on the interview, on both sides actually, because sometimes when we’ve been interviewed, some people know Watermill, but I also remember interviewing some people and they were like oh, okay, you’re a creator of Watermill and you work here, nice. Miłosz [12:22]: But that’s the difference between a project you just upload to GitHub and no one uses, and some project that some company actually found useful. So it’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this. Robert [12:48]: And I think it’s also nice when you’re recruiting somebody and this person can send you his GitHub or her GitHub and you can see that the person is working on some open source project after hours. And you can have some understanding on how this person is working and you can also you know feel that this person is able to, spend some time after job to learn something maybe or maybe create something cool that other people can also benefit from yeah it’s. Miłosz [13:19]: Also a nice way to give back to the community because we’ve been using many open source software before and this way other people can use our ideas which also It also feels nice, you can do it for the satisfaction you get from it. Robert [13:37]: All right but it all sounds cool but many libraries are like that and they’re not that successful so i think it would be good to discuss some things that really makes some projects successful and, some of my biggest observation from other projects and i know it may sound obvious but, if you would like to create a library that people like to use create something that people need because I know that it sounds obvious, but from other side, I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us. And, yeah, it also doesn’t mean that it needs to be some solution for a problem. Miłosz [14:36]: A very generic problem, right? Robert [14:38]: Yeah, because solving generic problems is more harder because it requires more effort. So probably it’s good to find something in the middle, like, for example, water mill. So, creating event-driven applications, it’s not something that every company does, but a lot of companies does. So, it’s big enough, let’s say, market, if we can say it this way. But, okay, we can say that if you’re building open source library, it’s pretty close to creating a product. So, you can say that it’s some kind of market for that. And it’s big enough that there are many people that can use that and benefit from that. but it’s also kind of niche so there is no competition around there so it helps a lot. Miłosz [15:25]: Yeah, and sometimes if you start a project and you have a problem you want to solve and then you look for a solution and you find an open source library so you might feel discouraged to try and build your own I think it still makes sense to try because you may have a fresh approach. You can provide a better API or better design or less dependencies. So there’s definitely a place for more libraries that solve the same thing. Robert [16:01]: Yeah, definitely. So yeah, I think it’s a good… Miłosz [16:04]: In Go ecosystem, it was clearly visible because in the very beginning there were only a few libraries to do everything. So everyone sticked to one, let’s say, HTTP router. Because there were no other options. But as other libraries appeared, many people moved on. Aside from routers, maybe CLI libraries are also a good example. So there’s still a place for better APIs or a different approach that will help people to do something in a better way. Robert [16:47]: Again, try to focus on the problem that you are solving and avoid trying to find a problem for your solution. So I’ve seen always that people found some cool technology or some cool idea, but they didn’t have a real use case for it. I think it’s probably easy to do AI now. So you can try to put AI in many places, and it will be helpful in many places. But in many places, it will be more looking for problem, for solution that you have. So you should also really watch out for that. I think it’s one interesting observation about this focus at the beginning on solving the problem. I think at the beginning, we didn’t know what word terminal is. So we had, as we said earlier, we had a problem that we had people that didn’t build any event-driven application earlier. And we didn’t have an idea, okay, we need to build a library for abstraction. Miłosz [17:51]: All use cases everywhere. Robert [17:52]: Yeah, yeah. Fun fact is that actually Watermill at the beginning, when it was alpha, it was named GoDDD. So it was probably something that, okay, it was around these domain events that we had there. And it started with that. Miłosz [18:07]: I’m still not sure why. Robert [18:11]: Probably we’ll never know. And yeah if you’re experimenting with time you are experimenting with time, it’s important to not keep this for you forever so create some version that will be useful for people and try to share it ASAP for feedback. Miłosz [18:40]: Just mark it as experimental so people are aware that the API can change so they are not annoyed. It’s normal to have some proof concept and then keep changing the API if you find a better way. That’s fine. Just make sure people are aware. But this early feedback can be super helpful though. Robert [19:02]: Yeah, and one of the best early feedback that you can have, and it can be also a nice hint how to find a, problem that people care about is using it in your daily job as long as you can because i know that it depends it’s maybe hard in some teams but like in our case so we’ve been building it after hours but we’ve worked in a company that it was fine to just use it in a real project that we worked in so thanks to that we had real-time feedback for that and we’ve been able to dog foot it with it but Obviously, it’s just a starting point. At some point, you need to share it outside so other people can give you feedback, and you can adjust to that, because you will be not always right from day one. Miłosz [19:54]: And after a while, it’s probably a good time to start with documentation. We have some docs in place, and the go doc we have is not enough, usually. Because the go documentation is not that bad, but it’s still… It’s sometimes hard to get started with just the code documentation. So, we found it very useful to have this getting started page, so someone can easily see how to install the library and some minimal example to get started, also some minimal theory. And then lots of real-life examples are also super useful, which is sometimes… I’m super happy when I see some library and I see the examples directory, I often go straight there, instead of trying to figure out how it works. Because you can just copy some snippets and it gives you a better idea of how it works, rather than trying to understand from the code itself. Robert [21:02]: Or clone it and run it locally and play with that. So it’s also pretty useful. And I think it’s also pretty close to our way how we believe that learning is the best. So actually learning by doing. So it’s really close to it. Miłosz [21:21]: And lots of our examples we have for Watermill are real-world examples. So they are more complex than a simple hello world. So you should be able to copy them to your solution and change them, and it should be what you need. Robert [21:36]: Yeah, so if you’re longer with us, you should know that we don’t like examples like to-do apps or some fake examples, because usually it’s just hiding a lot of complexity. Okay, maybe some Watermill examples aren’t that simple, but they’re as simple as they can to show how to solve these problems. Miłosz [21:59]: Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life. Robert [22:14]: Yeah, for example, we have the example with handing ordering in your messages. And this topic is not simple. There are a couple things that you need to do right. It also differs a bit for different PubSubs. But again, it’s easier to give you more complex example rather than you need to figure it out yourself. Because we know that it’s not easy because at some point we needed to figure it out as well. At the end, your life will be easier with those examples. One tip. So if you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero. So every example that we have, it’s included in the documentation, but we have also automatic tests that are checking if those examples are working properly. So it’s cool because when we are releasing a new version of WaterMill, we’re updating dependencies, we can just run tests on all examples and ensure that all of them are working previously. Miłosz [23:18]: It’s easy to break them without a version bump or something like this. Robert [23:26]: But it’s worth mentioning that it’s not that easy, unfortunately, to create such tests. Sometimes it’s taking a pretty long time to run them. But again, it’s still better than not having an automation for that. Because doing it by hand is 10 times slower. And it’s nothing worse than examples and documentation with snippets that doesn’t work. Because it’s just a super bad experience. Because if somebody is trying to use your library and your examples are not working, Well, probably the person may just leave your library and don’t use it at the end. Miłosz [24:03]: Yeah, but assuming they stay, you usually will have some early adopters of your library. It’s nice to have a place for communication, for building the early community. So in our case, it was issues and then Discord. Our Discord and dedicated channel. So it gives you a place to listen for this feedback from people and your suggestions. And also sometimes they contribute, which is super nice. You have your first contributors, so it’s not just your project, but you start to feel like you maintain it, actually. It starts to get more real. What other things help? Robert [24:53]: The next thing that is coming to my mind is the fact that Watermill is not a framework. So, we’ve covered it more in the previous episode about frameworks, but TLDR is that. We are not big fans of frameworks in Go for multiple reasons. And I think it also helps with building a library when this is not a framework. Robert [25:14]: For one reason, because it’s easier to adopt if it’s just a library. Because when you are starting to use some kind of framework, it requires more changes in your code. And I guess that many people may be also not… Miłosz [25:34]: So it locks you in. Robert [25:36]: Sorry? Miłosz [25:37]: It locks you in, so it’s harder to migrate out of it. And we chose this way of being more independent. So it’s easy to throw a water meal away, if you choose to do so, and replace it with something else. And that’s my design, because we prefer working with smaller libraries that are not as heavyweight as frameworks. Miłosz [26:13]: Maybe something similar is to how public API design matters. I think in Watermill the publisher and subscriber interfaces are quite good examples of how having a simple interface and the complexity hidden behind it can lead to nice API of the whole library. And we have the higher level components built on top of them. So it’s… quite… easy to use and fun to try. Because you don’t need to dig through the documentation to understand how it works. Use those two functions together and you say, oh yeah, okay, this is how this stuff should work. Robert [27:09]: And I think it’s also cool because it’s giving you a lot of flexibility. Because we are giving you some building blocks in Watermill, but we know that there are a lot of different kind of projects. And it’s just impossible to create a library that will fit everybody’s needs. But if it’s built in the modular way, it can be adopted much easier to totally different use cases. So you can just use some of the components, use some more lower-level components, for some cases, some higher-level components. So to give you some examples, so you have in WaterMill, you have high-level interface for handling events that is handling marshalling for you, but you have also lower-level API that you can use, interact with PubSub more directly. So it just will have message with slice of bytes payload, and that’s it. And it may be a great choice for some low-level processing or data pipelines. It was hard people that they were using it in some strange cases, let’s say. Miłosz [28:17]: So you can choose what to use, basically. Robert [28:20]: Yeah, and it’s just making a list of use cases that we can support much, much bigger. Miłosz [28:28]: And what about testing strategy? That’s also something we use, a unique approach. Robert [28:34]: Yes, it’s interesting because if you think about the abstraction over PubSub, how it works in Watermill, so basically PubSub is implementation of one interface with two methods. PubSub publish and subscribe. So you can guess that publish method is responsible for handling publishing of your message, and subscribe. Yes, you guessed. It’s for subscribing for those messages that you just published to your topic. And every PubSub implementation that we have basically needs to implement this interface. And at some point, when we’ve been thinking about how to test that and how to implement many of the PubSubs, we found out that actually, okay, if every PubSub should implement this one interface, basically every PubSub should work in a similar way. So your publishing message with publish and subscribing, and it should run back. Obviously, it may sound like a pretty simple thing. At the end, I think we have probably like 30 different use cases, or 30, 50, I don’t remember. Miłosz [29:41]: You can configure them depending if this PubSub supports them or not. The funny thing here is how you can adjust and adapt many different systems to implement this interface. Probably one unusual thing is the HTTP-based PubSub. So you can just do it over HTTP requests and responses. Robert [30:03]: It’s basically webhook. Miłosz [30:05]: Yeah. Robert [30:07]: We have also PubSub, for example, based on the file system. I mean, io.Reader, io.Writer. Miłosz [30:14]: So you can do it over files or whatever else you… We should probably build one over email. Robert [30:25]: Yeah, it would be cool. Miłosz [30:28]: But this goes back to this design of the API, right? If you choose this nice interface, the rest kind of falls in place automatically. And I see many libraries that don’t care that much about it. And you have these public methods or functions that take many arguments, and you have to check how they work before you start using them. And they could probably use some more high-level APIs that would be easier to use. So if someone is wondering if there’s still a place for some other libraries, I think there’s plenty. Robert [31:14]: And there’s one cool thing about having this one test suite for every PubSub that we support. This is the fact that if you’re implementing a new PubSub implementation, basically you don’t need to write a test because tests are already there. And to be super exact, we have implemented, for example, Google Cloud PubSub at some point without having access to any production system that was working on Google Cloud PubSub. But we know that, OK, it’s cool because it’s hosted. And we implemented it basically just based on test. And I think a couple months later, we joined a company that was using Google Cloud PubSub. And we think, OK, let’s try if it works with that. And we used this PubSub that we just implemented based on tests. And it just worked on production without, and a big adjust. So I think it was just some context cancelling or some really minor thing, but everything else, it just worked as expected. So it’s a cool thing because it’s ensuring consistency about multiple Babsops. So we can say that we have one standard of how PubSub implementation should work. And when you’re implementing a new one, it’s just much faster to do that, and even if you don’t have access to any system that works on this PubSub. Miłosz [32:29]: It’s just a nice exercise to try to bring your own PubSub implementation because of those tests. So now you have this nice system that tells you what the code you wrote is correct or not. Robert [32:45]: It’s a bit like a quiz. You have a quiz and you need to solve multiple challenges to make it working. Miłosz [32:54]: It’s also quite easy to do it, right? I remember the NATS PubSub we created. We did it over one day on a hackathon with a couple of friends. Yeah, because it’s just one interface. If you can implement it, then you’re done. Robert [33:10]: So if, for example, you are using some PubSub that we are not supporting in Watermill, well, we are open to accept contribution, obviously. But the most important thing, your tests needs to pass and be stable. If they’re passing and they’re stable, well, we can talk. But we’re also happy to help with that because we know that in some systems, debugging is a bit harder. It’s also useful to know how to use grep and pipe your logs to text files because sometimes… Oh, implementing PubSub may be not that easy when SDK is not best. Miłosz [33:49]: But it’s doable. Yeah, and you can start with the happy path. Because there are many different configuration options, usually. And once the implementation is successful, you have people asking for more features or changing the default behavior, stuff like this. But the HappyPath scenario is usually quite easy to do. So you can focus on this and later add the configuration. Robert [34:20]: There’s another thing that I think is also important when you are building a library that people should trust in. And this is some policy of how you are introducing breaking changes. For example, Watermill is since, I think, six years in the major version V1. So in other words, we didn’t make any breaking changes in the core library. We did some in some PubSub implementations just because some SDK was updated or we had some good reason for doing that. But in general we are avoiding that and in the core we didn’t make any breaking change so obviously it has a downsize because we need to even if we’re deprecating some methods we need to keep them backward compatible keep old functions still working but from another side it’s very, easy to update for every people and i remember that i was using some libraries that were doing some breaking changes without bumping a major version. It’s not the best experience. Miłosz [35:21]: Even with major bumps, people often tend to not migrate at all. Even if it’s just a couple of methods that change, it’s some effort. Especially if you use this library in some part of the system that’s critical, like messaging, or maybe it’s a part that is not used much, so you don’t know if something breaks. Then you are more likely to postpone upgrading. So, yeah, we want to keep the v1 promise. Robert [35:52]: And important disclaimer, so no breaking changes, it doesn’t mean only that the public interface doesn’t change, because it’s also about keeping behavior as earlier. Because I would say that it’s even worse, because if you are breaking your API in a way that the compiler will say you that, okay, it doesn’t work, it’s fine because you will see the compilation error and you can fix that. But if you’re changing your library behavior, but without breaking interfaces, it’s pretty bad because you may expect some, consistency in how it works. And in general, you don’t want to surprise your library users. Miłosz [36:35]: And probably the best way to avoid breaking changes is using configuration structs for the top-level methods. I think we had to bump the Kafka PubSub implementation to v2 very early, because we did more positional arguments and changing them is a breaking change always. Or you need to introduce a new constructor, which is annoying. Robert [37:03]: Yeah, I think we also made the similar thing with SQL PubSub because we needed to add more parameters in some interfaces that we have in that library. PubSub. And by the way, if somebody is surprised, you have SQL PubSub. So yeah, we have SQL PubSub. So if you are in the stage that you cannot effort, let’s say, to have extra infrastructure piece or Kafka, RabbitMQ or whatever, you can just use Postgres or MySQL. We have also benchmarks and we checked how fast it works. And for many systems, it will be more than enough. but it’s just a bit off topic. Miłosz [37:52]: On this topic, one more thing to add is to keep the V0.1 before V1 so you can experiment with the changes and people are aware this is an experimental version. But at some point, you probably want to release the 1.0 because it builds trust. If someone says your software is always in a beta release, they might not be eager to use it in the project they run. They will keep asking, is it production ready or not? But this buffer, which in our case was two years or something, quite a long time basically. But it was useful to experiment with different approaches, and then once we were ready to finalize the API, we could just froze it, and that’s how it stayed over time. And we kept adding more features, of course, more high-level components, but they all worked with the low-level API. Robert [39:03]: So the tactic that we often use is just keeping the old functions or old types that we had there, marking them as deprecated, and suggesting to use another one. So a good example is our SecureS component for handling events and commands. And at the beginning, we have some super weird, crazy, complex interface. Now, it’s still there, but it’s deprecated. And we have nicer API using generics now that you should use. But if you are using the old one and you don’t have time to migrate, okay, that’s totally fine. It’s still working as it worked five years ago. So I think it’s pretty cool. Miłosz [39:43]: And it wasn’t huge efforts to keep it backwards compatible. Most of the time, it’s not. You just need some deprecation commands and maybe keep some old tests in the repository. Robert [39:56]: I think the biggest challenge is often how to name the new version. Because often you use the good name already and deprecate that and have some good names. I think we didn’t do that, so we didn’t put any suffix like v2 or whatever, but in probably worst case, you can do it. I think we have also this with the protobuf marshaller that we have in Secure S component. So I think the old one was the protobuf and the new one is proto. Not perfect, but probably a bit better than v2 suffix or something like that. Yeah. Miłosz [40:35]: Naming and choosing a good API is one of the challenges definitely. Another challenge that’s kind of specific to Watermill is we support 12 PubSoaps right now. And most of those systems is quite complex. Like Kafka can be configured in many ways and used in many different scenarios. And we are not able to run all of those systems in production. We use some of them. So yeah, that’s one of the challenges here is to keep up with the community. So you have to keep it in mind that once the library starts growing and future requests are coming and you need to keep up and review them. And if you are not using some Pub-Sub, in our case, in production, it’s sometimes hard to tell if the changes make sense. Miłosz [41:41]: In general, maintenance is important, because if someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea. Robert [42:06]: Yeah, I think it’s often also challenging when your project is kind of feature-complete. Like, okay, I wouldn’t say that WaterMill is feature-complete because we have still many ideas what we can add there. But from the other side, the features that are there are feature-complete. I mean, we can probably just not touch WaterMill for the next 10 years and it will be enough. We maybe just can update other PubSub implementations, but that’s it. Miłosz [42:33]: It’s stable. Robert [42:35]: Yes, yes. Miłosz [42:36]: Yeah, but very often some security updates also make sense or library bumps, version bumps. Robert [42:44]: Or license changes. Miłosz [42:46]: Yeah, that also happens. So that’s one of the challenges. So it’s not only challenging to start, but then you also need to maintain it over time. Miłosz [42:58]: Maybe to change the topic a bit, we should talk about marketing. Which is probably not the first thing you think about when you consider starting an open source project. Robert [43:11]: Yeah, but as we already mentioned, running open source project library, it’s pretty close to building a product. So I think we’ve been lucky to already build a couple less or more successful products. And I think it also helped us to learn multiple things. But I think one of the most important things is that it doesn’t work. So just creating great product, great library isn’t enough for people to use that. So you need to let other people know that it’s great product, it’s great library, because it doesn’t work in the way that people will find it. No, they won’t. The chance is really, really low. Miłosz [43:54]: Yeah, and there’s no easy way to track how popular the project is. GitHub stars are one way, which can be a vanity metric. It doesn’t tell you very much. Robert [44:07]: You can buy them. Miłosz [44:08]: You can buy them, yeah. But it also gives you some social proof. And in general, if your library is used, people will be more eager to use it as well. I remember once, when we joined the company, at the time when we released the 1.0, I think. I remember we wanted to introduce it internally to other teams. And there was some kind of internal library like this before, but it was kind of limited and didn’t have a nice API. So there was this meeting of senior engineers and this kind of meeting where you have 10 or 15 people that gather to decide a thing in one hour Robert [45:02]: And it costs a lot of money. Miłosz [45:04]: Yeah it never ends with a decision it often ends with another planned meeting and I remember we drifted from the main topic and just kept talking about which library we should use and then one of the principal engineers starts reading mail on his phone because he’s bored and he finds Golang Weekly newsletter and water mirrors featured in there and he says out loud guys you are in Golang Weekly and somehow this ended the conversation it was like a proof that oh yeah this library I’m sure it’s worth something Which is funny, because it didn’t mean some company used it on production or whatever. Maybe they did, but basically social proof is something you need to succeed in open source too. That’s why often projects also add some logos of companies that use it to the readme or website. Because then you go like, oh, okay, this company uses it, so it must be a good library. But of course, they don’t tell you that this is used by some obscure testing framework this company uses or something like that. Robert [46:33]: Or it’s just used in some service that two people are using in this company. But, you know, it’s often like that. Miłosz [46:41]: But long story short, you need to promote it, whether you like it or not. And the obvious answer is where to do it? Is Hacker News and Reddit? What other places you suggest? Maybe some conferences? Robert [46:59]: Yeah, so this is the thing that, for example, I like to do a lot. Maybe some of you had the chance to see my presentation on some conferences. I really love to talk about watermelon conferences. Also on every bigger release that we have, we’re trying to write some blog posts, that we can share what’s new there. Miłosz [47:24]: Also the examples are one way to promote it sometimes we have more complex examples and some blog posts around it, like the one with servers and events which also features Watermill, but it’s not only about Watermill, it’s how to do something very specific in your application. Robert [47:47]: Yeah, and some of you may ask, all right, but I’m posting that and nobody cares and nobody’s avoiding that. And I think it’s often all about going back to the one of the first points that we have. So maybe it’s something that doesn’t resonate with people, because maybe they don’t understand how to use that, or maybe they don’t have use case for this library. And yeah, I think that this is part of why Water on You is that popular. It’s solving real problems that people have and is the key. Again, if people are not happy about using that, maybe it’s not solving problems that they have. Maybe you should try to find another idea that you can help people with a form. Maybe people don’t understand what this library is doing, what problems it’s solving. Quite to focus on use cases, on emphasizing what problems the library solves. And it will help a lot when you’re posting it to Hacker News or to Reddit. Miłosz [48:46]: I was browsing GitHub today and I found a library with README and there was a big banner that says share your ideas for your use cases so I think that might be a sign you don’t know why this library exists so if you can come up with some real world examples featuring your library, that’s probably a good sign yeah, Robert [49:14]: So watch out for looking for problems for solution that you have, because often it may be not the best thing yeah, All right, so we’ve shared a couple things about watermills, what we believed it was successful. I hope that it will be pretty useful if you’re building your own project or maybe you’re looking for some inspiration. If also watermill and eventview application sounds useful for you, but you didn’t have any chance to build such kind of system and you don’t know how to do that, Obviously, we have our examples, we have our documentations, but we also know that just reading examples, reading documentation is not enough for learning. So we already covered it in the previous episode, I think. But if you are looking for something about event-driven applications, you’re pretty lucky, probably, because twice a year we are running our online training about building Go event-driven applications in Go. And actually we’ll be starting in next three weeks. And it’s also a pretty special edition because we have a lot of updates. So it will be V2 of Go event-driven training. Breaking change. Yes, but yeah, it’s exactly breaking. Miłosz [50:43]: What didn’t happen to Watermill. But sometimes it’s better to do it. Robert [50:47]: Yeah, yeah. So what we have made, we’ve gathered all the feedback that we had from people from previous editions. and we spent a lot of time to implement them, and we believe that it will be better than it was earlier. This is also the first time when we will be doing it in the cohort way, so we will go with the content over four weeks with a group of other people. So let’s see how this form will work. We believe that it will be much better. Miłosz [51:23]: And everyone who bought it before will have access to this new one, like we promised. So we get all updates for free. Robert [51:34]: Promises are important for us, so we’re keeping that. One of the promises is that when you’re buying at this point, it’s the best price that you have all the time. So you can join in-house here, probably more or less, but it will be more expensive. So it’s definitely we encourage to join earlier. And you have also lifetime access for that. So it’s not like you need to finish it with this cohort now. You can do it in your own time. When you have time, you can do it now, in one year, whatever works for you. Robert [52:09]: But yeah, it’s just available from time to time. It’s not available for the time. So if it sounds cool for you, just Google it. It’s named Go Event Driven. There’s a page about that. And then again, it will be available in three weeks. So stay tuned. Miłosz [52:26]: And actually, Go Event Driven is also a nice synergy with Watermill. Because we feature Watermill in the training. Because it’s about event-driven applications. And it lets us show how to build event-driven applications from a high-level perspective. But it’s not a training about watermill so we don’t make money on watermill directly but it’s this synergy where we have this incentive to keep improving watermill to make the training even better so I think it’s a quite unique example of how you can take an open source project and connect it to something commercial but also So it’s not about sponsorships, or changing the license to closed source, or maybe even running an enterprise edition. Which often doesn’t work, because some cloud takes your source code and does it cheaper. Robert [53:30]: And it’s also worth mentioning that we’re also not stripping documentation just to force you to buy the training. I mean, if you already build event-driven applications in other programming languages, Watermill documentation will be totally enough for you. You can go over the documentation, understand how Watermill works, and that’s it. Because some people may think that, oh, you need to buy training to understand Watermill. So it’s separate. And it’s also separate in the other way. So it’s not training about Watermill, it’s about universal patterns that works in every programming language. So we’re using Watermill because it’s making stuff easier, but we’re planning in the future also to create additions in other programming languages. Because again, there’s no training about Watermill. You can replicate it in any programming language and it’s universal knowledge that works now, worked 20 years ago and probably will work in 20 years. Miłosz [54:28]: Well, we will have even given AI systems. Robert [54:32]: Maybe AI will learn from our training. And as always, when we’re mentioning WaterMill, we always are sharing a big thank you to all contributors. We mentioned that we have more than 100. It’s hard to say because it’s across almost 20 repositories, and it’s hard to deduplicate and count them, but I think it’s more than 100 people across all repositories. Hard to say the exact number. But yeah, real thank you, because without you, Watermill wouldn’t look like you now, because we’re just two, we’re 100. So it’s really, really helpful. Miłosz [55:14]: Right. Let’s move on to the Q&A then. Robert [55:17]: Yeah, so if you have any questions about Watermill, about anything, just leave it on the chat, and we’ll be more than happy to answer them. Yes, I see that there is some comment about focus on me. Yeah, so we’re fighting with cameras to get the good settings, but yeah, I see. Probably it’s pretty good now, but light may be not perfect. Although, yeah, we are trying to improve every episode. Adrian is also asking, when creating a project, do you consider performance at the start of the project? Is it wise to make it run first, then think about performance later? Because performance plus design do go together right. What do you think, Niosh? Miłosz [56:14]: Yeah, good question. Robert [56:16]: It depends. Miłosz [56:19]: So for what I mean, we have benchmarks. and that’s kind of objective way to measure how fast it is. But we never explicitly focused on performance, because most of the time it’s not that critical. Even for the SQL PubSub, which is super slow, we calculated that you can process, I don’t know, a few million messages per day or what was the count? Robert [56:55]: Per hour probably, I don’t remember. Miłosz [56:57]: It’s slow but still pretty fast for most use cases and I think we never had anyone complain about performance but it’s a good comment about the design. I guess one example could be if the design didn’t support batch requests, that could be pretty bad. That’s probably something to keep in mind. And that’s probably different in open source projects than some internal libraries. Because if you finalize the API and then it turns out it’s a blocker for some performance, action, then it’s a problem. So, for example, the publish of Watermill Publishers supports multiple messages in one batch. I guess that might be one example how it can help. If all you could do was publishing one by one, that might be a big factor for performance reasons. Robert [57:58]: Yeah, but I think the most important thing is about thinking about use cases for the projects, because in some libraries it will just doesn’t matter and in some cases it may bother yeah. Miłosz [58:09]: Very often it won’t matter Robert [58:11]: Yeah for example if you’re building i know some ai stuff probably doesn’t matter because okay you can optimize every memory allocation order your struct fields to use as least memory as you can but when you just call lm it will be super slow and what you can do Oh. Miłosz [58:29]: I know. Good example. Very popular in Go ecosystem is doing benchmarks for HTTP router libraries. And each library claims they are the fastest. And they have charts where you see that they are, I don’t know, faster in nanoseconds than the other ones. But the truth is, if you use HTTP, then you communicate over a network and each request takes, I don’t know, 50 milliseconds or 100 milliseconds roundtrip. Then this micro-optimization is not that important really. Robert [59:14]: And probably you’re using JSON for communication. So come on, if you’re using JSON for your API and you’re talking about optimization, probably it’s not the issue. Miłosz [59:25]: Someone will now say they have a very specific use case for this and they need to process billions of messages per second. And yeah, maybe it’s important. But for many projects, it won’t be important. Yeah, I guess advice is keep it in mind in terms of API, but probably don’t focus on it as the priority for the first release. Robert [59:52]: So in other words, if you can optimize performance later without breaking changes, maybe it can wait for a while. Because it’s all about not introducing breaking changes. Like in Watermill, we can probably optimize a couple of things in terms of performance, but it will not make any breaking change, basically. Miłosz [1:00:13]: Unless maybe you want to create a similar library, like one that exists already but that is faster or makes zero allocations or something like this, then it might be useful. Robert [1:00:31]: All right, so the next question, when creating a wrapper around, say, GCP pops, what separates from lower-level implementation versus higher-level implementation? What typical feature should be included? Miłosz [1:00:58]: Higher level implementation. Do I get it right that it’s about higher level GCP pubsub features? So basically, the Watermill wrapper around GCP pubsub is this… You implement this low-level interface only. So you implement the Publisher and Subscriber, and then all other components of Watermill work with those two interfaces. So you get the rest working out of the box. I’m not sure if this is the question, but the typical features are basically publish and subscribe. That’s the short story, but there are often very a PubSub-specific configuration you need to include. Okay, probably one very often feature that appears in most is consumer groups. Some kind. Or in the case of GCP PubSub, this is subscriptions. So you need a way to configure those. I’m not sure if this is the answer. So if you have a Yeah, if you have a follow-up question, you can let us know if this is what you meant. Robert [1:02:24]: Yes, Adrian mentioned, true, at the end of the day, which HTTP package we use, it may not be that important. Yeah, it’s also worth mentioning that, you know, in this benchmark wars of HTTP routers or whatever, often the routers that are winning those wars are not fully compatible with HTTP protocol. It’s also worth mentioning. It’s cheating a bit, but for specific use cases, they are faster in it. But again, we are mentioning it very often that you can optimize allocations, struct, order, but just making one more SQL query will just kill all your optimizations. Again, there are cases when you’re not doing SQL queries, you’re just doing some CPU intensive operations. Robert [1:03:20]: In this case, it totally makes sense to optimize. And if you’re a big library, just think about value kpi in this case and you can optimize later. And I think it’s also useful to if the library is when performance is important, just share benchmarks with people because, if they care about performance, it may be good. It may be just using the Go benchmark that is built-in. For these cases, it may work. For example, in Watermill we have special benchmark, because we are not measuring it in go-run time. We are more interested in how many messages we can publish or subscribe in one second. Miłosz [1:04:01]: Yeah, I agree that the HTTP package is probably one of the least interesting things about the project I want to decide. I mean, I want a nice API and probably middleware support. But other than that, I want it to be like Watermill, so I can replace it with anything else if I want. Robert [1:04:24]: If you feel that maybe you need to optimize your HTTP communication, maybe use gRPC, for example, that will be faster if you really need. Again, optimization when you’re using JSON for communication is a bit… Yeah. Miłosz [1:04:42]: Yeah, I wouldn’t spend too much time deciding on the library, basically. Just pick something you know or you like, and that’s it. Robert [1:04:52]: Right, so it seems that this is the last question so far. So let’s wait for a couple more seconds for questions. Miłosz [1:05:06]: So probably a good time to remind you about the subscribe button or the five star writing button on Robert [1:05:15]: Podcast applications or even better joining our newsletter so you will be sure that will not miss an episode notification so you can have notification before life one week before or summary and you know when you’re clicking subscribe it’s also cool but you are never sure that your notification will be there with our newsletter you are always sure plus you also know what all other updates that are doing uh what else so yeah five star reviews uh share with your friends so it would be also cool when this knowledge can go wider and if. Miłosz [1:05:55]: You have any open source libraries share with us in the comments so we can take a look Robert [1:06:01]: Or if you’re using watermill production also please let us know in comments it would be cool to know if it’s useful for you because again we don’t know basically we just know from some people that we’ve seen on conferences but, we don’t have any telemetry or whatever so we don’t know if you are using watermel. Miłosz [1:06:22]: Right i think no more questions thank you everyone for joining us today Robert [1:06:27]: Yeah thank you Miłosz for today. So see you in two weeks. In two weeks we are discussing what’s the difference between sync and async architecture. When we recommend to use that, we’ll also go deeper into the topic what’s the difference between PubSub and Message Queue. Miłosz [1:06:46]: So wehave a very related topic in two weeks. Robert [1:06:50]: Thank you for being with us and see you in two weeks. Miłosz [1:06:52]: Bye-bye.

Quick takeaways Simplicity isn’t enough for complex applications - while Go’s syntax is simple, complex applications still need proper design patterns; primitive code easily becomes spaghetti code in large projects. Reading the standard library isn’t the best way to learn Go - it’s optimized for different goals than typical applications and might be confusing for beginners. Router libraries are better than the standard HTTP package - libraries like Chi or Echo come with a nice high-level API. Struct-based configuration is better than the “optional pattern” - structs are easier to document, discover, and maintain than the popular With-options approach. There’s no one best project structure - starting small and evolving your structure as needed is better than following a dogmatic approach like the unofficial “Go project layout.” Writing stubs by hand is better than using mocking libraries - manually written stubs are easier to debug and encourage better interfaces than reflection-based mocking libraries. Code generation is better than reflect - for ORMs or dependency injection, it gives you compile-time checks and better performance. Generics are mostly useful for libraries, not application code - while everyone waited for them, they’re rarely needed in typical service-level code. Channels and goroutines can be overused - they add complexity and should only be used when concurrency is actually needed, not as a default approach. Go’s error handling is fine for most projects - explicit checks make code easier to read, though built-in stack traces would be helpful. Memory optimizations are often premature - micro-optimizations waste time for typical API services where network latency is the bottleneck. Introduction In this episode of No Silver Bullet, we share some of our unpopular takes on the Go programming language. After working with Go for eight years on all kinds of projects, we’ve seen many discussions about what idiomatic Go means. We talk about what worked for us, but we keep in mind that different projects have different needs. We question some common Go beliefs and share tips we’ve picked up along the way. Notes Wild Workouts - Our example Go project that shows a more complex application structure Watermill - Our Event-driven application library for Go HTTP Routers we recommend: Chi and Echo Go in One Evening - Our hands-on Training for learning Go quickly Clean Architecture episode - Previous episode that goes deeper into project organization Go Developer Survey Results - Shows that ~75% of Go developers build API/RPC services Google’s Go Style Guide - Many useful ideas, but be careful about being too dogmatic about it Quotes Go is simple, but sometimes it will end up as a disaster if you write too simple code for complex applications. You can write simple code, as long as we are building a simple application. But for something more complex, it starts to be more tough. Robert The common question in the community is ‘how to learn Go the best?’ One common suggestion is to read the standard library because that’s where the idiomatic Go code lives. But the standard library has completely different goals than a web application or API server or CLI application. Miłosz The biggest risk about having strong opinions is starting to be dogmatic and hard to be convinced by somebody else. We’ve seen people that were very strongly opinionated, and when you try to convince those people, it was super hard, even if your arguments were pretty logical. Robert This podcast is called No Silver Bullet because we believe there are no easy answers. To grow as an engineer, you need to keep an open mind. You can’t get stuck on a single fixed point of view because things get updated very fast What works in one context may not work in another. Miłosz When you create your application code, you definitely should not start with something generic. When you start with something generic, it will be over-abstracted and much more complicated than it needs to be. After a year, you’ll find out it didn’t need to be generic because you just have one use case. Robert Universal patterns exist for some reason. Probably someone wanted to solve some issue before. Design patterns are especially problematic - many people have this allergic reaction when you create a strategy pattern in Go and some people might react, “Oh no, this is not Java, please.” Miłosz Is structuring your project important? Ask a question: Will it help? Maybe it will not help. Maybe it doesn’t matter. Robert If the network roundtrip takes 100 milliseconds, why would optimizing the nanoseconds matter? Most of the time it’s a waste of time. Miłosz Timestamps 00:00:00 - Introduction 00:05:20 - Simplicity is not enough 00:10:55 - Don't read the stdlib to learn Go 00:15:38 - net/http vs other libraries 00:18:44 - Avoid the options pattern for config 00:21:48 - What's the best project structure? 00:34:38 - Avoid mocking libraries 00:38:53 - Code generation is better than reflect 00:44:32 - Generics are overrated 00:48:03 - Channels and goroutines can be overused 00:53:10 - Error handling is fine 00:58:58 - Memory optimizations are often premature 01:03:54 - BDD style testing libraries aren't helpful 01:09:15 - testify is useful for asserts 01:12:58 - Interface naming conventions 01:15:26 - Q&A Transcript Miłosz [0:00]: Do you know what is the best project structure? Should everyone read the standard library? And will error handling in Go ever improve? Miłosz [0:07]: Go may be a simple programming language, but there’s a never-ending debate about what idiomatic Go means. Today we discuss our takes on some of the most controversial topics. I am Miłosz. Robert [0:19]: And I’m Robert. And this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We spent almost 20 years working together across different projects and teams. We have seen Go evolve from its early days to what it is now. We learned that following advice like always do X or never do Y doesn’t work and it can limit your growth. In this show, we share multiple perspectives that will help you to make smart choices and grow into principal engineer level. Miłosz [0:48]: And depending on your background and experiences, some of the opinions we share might seem as popular or unpopular. and this is fine. We share what worked for us before. So keep in mind, it’s our opinion, but let us know in the comments what you think, what’s your take on the topics and share your questions for the end when we will have Q&A and if you have any controversial opinions of your own, that’s great. We can discuss them together. Robert [1:24]: Yeah, totally. would love to hear your unpopular opinions and include it to this episode. Robert [1:31]: And yeah, probably it’s worth starting why this episode is about Go. So if you don’t know us for a longer time, Go is our language of the choice, let’s say. So I think it’s the first language where we stayed for longer and we feel that it’s, we feel productive with that and we’re quite happy about this language. But as every language, our technology it’s not perfect and there are some things that are not that Miłosz [1:58]: Perfect it’s probably quite common story I’ve heard it a few times before but someone was moving from one language to the other and they stumbled on Go at some point and just stayed and maybe the reason is it’s a nice, get stuff done language so you know you can focus on what’s important and not about and the language specifics or features and so on. Robert [2:26]: Yeah, but it’s definitely the best one from the worst. If we assume that every language has some downsides. But I think also probably many things that we’ll cover today should apply to other programming languages. So if you’re not writing in Go, don’t leave us, because you may also learn something that maybe can apply to your other programming languages. Miłosz [2:51]: And we’ve been working with Go for about eight years now on production professionally. So everything we talk about, we base on this experience. And have we seen it all? Probably not. But we’ve seen different kinds of projects and teams and companies. So basically, the rule is simple. If we say that something works for us, It means we’ve seen it work in a few different contexts. But it doesn’t mean it is the same in every other context. This podcast is called No Silver Bullet because we believe there are no easy answers like this. Once again, show your perspective. Maybe there’s something we consider as a good practice and you’ve seen it doesn’t work. So that would be great to hear. Especially to grow as an engineer, I think you need to keep an open mind. You can’t get stuck on a single fixed point of view because things get updated very fast. So what works in one context in one project may not work in another. Robert [4:13]: Yes, I think it’s important to be open-minded and not be dogmatic because I think the biggest risk about having strong opinions is starting to be dogmatic and hard to be convinced from somebody else because often we’ve seen people that were very, very strongly opinionated about something and when you try to convince those people it was super hard, even if your arguments were pretty logical. Miłosz [4:40]: And on the other hand, there are people who start writing in Go and they worry that they are not doing it in the idiomatic way, which can be also a trap because it’s hard to tell what’s idiomatic without the full context. So it may be also helpful to hear different perspectives today. Robert [5:02]: Yeah, and if you would like to, we already covered this topic pretty deeply in, I think it was first episode about low quality code. So please check this one. So we discussed more about this, what’s worth doing Robert [5:15]: and when it’s worth doing shortcuts. Okay, I think we can start with the first unpopular opinion about Go, and it’s that simplicity is not enough. So one of the selling points of Go is that it’s a simple language, so some people are going pretty far with that and saying that, okay, Go is simple, you should write simple code. Robert [5:38]: I kind of agree that I think we should write as simple code as possible, but no simpler, because usually it will end up as a disaster. So some examples that we’ve often seen was trying to build some complex applications with some complex domain logic, like a financial application, like a crowd. And okay, maybe it will work for a while, but after adding more and more logic, it will suffer a lot. And it’s not that language specific, I think, because, okay, we’re now talking in context of Go. Again, a lot of people are saying that it’s simple language, but I think it’s something that is happening in every language. So Go have pretty simple syntax, that’s correct, but if you are writing complex logic, it doesn’t matter if you are doing it in Go, in PHP, or Java. But if you approach it in a too simple way, as a crud, it will just start to be overcomplicated. Miłosz [6:37]: Maybe the reason Go is more prone to this is people come to it from other languages, maybe some very complex projects where someone went too far with patterns and it was all overcomplicated. So when you start to write this simple syntax, it feels great, like a fresh air. So that’s pretty cool. But you can also go too far and try to do it. Trivialize it, write primitive code, let’s say. Robert [7:10]: Yes, like none of design patterns are useful, just write simple code and it will be simple and nice. And that’s true, as long as we are building simple application. Because for simple application it will totally work. But for something more complex, it’s starting to be more tough. Miłosz [7:28]: Design patterns are especially problematic. I think many people have this allergic reaction when you create, I don’t know, a strategy pattern in Go and some people might react. Oh no, no, this is not Java, please. Robert [7:47]: Don’t make Java in Go. Miłosz [7:51]: You probably don’t want to do it if it’s too early, but if you have an else-if structure that goes to 20 cases, maybe you could refactor it into a pattern that’s easier to read and maintain and so on. Robert [8:06]: Yeah, totally. I would say, again, it’s important here to not be dogmatic. It’s important to be mindful, let’s say, know those patterns, not say like those patterns are stupid and obsolete. It’s not the case. I mean try to go with simple code, but know what to do if you have some more complicated cases to tackle. Miłosz [8:27]: The universal patterns exist for some reason. Probably someone wanted to solve some issue before. Robert [8:34]: And they were doing it like 50 years ago. Miłosz [8:37]: In a language that’s not so far from on-go looks like right now exactly it’s still the same paradigm i think if you compare java a few years ago we’ll go it probably it’s not that far with syntax so yeah it’s a similar case, Maybe besides patterns, what comes to my mind is constructors. So I know some people also don’t like creating them in Go because, yeah, you can create just a type with proper zero value. And it often works well, sure. But sometimes you want validation. In web applications, very often you want the most entities. Robert [9:22]: Yeah, and if you’re building, again, simple application like your team, if it’s three people, five people, okay, it’s totally fine. You can live without constructors. You can watch out to not break things. But if your team is 10 people, 20 people, 30 people, it’s starting to be critical because anybody can actually touch your code. And you need to have some way to ensure that this person will not break this code. Miłosz [9:49]: And this validation will be in some place. The question is, do you want it in one place behind a constructor or do you want it scattered somewhere in your handler? Robert [9:58]: And this is a good example. I mean, simple code to… People that love simple code may say that no, no, no constructors. And I know that it may sound crazy, but unfortunately it’s what sometimes we’ve seen. Miłosz [10:15]: So simplicity won’t save you without the proper design. It’s possible to create spaghetti code in Go as well. Robert [10:26]: Exactly. I think it’s an important point that Go is not different in this case. You can create spaghetti code in every language. Maybe in some languages it’s a bit easier because you can write obscure code. In Go it’s a bit harder to do. But again, you can still have code with very big cyclomatic complexity like City, and it will be also unreadable. Miłosz [10:54]: Okay my next take is about ways to learn go and it is a common question in the community how to learn go the best and there are many many options but one of the common suggestions i’ve seen is, you should read the standard library because this is where the idiomatic go code lives, And I guess it might be because in the early days, there were not many open source libraries. It was hard to see some real life examples of Go. But I think for learning, it is probably not the best way to learn it. Because the standard library has completely different goals than a web application or API server or CLI application. or whatever you create. Robert [11:49]: Yeah, probably you can compare it to advice like if you would like to learn driving a car, you should go to Car Factory and learn how to drive a car. Miłosz [12:00]: Yeah, I would. That’s a good analogy. Unless you want to create a standard library for your own language, that is a great choice. Robert [12:08]: Or your own car. Miłosz [12:10]: And there are many great ideas in standard library, of course. So it makes sense to check how something is done. But I think if you are just starting out, it might be confusing because some of the code is not that readable. It’s optimized to be fast, to be generic, to cover all possible use cases. It has to be forward compatible because Go will never break the API, I promise. And, yeah, it’s probably not what you want to learn first. And one example might be that, speaking about constructors, I think the standard library quite rarely uses them, which might be fine because the zero value has some same defaults. For example, the default HTTP client works like this. You don’t need an instructor because the zero value is fine, which is a cool design. But sometimes you need to validate the inputs, basically, and it’s something you want in your application, probably. So I will be careful with this. Robert [13:31]: I think the thing that you mentioned a bit earlier, the promise of backward compatibility, it’s making it a bit harder. So, if you’re making constructor, it’s harder to add another parameter because it will be a breaking change. So, we know it from the library that we are building Watermill, and sometimes it’s actually pretty tough to avoid some breaking changes. And yeah, if you’re doing a standard constructor, it’s just hard to add one parameter. But if you’re building web services or any kind of web applications, you basically can You can do breaking changes in your internal code because compiler will say that it’s a breaking change, it doesn’t compile, you can update all the code, but it’s not a standard library. Miłosz [14:20]: So what to do instead? There are tons of resources and books, two of Go, documentation, videos and so on. We are a bit biased because we like learning hands-on, which we mentioned in the previous episode. So if you want to learn or check it. Miłosz [14:38]: We also have the going one evening training where you can use hands-on approach to learn by writing code. Robert [14:47]: And we know that a lot of people don’t believe that Go, you can learn Go in an evening, no way. But yeah, you can do that. Miłosz [14:54]: Yeah, but if you like to just read instead, there are tons of good resources. And there are many, many open source projects right now that you can take a look at how they work. Robert [15:04]: We actually have R1s also, Wild Workouts. It’s a pretty popular Y, I think, on GitHub. I don’t know how many stars it has now, like 5,000, 6,000, something like that. Also worth checking because you can learn from a project that looks like a project that you will do in your job probably as long as you’re not building a language or libraries and you can just see how to do it properly and not learn from your mistakes but from our Miłosz [15:36]: Staying with standard library there is this opinion that the standard HTTP package which NetHttp is quite powerful and good to go compared to other languages, which I agree with. But I would also add that it is not that great to be used by default in most projects when you start. Because the API is not that great to work with compared to router libraries, and it doesn’t give you much to stick to the defaults, unless you care about dependencies size or something like this. So what we like the most is either GoChi or Echo, which gives you a very lightweight router out of the box. And Qi is compatible with the standard library. Echo is not, but gives you more sane error handling. Robert [16:41]: It’s actually pretty compatible. I mean, it’s not using totally different types. You still have access to the standard library types. Yes, the HTTP handlers are not fully compatible, but you can have some magic adapters, let’s say. Miłosz [16:54]: Yeah, so in the very beginning, we stick to 3 because we wanted to be compatible with the standard library for some reason. I’m not sure why. Probably because it felt idiomatic to do it. And then we switched to echo because the error handling was promising. Robert [17:11]: In other words, I think it’s pretty cool that you can return error from HTTP handlers. Because from a practical point of view, it’s a very common thing that you’re returning error from your HTTP handler. And it’s just easy to mess it out if you not return from HTTP handler. So if you have checking if the error happened, but you will forget to return. It happened probably to any person and it’s not perfect. Miłosz [17:37]: The error handling gets verbose otherwise. And I think once we try to do it ourselves, like create our own wrapper on top of G to have this error handling. But Echo does it for you, so that’s pretty cool. And maybe Barton to mention that those are just routing libraries, not full frameworks as some other. And we prefer it this way because it’s easier to replace or doesn’t interfere with the rest of your code. Just HTTP handlers. Robert [18:09]: So we covered it a bit in the episode about frameworks. But TLDR is that you can pretty easily replace that. So if you decide that, okay, I don’t like Qi or I don’t like Echo. You can switch to any of this and it will take you probably less than a day for a pretty big project. So that’s cool. And I will be not afraid to be vendor locked in. Miłosz [18:32]: Unless you couple it with your application. Hopefully not. Robert [18:36]: Yeah, but you can also do it with the standard library. You spend some time. Robert [18:42]: Alright, so next unpopular opinion from me is using the optional pattern for configuring things. So you know those with XYZ configuration instead of using struct. And there are a couple of reasons why we don’t like this pattern. So the first reason is it’s hard to find what options you can pass. So you need to go to file and… Miłosz [19:07]: Sometimes it has the same package as the constructor. Sometimes it’s a completely different one called options so you need to look it up sometimes. Robert [19:15]: You have two packages for example in cmp library for comparing stuff you have two packages where options are stored Miłosz [19:24]: Many libraries also use options package for keeping those options which is cool but we have a big project with many dependencies and in your ide when you type options it gives you 10 different libraries. Robert [19:40]: Yes, and I actually don’t have a good idea why this pattern is so popular. Miłosz [19:45]: Yeah, I’m the same. I think I’m missing something. Why is it that much used? Robert [19:52]: But long story short, what we are doing in, for example, Watermill, we are just using structs because it’s super simple. So you are just navigating to the struct. You see all the options that you can pass. You can add a method for set defaults, for example, so you can see what are the defaults. That’s it. It’s deadly simple, just works great. I’m not sure why it’s not the standard. I hope it will be, but… Miłosz [20:16]: It’s easy to find the documentation. You just jump to the structure. You have all the fields there. What I also like to do is an IDE. You can have this command, fill all. So you just create empty struct, fill all and delete what you don’t need. Exactly. It’s very easy to configure then. and zero values can have the same defaults anyway. So it’s not that different from using the with options. It’s also easier to conditionally change some configuration. For example, if you have some input parameter, depending on which you want to use different options or not. If you use the options, you need to create a slice of options first, and then append to it. And with config, you just set the value and that’s it. And so if you’re just some fun of using this pattern in the chat, please let us know. What are we missing? Because maybe there’s some use cases that we didn’t see yet. Robert [21:19]: I think we can just recommend using struct because again in Watermill we are using it with zero problems, zero backward compatibility issues. I think standard library is also not using this optional configuration pattern. Miłosz [21:34]: A good example where simpler is actually the simpler option. I mean, when it’s useful to use a simpler one, just a struct. Robert [21:44]: Yep, definitely. All right, so the next unpopular opinion is about structuring your project. And I think… Miłosz [21:52]: Oh, most common discussion. Robert [21:55]: Yeah, it’s a controversial thing. So I think… It’s the thing that structuring your project is the thing that a lot of people are often very dogmatic and blindly following some patterns that they don’t have idea why they are doing that. So I have one good example, the Golang standards project layout that a lot of people are following. Miłosz [22:22]: Renamed. Robert [22:24]: Yeah, yeah. And it’s quite funny because a lot of people are following that thinking that it’s official guideline, but it’s not. If you go to issues, there is one closed issue that Golang by trainers are saying that it’s not official. Maybe you should rename it or something because it’s misleading so this is one thing that i often found the other thing that is sometimes controversial is about using some more complex structure like clean architecture or hexagonal architecture because many people are saying no no it’s making java from go we touched it very deeply in the previous episode, I think, about architecture. Miłosz [23:10]: Two episodes ago, yeah. Robert [23:12]: OK. If you are interested, you can just search for it. Miłosz [23:17]: And this is also a common question. I think maybe because of this empty page syndrome, when you start a project, you have nothing in your editor. So you feel like you need some foundation in place to start. And many people try to find the best way to start the project, to organize it. And you don’t really need to do it yet before you know how the project will look like in the future. So we recommend starting small and not sticking to any particular project structure if you don’t need it yet. Because it’s easy to go too far and overcomplicate it before it’s already needed. There are some nice, maybe not patterns, but some tips people use, like the cmd package for keeping all the binaries. I think that one is pretty cool, because it’s very easy to find the entry points to your application. Unless you create a library, maybe. I would probably use it in most projects. Robert [24:28]: Yeah, it’s also good to remember that you can use internal package, So the package that you can only import from the same or deeper level of your application. So if you are building a library or some services that are shared in the codebase, for example, you have modular or monolith, it’s pretty useful to use this package. Miłosz [24:49]: Some people call it PKG or SRC. I don’t really care that much, but I like to have this separate directory in the project, Because very often you have other stuff like Docker or Dev or Tools or API definitions. Basically some other files that are no Go files in your project. Robert [25:16]: Yeah, and it’s starting to be pretty messy if you’re just keeping each of the directory. Miłosz [25:20]: Yeah, especially for someone who is just joining the project, it might be very hard for them to understand where is the Go code, There is some helpers, some not ready to go files. So whatever you call this internal package, I think is a pretty cool idea. And it helps to find your way around the project. Also depending on the project type it might be a bit different right for applications for cli apps for libraries you probably will have a bit different structure, so it doesn’t make sense to stick to one everywhere you don’t need to look for one perfect solution and. Robert [26:06]: I think it’s probably also not that important that as people are Miłosz [26:10]: Exactly going. Robert [26:12]: Into very deep discussions i mean it’s just structure it just directories you can change it pretty quickly that’s it Miłosz [26:23]: But there is one package structure i dislike and it is the flood structure and, maybe it is also because of how the standard library looks like So I’ve seen many projects use one level of packages, and it’s supposed to be simple. But I think this is exactly this issue where trying to be simple tends to be something that’s hard to grasp. And I recall projects that have the same level, have packages like app, user models, database, handler, all kinds of names. And all those packages import each other in different configurations. So it’s quite hard to understand what’s going on. And again, especially for a newcomer, you just land on this project and you have to basically read it end-to-end to understand what’s going on. So that’s one structure I would not recommend, probably. Robert [27:41]: And yeah, I remember that there are some people thinking that, okay, if you’re, I mean, that project structure can work for really simple services, and it’s fine. If you have really, really simple service, and it for some reason needs to be a separate service, it’s fine. But I think it can also lead to some kind of pathology when if you see that, okay, my service is too complicated to be in one package, I need to split another microservice. And it’s a trap because you start to have really granular microservices and will start to have more microservices than people in the team. Miłosz [28:20]: Jim in the chat mentions that maybe we should read Robert Martin’s packaging principles. So I think this is clean architecture, basically what we covered in the previous episode. And yeah, this is what we usually suggest for more complex applications. And in short, it’s just grouping the packages in a few layers. So it’s easier to understand where’s your logic code, where’s your database implementation, and where’s your HTTP handler. And even if you look at standard library, there are some grouping packages like encoding for JSON and other encoders and HTTP in net. So it’s a similar idea. Robert [29:04]: And I know that people are sometimes thinking that, okay, it’s actually not like that in standard library. In standard library, you have package per one thing. But yeah, I agree. that it’s totally compatible and it can be idiomatic in Go. It’s just about not going too far. Miłosz [29:27]: Another package that’s often called unnecessary or forbidden is the common package. Which is an interesting idea because it’s hard to imagine a project where there isn’t common code. So I guess the standard recommendation here is to just move your code from common to where it belongs. Sometimes you can but sometimes it doesn’t make much sense and I think it’s pretty fine to keep this common package for stuff that don’t belong anywhere and I don’t remember it being a problem except for where you keep your entities and logic there that is pretty much it’s. Robert [30:15]: Just you know this hard discussion what domain logic is and if it should be there or not. But I think what’s pretty useful here is thinking in terms like if you are building your common package, is it something you could publish and other people can use in totally different projects? Miłosz [30:38]: Yeah, that’s a good idea. Robert [30:39]: Yeah, this filter can be pretty useful for filtering out things that are your business logic of your application and are not. And sometimes also some people are recommending that instead of creating some shared code, you can copy it. And okay, sometimes it works, but it doesn’t work for things that should change together. Because there is sometimes some logic that should always work the same, and it will be not a problem when it will be replicated in multiple places. But sometimes you have common logic that you need to be sure that it’s working the same way across entire codebase. And if you forget to update it in other places, you may have a problem. Miłosz [31:26]: Yeah, so maybe the issue here is not having the common package in itself, but deciding what goes there. If you just put too much in it and too many domain-rated stuff, then it becomes a problem. Similarly, you will create a common library that all services use. So I prefer to use the smaller libraries. And similarly in a package, you don’t need to put everything inside the common package directly. You can use sub packages. But I think this grouping is quite handy. Robert [31:59]: And there is a related question. So if you have considered packaging by feature, not a technology layer. So I would say that it’s a different layer, it’s a different level of packaging. So usually in a more complex project, we have, let’s say, two levels of modularization. So one is by, let’s say, technology or layer that you have like application, interfaces, adapters, etc. And above that, you have like users, orders, okay, it’s probably a very bad example. Miłosz [32:32]: More like modules, maybe? Robert [32:34]: Yeah, yeah. Miłosz [32:34]: I’m not sure if this is not about vertical slice architecture where you slice the project by each feature. Which I tried to play with it i didn’t come up with anything that i liked yet so I can’t really tell if it’s a great idea maybe in the future but. Robert [32:57]: Uh you know also separating by um so i said that you know users orders etc it’s basically example because it’s actually in practice it often doesn’t work. Miłosz [33:09]: It sounds too easy, too naive. Robert [33:12]: Unfortunately, but I think the proper modularization is really close to separating by feature. You should separate your project by independent things that you can modify to add some functionality. So for example, you are modifying some functionality, you should not need to modify 10 modules for example for that. You should be able to do it in one module and it means that your modularization is proper. So I think that it can actually be close to that. Unfortunately, often it’s not the case. But from the other side, you don’t need to also have very granular modularization for simpler projects because it can just overcomplicate stuff. And I think I’ve… Okay, it’s hard to say because I’ve seen multiple projects where they’re too granular or they were… Well, they could be more granular because it was just building with the time with more features in one module. So, unfortunately, it’s sometimes hard to get balance here. Well, yeah, I think it’s something around that. So even if you have modules, probably it should be pretty close to features. Miłosz [34:19]: Yeah, probably. I will try vertical slides at some point. Robert [34:24]: Easier to say than do. Unfortunately. Robert [34:29]: Alright, so time to next Antipolar Opinion, if you don’t have anything to add to the previous one. Miłosz [34:35]: Let’s go. Robert [34:36]: So, if you are with us for a longer time, you may know that we are not big fans of mocking libraries in Go. And there are multiple reasons for that. The first reason is that most of them, or I would say all of them that we tried are just hard to use because of a lot of empty interfaces, a lot of reflection. So because of that it’s pretty hard to work with that because you can basically pass anything to the assert or to the implementation and it’s really hard to debug that. And our solution for that is quite simple and I know that it’s a bit controversial, but our solution for that is actually writing stops by hand. And some people are saying that, but okay, it will be super hard, sometimes interfaces are super big, it will take ages to write those stops. I think I would think about that in another direction. Like if you have big interfaces, it’s not a problem of your stops. It’s a problem of your big interfaces. Miłosz [35:39]: Small interfaces make it much easier. Robert [35:41]: Yeah, yeah. And we already in some episodes were covering that it’s a similar problem with DI. So if your DI is complicated, it’s not a problem of your DI. It’s a problem of your dependency graph, basically. So the similar situation is with mocks. and okay sometimes in some projects where you are adding some tests it’s just hard to write those stuff by hand okay it it may be fine to write the use some library that can generate it for you it’s quite fine but i would recommend to try to write them by hand and i know that for some people it’s not obvious but in this case you can just keep this stub implementation next to the real implementation so thanks to that you don’t need to copy this implementation over multiple places so it doesn’t require much more effort to use that than using real mocks and at the end it’s giving you much more flexibility and it’s much easier to debug and that’s nice and Miłosz [36:47]: I think it also encourages you to test your your code and not not the mocks What I don’t like about mocking is that many libraries give you those methods that you can test what method has been called and with what arguments and so on. And I think it kind of misses the point of testing because you don’t really care what some method, what some repository or client has been called with. You care about the end result. So I like to design those fakes like in-memory repository. That works kind of similar to the original one. It’s much simpler. And what’s nice about it is you can replace the implementation and then run the tests on a very high level and everything should work as a production. And you don’t need to care about what has been called, what hasn’t been called. If the end result is correct, then it’s fine. Robert [37:47]: Yeah, it’s the worst if you are changing some logic and you are running your unit tests and everything explodes because you called one method that were not called earlier and you need to fix 100 tests just because you called something extra. It’s a disaster. And it’s even worse if you’re using your mocking library in the wrong way and it’s not thread safe. So if you’re running some tests in parallel, it explodes and you have no idea which test is failing. So yeah, if you wrote your stubs, well, the problem doesn’t exist. And if you are looking for some inspiration how to do that, so on our blog you can check article named the Go libraries that never failed us, 22 libraries you need to know. I know that it doesn’t sound like it covers how you should write stubs, but it’s actually There’s one example there that you don’t need HF library for mocking. Robert [38:51]: Okay, another thing that I think it’s sometimes a bit controversial, especially for some people that are newcomers to the language. So this is the fact that in Go code generation is pretty popular, but I think it works pretty well. If you will take example of ORMs or dependency injection libraries. So it’s similar case like with mocks. So if you are not using code generations in many things like your ORM, you end up with a lot of empty interfaces. You will have no compile type check format. Miłosz [39:32]: You have no strong types in a language that has static types, so you lose some of the benefits of using Go in the first place. Robert [39:39]: Exactly. And for ORMs, it has also one important thing that you probably care about performance pretty much. And if your ORM is pretty heavy on reflection, it will just slow. You cannot overcome that. And if it’s generated, it’s just static and it’s fast. So it’s quite cool. Miłosz [39:58]: Also easier to understand. If you really like to know how it works under the hood, you can open the generated file and very often it’s quite readable. Robert [40:08]: Yeah. And a good example is dependency ejection. So for example, we quite like wire. So in our projects, we don’t use a library for dependency injection. We are just doing it by hand. But if you really need to use any of it, we recommend wire because it’s code generation based and you can just go to the generated file and see how everything is injected. So debugging of that is pretty simple. With reflection, good luck. No way. Miłosz [40:34]: Yeah, it’s quite hard to understand how Reflect works, especially in a library you don’t know. So it’s quite hard to debug. And one other thing, libraries using reflect tend to overuse i’ve seen is struct tags so struct tags are pretty cool idea for marshalling it doesn’t get in your way and you know it’s easy to specify, but for other things i think it’s questionable because it gives you this this magic, behavior that’s not compile time checked one example is validation libraries i’ve seen some quite popular ones that let you specify so many things there, like how long you expect a string to be. Basically, if you look at the syntax they let you specify in a strike tag, it’s like a completely different language. Robert [41:34]: I think when the fun starts, when you are using this kind of library and you are just not checking how long the string should be, It’s a fun start begins when you are checking something if some field is set. Miłosz [41:48]: Yes, they also let you do it. So you need to write tests for it, which you probably want to do anyway. But I think if you use Go for its static types and compilation and all this good stuff, It kind of misses the point to overuse Reflect like this. And ORMs are similar with how they let you specify field types and relations in struct tags and so on. Feels like a cool idea because you have this one model that describes everything, but it can be overused. One of the things I like Go for is that it’s easy to read. You can take a look at the code and you understand what’s going on. There is no magic behind the scenes like in dynamic languages, where a function call can trigger something you don’t understand because there’s some decorator used somewhere or something like that. Robert [42:57]: Yeah i have ptsd when you are saying about it Miłosz [43:00]: Yeah i remember such stuff from python um or. Robert [43:05]: Php you remember magic methods Miłosz [43:07]: Yeah so yeah so if you use go i would stick to the, classic code as much as you can and avoid to reflect entirely unless you really want to do it. I remember one project when we tried to create our custom tags and be smart and be smart and avoid writing code by hand and yeah it wasn’t a good idea in the end, because just digging through the reflection code is not fun at all. Robert [43:43]: Yeah so alternative if just write those ifs. It’s not that hard. Maybe it looks super explicit, but it’s not bad. Miłosz [43:53]: Or generate your code, which is also quite easy to do with Go. Robert [43:57]: But probably not for validation. I think for validation, I think it’s just a matter of a couple ifs that, again, it will look boring, but don’t be afraid of boring code. Boring code is not something bad it’s something good actually Miłosz [44:13]: Yeah and with you know with good ideas and with things like copilot becomes much easier to to write boilerplate so it’s one less argument it shouldn’t take you much time to do it anyway definitely all. Robert [44:31]: Right another thing that i hope it will be controversial because you may remember that one of the most anticipated features on Go were generics. Robert [44:41]: And you’re here. We have generics for, I don’t remember how many years. Miłosz [44:47]: Two years, probably. Robert [44:48]: Yeah, and well, how often you are using generics in your service code? I don’t remember when I last did that. Miłosz [44:59]: Yeah, not much. For some for some more internal libraries. I’ve used them. We use them in Watermill for CQRS handlers. We also use them in event sourcing library. That was cool. But I think for service level code, Probably not that much. Robert [45:23]: And I think the secret may be in the name. Generics. So like the name states, it’s for some things that are generic. So for example, libraries that we mentioned. But usually when you are creating your application code, you definitely should not start with something generic. Because often when you start with something generic, it will be over abstracted and it will be just much more complicated at the end than it needs to be. And probably after a year you will find out that it didn’t need to be generic because you just have one use case that works in exactly one way. Robert [45:58]: And even if you have two cases just copying this part of code and not having it generic it will be much much easier and it will make your code much nicer. But for libraries definitely it’s useful but I wouldn’t say that it’s that big game changer compared to how much people wanted to have generics in Go. And yeah it’s cool that we have it because the secure component in WaterMule, it’s cool because it’s making the syntax much nicer. I love it. But I would say that I don’t see that many more cases when it helps. And I’ve seen a lot of cases when it just makes code unreadable. Really, if you know some generic quirks, you can do such ugly code that nobody will have an idea how it works and how to make it compiling. It’s sometimes like when I play with TypeScript. I mean, I have no idea how to make it compiled, so I need to use compiled or whatever. But I’ve seen such kind of code with GoGenerics sometimes, so watch out for that. Miłosz [47:03]: So it’s a similar rabbit hole as Reflect. It can be useful, but it can be overdone. So you can find yourself trying for a few days to make it work instead of just creating something straightforward, but for stuff where you want, It’s something universal, that might be a good idea. I remember one thing on the service level we had was decorators for application level commands. So we had stuff like metrics, logging and tracing decorator. And it actually worked pretty cool with generics. Robert [47:40]: Yeah, but it’s also something that we’re able to do with code generation. True that it was a bit tough because if you are doing your own code generation, Unfortunately, it’s a bit hard to do, because there is some tooling, but it’s not that great, so… Miłosz [47:58]: Yeah. One more feature that’s not always needed and can be overused are Tunnels and GoRoutines, which are one more great part of Go that everyone loves, I guess. Miłosz [48:15]: And for running concurrent code, yeah, that’s a great idea. But sometimes I’ve seen projects where they were used for some unknown reason. Like instead of calling a function, you would have some worker in the background that would communicate with channels with some other function. And it may sound like a cool design, but remember that it also has lots of complexity, especially for stuff like synchronizing routines, closing channels, accessing some common memory and so on. So it can be also painful. And also the concurrent code is not always faster than the segmental code. Robert [49:05]: Yeah remember that running a routine it also does some allocations Miłosz [49:11]: And if your code is CPU-bound then you are limited by the cores anyway so it might not be the best idea. Robert [49:18]: But i think it’s similar case like with generics so when you are building libraries channels may be pretty cool to simplify your life and build pretty complex stuff. But in typical application code, maybe it’s not that often useful. Sometimes it is, but often you can use, for example, error groups to do some synchronization. It also works nicely. Miłosz [49:44]: And it has more high-level API. It’s harder to misuse. And with channels, it’s easy to have some unbuffered channel that makes you stuck at some point because no one is reading from it and so on. So you need more tests, you need to be more careful with design overall. Robert [50:07]: That’s true, but it can simplify live with libraries. like an example of WaterMule. So I think it’s a nice example because we’ve been able to use a lot of patterns from Go that make implementation much easier. But it’s a library. I think we should do an episode about WaterMule. Miłosz [50:24]: Let’s make it the next one. Robert [50:26]: Sounds like a good idea. And one more thing about channels is that if you are using it in your application, you need to also remember that they are not persistent. So, for example, if you are triggering some workflow from your HTTP handler to channels and your service may die, for example, now you’re using Kubernetes, you never know. The service may die in the middle of processing your request. Your operation will be just lost. So remember that channels are nice, as long as losing this operation because it’s in memory will be not a problem. And if it will be a problem, remember that something like Message Broker exists, like PubSub, like Redis, Miłosz [51:12]: RabbitMQ, Kafka. Or cloud-based PubSub. I often see people ask about Watermill and the GoChannel PubSub we have. So we have this PubSub implemented in a water mill where it communicates through Go channels, so we don’t need a separate infrastructure. And it seems like a good idea because you can just have it running with no other infra. But it’s also quite limited. It’s not the same as most PubSub are. We use it mostly for testing purposes or stuff like that. It might be tricky if you try to use it for some production-grade system, because there are some… Persistence is probably important, or another thing that comes to my mind is buffering. So if you use unbuffered channels or the buffer overflows, then your handler will hang. There’s many complexities you have to think about. Robert [52:17]: Yeah, and if you are not building that big application, like you are building a smaller application and you have persistent file system, you can also consider PubSub based on SQLite, because we will very soon have it in Watermill. Thanks, Dima, for implementing that. And I think it will be a pretty cool alternative for channels, because you can persist your PubSub with SQLite, that will be super fast, and you can just store it in your file system. And it’s a nice alternative. If you are going into a more complex system, just use some production grade. Miłosz [52:58]: Or use SQL-based PubSub, if you already have the database, you probably have. Robert [53:02]: If you’re in the middle, let’s say, of your pop-up journey. Miłosz [53:08]: Okay, my next one is will be either unpopular or super popular, depending on who you are. So my take is that error handling is completely fine. In go. The checks are boring to write, but they are great for reading the code, because you know what’s happening. And you can just use a compiler ID shortcuts to generate them. But one thing we miss is stack traces by default. That would be cool. Robert [53:43]: That’s true. Because you always have this code that has no raps, and you’re later spending minutes, hopefully not hours, to try to find from where this error went to where it went. It’s tricky part. for stackraces, well, it could be helpful here. Miłosz [54:02]: Yeah, I wonder if the next proposal will make it to the spec, because I’ve seen there’s another one for using some syntactic sugar. So, let’s see. Robert [54:16]: Okay, in general, for example, I’m super happy about the error handling in Go because it’s explicit. It’s not like with exception, you’re quite not sure what can be drawn from the bottom of the stack. And in Go, you’re sure that, okay, this returns error. Should I do something with that? Shouldn’t I do something with that? And it’s super strict, and I think it’s making implementations much more robust. Miłosz [54:43]: Yeah people complain about handling the errors but the other side is if you have function that returns no errors that’s actually super cool because you know it is you know pure function, and it doesn’t have side effects probably. Robert [54:59]: Or for example have some function that you’d know that you don’t care that it returns error so for example sometimes you have cases when you are converting string to boolean but for example you don’t care if it’s drawing error if it’s just false. Okay, it’s false. Whatever, I’m just ignoring this error. But you’re doing it explicitly. With exceptions, it will be a big chance that some exception will be thrown. It will go up to the stack to your handler and it will throw to person 500 just because you were not able to parse bullying. Miłosz [55:28]: Like a panic. What do you think about panics? Robert [55:31]: I love panics. So I know that a lot of people are like, don’t panic. Yes, yes, I know. But I think it’s not that easy. So there are many cases when you shouldn’t panic. You definitely shouldn’t panic in goroutine because i’m not sure if all of you know but if you’ll panic in your goroutine and you will not recover it your entire process will exit with non-zero code not the best thing in http servers not gracefully Miłosz [55:57]: Shut down but just exit. Robert [55:59]: Yeah yeah so it’s if you are using channels for your asynchronous operations it will be lost so it’s not the best thing. In general, if you’re writing libraries, for this reason, probably you should watch out when you’re panicking because you can just accidentally do this favor of killing service for somebody. And it’s also pretty hard to debug. So not bad. But there are some cases when I like to use panic. So obviously in some dirty code, it’s nice. So for example, in my IDE, I have shortcuts to do if error is not nil, panic. It’s pretty useful. But there’s one case when I really like panics and it’s a production code and it’s related to dependency injection. So if you’re injecting things and when running some constructor in your main, return some error. And I think it’s totally fine to panic in this case because it’s a non-recoverable situation. So if you’re not able to inject dependency, your service will not work. So it doesn’t matter what you will do at the end. And panic is nice because it will give you the stack trace and it’s making it a bit shorter. And I think in general, this is the situation when panics are good. So when you have some non-recoverable situation, basically. Miłosz [57:19]: Yeah, sometimes also for a better API. We have a function that’s very often called and handling the error there would be annoying. And there’s just one error you need to check inside it, but it’s basically a sanity check. It never happens, but just for completeness, you need to do this entity check, then I think it’s pretty fine to… Robert [57:48]: I no longer make errors like it should not happen because it always happens. Miłosz [57:54]: What comes to mind from our project is about getting some config value inside the function. Robert [58:03]: Yeah, and without this config, it will totally not work. Miłosz [58:06]: So in theory, the config library should handle this for you, but you never know. So why not just throw a panic in this case instead of making your API harder to use in this case? Robert [58:20]: Yeah, exactly. Miłosz [58:22]: But it’s an exception, not a rule. Robert [58:26]: Always having exceptions in Go. Yeah, but again, I think what’s important here, again, not be dogmatic, because I think a lot of people are dogmatic that don’t panic, never ever panics are evil. But I think it’s not like that. There are some cases when it’s fine but just be mindful when you are doing it. Miłosz [58:48]: And remember about recovers on many levels. Robert [58:52]: Yeah, definitely. Miłosz [58:56]: Cool. My next one is about memory optimizations or optimizations in general. So So, I fancy discussions like this, like knowing what makes more sense, what’s a better trick. One example is struct field ordering. So if you order fields in your struct in a certain way, then they fit better into memory and it takes less memory allocation. Robert [59:26]: And you feel so smart. I just ordered my struct field to use a bit less of memory. Miłosz [59:33]: Yeah, and there are many such tiny improvements you can make, like pre-allocating slices instead of appending in a loop, let’s say, if you copy a slice. Or using a buffer builder instead of appending to a string. like our, yeah, my other ideas. No. Oh, yeah, I know. The string with map that maps to an empty struct instead of a boolean. Robert [1:00:10]: I actually remember one more. So concatenation of strings with this function. I don’t remember the name. But to not concatenate strings with plus, but instead using this… Miłosz [1:00:23]: The builder. Yeah. And, you know, GoTest has this benchmark feature that lets you write two functions and compare them and you can easily see, like, wow, this is five times faster. But, you know, if you take a look at the Go survey, I think 75% of people write API services in Go or web servers or something similar. Robert [1:00:50]: And what’s the bottleneck in this kind of servers? Miłosz [1:00:53]: Yeah. So, you know, if the network roundtrip takes 100 milliseconds or 50 or something like that. Robert [1:01:01]: Let’s take one millisecond even. Miłosz [1:01:04]: Yeah. So why would optimizing the nanoseconds matter in this case? So most of the time it’s a waste of time. And of course, a huge disclaimer depends on what your application does. If you write some, I don’t know. Robert [1:01:20]: CPU intensive or memory intensive application. Miłosz [1:01:23]: It’s tough for microcontrollers or whatever, of course, it matters in many cases. But I’m talking about when you create an HTTP handler and then you spend time discussing this in a PR, whether this should be pre-allocated, sliced or not. And it’s probably a waste of time. Robert [1:01:41]: Or maybe you are also writing a service that is handling really, really huge traffic and your servers’ costs are important, it may be also the case. Because sometimes it’s the case. But again, from our experience, it’s a smaller percent. Miłosz [1:02:02]: But the difference is you optimize once you feel the pain. Instead of trying to predict that one year later we will have such huge scale and that we will need this optimization. Robert [1:02:12]: And I think it’s useful to actually use this benchmark in opposite ways. So if somebody is suggesting you like, oh, no, you’re wasting such much of memory or CPU. You can use benchmark and show that it’s just that small difference and compared to, for example, SQL query, it’s like nothing. Miłosz [1:02:34]: Yeah, maybe just add an index in your table and it will be much, much faster. Robert [1:02:39]: Undo one less SQL query. Or maybe if you’re calling some external service, maybe do it in Go routine if you can. and just do things in parallel. It will give you probably a couple orders of magnitude more and then optimizing some allocations. Miłosz [1:03:01]: A similar concern is about garbage collection often. And I’ve seen it most often in regard to games. Like this opinion that you can’t make games in Go because of garbage collection. But chances are you’re not creating a game or even matter. And it turns out it’s actually pretty fine. There are many games created in Go that are all fine. And most of the time you don’t need to care about it that much. Unless, of course, you build something that’s very specialized. But I’m talking about a very common scenario of creating API or HTTP servers. Robert [1:03:39]: So again, be mindful. Just not apply some dogmas blindly. Miłosz [1:03:48]: Yep. Robert [1:03:52]: So maybe one more thing about testing again. So, the BDD style testing libraries are not helpful, or I would say even more, they are harmful, for multiple reasons. So the first reason, I think all of them are not compatible with standard libraries. So they are kind of using some testing from standard libraries, but in general they are not compatible. So you need to learn some new tools, how it works, and your assets will be not compatible. The new people in the team will need to learn something new, and it will be just less readable. Miłosz [1:04:31]: They have nice outputs, though. That’s a selling point most of the time, I think. Robert [1:04:37]: That’s true. But I think one of the biggest arguments for using this kind of frameworks is that it’s making a nice structure of your tests and I agree. I think it’s nice that you have this given, when, then, and it’s nicely separated. But from the other side, you can achieve the exactly same concept in standard library. Because nobody is saying to you that you cannot split every of your tests to three parts. With given, when you are doing some prerequisites, in when you’re doing some operation and at the end you’re ensuring that everything, what should happen. And I think it’s, I’m trying to do that because it’s making tests much more readable. But again, I don’t need to have special library for that. I can just do it with the standard library testing. So I think the most important thing is mindset. And it may be actually a good idea to maybe try to use this kind of libraries to try to understand this mindset. But I would avoid to use those kind of libraries in your services. Miłosz [1:05:46]: The big promise is using Garukin syntax to collaborate with your product owner or maybe some stakeholders or whoever. Yeah. We tried it before. Robert [1:05:59]: Yeah. Miłosz [1:05:59]: But we never made it to work really. So the idea is that you can create this framework for someone non-technical, I guess, to create the test cases for you in this given window scenario. And then you can just run those tests. Miłosz [1:06:19]: But I have never seen it work out in practice. Usually it’s developers who write those scenarios anyway. Robert [1:06:27]: Yeah, and you’re just wasting a lot of time on doing the mapping because you need to have parsers for this text, you need to do a lot of mapping. And at the end, when developers are adding those I would say now three times more time because you need to support that many things. Again, you can have very similar concepts in your standard Go library code tests. And again, maybe somebody like your product owner will be not able to edit that, or probably your product owner will anyway not write this. And if you follow those principles, you can even sit with your product person and if you did a good job with writing nice helpers, it can be even readable with your help to this person. So I think this actually worked for us a couple of times, so you can see with this. Miłosz [1:07:20]: If it reads like in English, you can probably just show it to someone who’s not technical and they will be able to read it and suggest what’s missing. Robert [1:07:29]: So again, I recommend to play with that, to maybe understand those principles and see how it can look like and give you some inspiration. But from our experience, it never fulfilled both promises that it gave. Miłosz [1:07:45]: It was usually easy to start. And I think most examples in Garek can show you this very promising idea. You have this table with quite simple use case. We have something like, when I do this and do this, then something happens. But once you try to do something more complex with it, you need to add more cases and it starts to be kind of bloated. Hard to understand. Robert [1:08:15]: Yeah, because Gherkin works nicely for some really high level tests, like end-to-end tests. But often it’s not enough to have some HTML selectors in Gherkin because you need to match the element that you need to click to do something. And it doesn’t make sense at the end. So you just Miłosz [1:08:31]: Keep switching between the scenario and the test code to make it work together instead of just doing it in your tests. Robert [1:08:40]: And parsing is harder and harder when you need to use some special characters. Miłosz [1:08:46]: Yeah, well, maybe it’s worth to take a look and try to at least to inspire yourself to write tests in this given one then approach. That’s actually a pretty good idea. Robert [1:08:57]: Yeah, and I think I would be happy to see it much more often because it’s actually not as often used as it could be. Miłosz [1:09:09]: Okay, maybe the same with testing libraries. I see a few people quoting Google’s style guide which mentions that you shouldn’t use assertion libraries. But we actually like Testify for asserts and it’s pretty easy to use, doesn’t get in your way. Most it does what you want. Robert [1:09:34]: And I think we can probably also say that it’s industry standard. In most companies it’s just used, in many libraries it’s also used. Miłosz [1:09:46]: Yeah, and that’s the thing about following those style guides or official guidelines. I think they are helpful, there are lots of useful ideas there, but it’s also easy to take them too seriously and dogmatic. And you need to remember you are not Google. And the best example probably is how Go started with GoPath, which worked well for Google because they used the monorepo and so on but for everyone else probably almost it was painful and weird to work with Dependency management? Robert [1:10:20]: Why you need that? This is everything on our monorepo Miłosz [1:10:24]: It was this vendor, yeah I’ve never seen why using Testify would be a problem it gives you nice outputs mostly out of the box you can add some extra context if it’s missing. Robert [1:10:43]: I think of an argument is that it’s implicit, you need to learn how it works under the hood and I agree but from other side you will do it once But for something like equals, Miłosz [1:10:55]: We don’t really need to understand how it works. Robert [1:10:57]: It’s just equals, basically. And you have to remember that sometimes you need to use equal values, for example, if you’re not asserting exact types, for example. Miłosz [1:11:06]: Yes, there are some caveats, but maybe it’s one library where actually Reflect is fine. But it’s a testing library, so that’s a bit different. Robert [1:11:16]: It’s a library. Miłosz [1:11:18]: Yeah, you don’t need to care about compile time checks in your tests because it’s just testing your code. Robert [1:11:24]: That’s true, that’s true. So again, I think it’s good to have exceptions from… Miłosz [1:11:31]: Guidelines, yeah, and maybe make it work for your project, your team. And not be dogmatic. Yeah, agree on something together, which is also a nice idea to have some internal guidelines instead of just committing 100% to something external that maybe is generic and works fine in many contexts, but for your specific use case, maybe not. Robert [1:11:57]: And I think that there is one thing in Testify that I don’t like. So those are testutes. I’m not sure if it’s still valid, but last time when I tried to use that, testutes have one big bad limitation. so they were not able to run tests in parallel. So basically, your tests needed to run sequentially. And from my perspective, it’s a deal breaker because many of the tests just need to be able to run in parallel when there’s any I.O. or any slow operation. And without that, your tests will be just extremely slow. Miłosz [1:12:32]: Right, so we had to use the test main pattern. Robert [1:12:36]: Exactly. Miłosz [1:12:37]: But some years passed, so maybe we should take a look at it again. Robert [1:12:40]: Yeah, and again, sometimes it’s just a matter of running some helper function before or at the beginning of your test, and it’s fine, and using subtests from the standard library. And I think our last unpopular opinion, so it’s about naming. So you may read in, I think, Effective Go probably, that you should name your interfaces with suffix like error, like reader, writer, follower, user finder. Miłosz [1:13:21]: It makes sense most of the time, but sometimes not. yeah. Robert [1:13:25]: So for example if you have interface that have four methods it can lead to something strange Miłosz [1:13:33]: And i’ve seen people overuse it sometimes like you have a method called user by id and then the interface is user by idr come on can you just call it user finder or. Robert [1:13:46]: Repository i mean it’s just user repository. Miłosz [1:13:50]: That’s probably a very minor thing, but I’ve seen it overused sometimes. Robert [1:13:56]: That’s true. I think it’s again about this being mindful when you are writing software, and if you see that you have an interface named user by IDFIR, it’s something wrong there because it’s just hard to read it. Miłosz [1:14:12]: Yeah, exactly. It won’t tell you much about what it does. If you just stick to this pattern everywhere. Maybe just come up with something that describes it better, even if it doesn’t follow this idea. And my final one about naming is, I don’t care if you say Go or Golang, which for some reason is controversial. I don’t know. It’s like the common discussion. Someone says, how to do X in Golang? Someone comes in and says, first of all, it’s not Golang, it’s Go. I don’t know why. I don’t care. Everyone understands what it is. Robert [1:14:59]: That’s true. But I think it doesn’t help that language name is not that unique around everything. Miłosz [1:15:07]: Hey, it’s not the worst name. It could be called JavaScript. Does it make sense? No. Naming is hard. Everyone knows it. Just live with it. Robert [1:15:24]: All right, cool. So let’s see what questions we have on the chat. All right, so Jim mentioned that obscure is not spaghetti, and spaghetti is not cyclomatic complexity. And yeah, that’s right. I think maybe we mentioned that it’s close to be equal, but no, it’s not, I would say that it’s maybe some shortcut, but it’s often connected in some magical way that when you have obscure code and cyclomatic complexity is big, there is a big chance that it may spaghetti code but yeah it’s Miłosz [1:15:59]: Yeah i guess those are separate things obscure will be hard to understand spaghetti would be many side effects that you don’t see something similar but maybe jim you can tell us more what do you mean what do you mean by this uh but yeah there are many aspects for sure and this comes to this discussion about simplicity right so it’s it’s hard to, to decide what’s simple because of it, because there are many aspects. Robert [1:16:32]: Another comment from Jim is that dry is tension with loose coupling. Any removal of the application implicitly creates coupling that didn’t exist and may be a source of future problems. And it’s true, but sometimes things are coherent. So basically if things are coherent you would like to have them coupled because it’s kind of expected. So if you have one logic that is expected to work everywhere in some way in every place. Well, unfortunately, it’s coupling, but you cannot close your eyes and say, oh no, there is no coupling there and just copy it everywhere because we’ll have a lot of problems with keeping it up to date everywhere. Sometimes maybe it shouldn’t be some library code. Maybe it should be in a service. It’s hard to say because there are many examples, but I would say that I agree with that, but there’s sometimes also one more complexity Sometimes things are just coupled because they need to be coupled and it’s making your life easier. So I would say that coupling is not always bad. Coupling is sometimes coupling and it exists and we cannot ignore that. Miłosz [1:17:43]: Yeah, but if you overuse dry, you can end up with too much coupling. I think that might be the point here. And yeah, duplication is the best way to lose the coupling as a tool. We have a blog post about this as well, on our blog. And yeah, we’ve also seen that overusing dry is usually a bad idea, unless you know what you’re doing. Robert [1:18:10]: Yeah. And I think it’s also one tricky part here that we’ve seen a couple of times, that it’s not silver bullet that you can always get common logic in library and just use it everywhere, or you can move it to some services and service can handle that. So in other words, sometimes it’s better to have a service that is keeping this common logic and it’s good, and sometimes it’s better to keep it in the library that is shared across. Because sometimes we’ve seen migrations in 2A. So for example, there was some library, some common code, it was moved to the service, just to figure out that actually having a service handling that is the worst idea, it’s better to have a library that keeps this logic common somewhere. But again, it unfortunately depends. Another comment from Jim. So yes, the focus on testing should be almost always on results, not with internal functions or code. Miłosz [1:19:19]: Yeah, yeah, that’s why be careful with mocks and at least we prefer stabs or fakes instead of mocks for this. And black box testing. Robert [1:19:34]: Right. Strange combination of opinions. Love panics, but hate exceptions. Love explicit error returns, but one stuck traces. Miłosz [1:19:43]: I wouldn’t say we love panics. They have some uses. We don’t use them for normal errors. Robert [1:19:52]: I hope it didn’t sound like that. Miłosz [1:19:54]: That’s maybe the difference. Exceptions are just used everywhere in some languages instead of errors. And panics are just for the very exceptional for the exceptional cases not for errors yeah. Robert [1:20:11]: And fact races so I would say it’s a matter of pragmatism that well, error wrapping is cool but what you’ll do if somebody forgot to do that and it’s also not perfect that you need to do that again, co-pilot helps with that sometimes those messages are super stupid but the reality is that But spending hours on figuring out the wrap strings, it doesn’t make sense. Miłosz [1:20:38]: Maybe one of the issues where it shows is if you use a library that doesn’t wrap errors and you receive some obscure error from it, and then you have to look at the sources and find what it is and understand what’s going on. And what stack trace could help you. And it doesn’t need to be stack trace of all the levels, just of the most inner level, where the error originated. So you see the previous function calls. Robert [1:21:14]: Yeah, and I think I talked to somebody from Go team about the problem of stack traces with errors, and I think one of the problems was performance. Miłosz [1:21:27]: And that’s again the issue of standard library it needs to be universal yeah yeah probably in some projects it wouldn’t make sense to allocate memory for it so yeah that’s but. Robert [1:21:40]: Yeah maybe someday because from other side we have the runtime tracing that it has it and it’s super fast. The overhead is like 5%, 1%, I don’t remember. Maybe someday. Please, please, please. Mate of Clive saying that I feel the least thing dev use Go is building web services. I always find jobs for DevOps engineers. Miłosz [1:22:17]: So I mentioned the results of the Go developer survey in the official one. And if you take a look there, the top one is API servers, about 75%, and some other high percent for servers returning HTML and such. So maybe this means everyone writes web services, but they have no jobs. Robert [1:22:45]: Yeah, but probably it depends. Miłosz [1:22:47]: Yeah, I think this is an accurate observation that there are not that many jobs where you can see Go used for web apps. Robert [1:22:56]: For other words, how many Kubernetes operators you can write? Come on so I think it may depend a lot on the market but yeah our observation is kind of consistent with the survey that we mentioned that I think most of the jobs were around services but again I think it may be also depending a lot on your bubble so if you are more into DevOps bubble you may see more jobs postings actually Miłosz [1:23:26]: It made me think because it’s true that But I don’t see many jobs about using Go for web apps in general. So what do all those people write that reply in the survey about web services? What do they do? Maybe just some open source apps? I don’t know. But at least, yeah, maybe we are biased, but we mostly write to web apps in Go and API services. We don’t do any DevOps stuff. Robert [1:24:05]: And we didn’t struggle to find any web services, probably it’s better words. Again, but it depends on the bubble a lot. Probably it depends a lot on your local market. If you’re looking in the local market remotely, remote is different. Okay, and the last question so far. So if anybody has another question, we’re waiting for a while. And the last question so far is saying I joined kind of late to this podcast and I’m not sure if you mentioned about naming convention in Go using name variables with one letter or three as maximum I mean code should be readable I think it’s actually a pretty good one we’ll cover this one what do you think? Miłosz [1:24:50]: Yeah, code should be readable I agree, it’s hard to give some rule here here. Of course, short variables are nice if it’s used in a small loop or something like that. Sometimes it’s easier to read one word than four words together, but I wouldn’t have any strong role here like three letters max. That’s a bit crazy. Robert [1:25:24]: Yeah, but I think sometimes it may improve, my opinion, readability a bit, but I think it’s all about not overusing that. So sometimes when you are… obviously, if you have struct receivers, it makes sense because you know that you are within some struct and you have receiver of the struct, so you know that S is, for example, service or something like that and it will just make a lot of code much shorter but if you are using all variables that are at most three characters length it may be not readable so i think it’s all about the context so if you see that in this context you know this small shortcut what it means it’s fine but it’s all about not going to extreme i Miłosz [1:26:09]: Think there are some guidelines like this in effective go maybe or some other document where it’s like this recommended exactly as you said. Depending on the context, just longer or shorter variables. If it’s a long function, and you use the variable across it, many times maybe longer is better, so it’s clear what it does. Robert [1:26:29]: Yeah, so I think it’s about some common sense. So if you see that it’s not readable, just use the entire name. And I would say that in most cases, we’re using the entire name to make code readable, but we’re also not afraid of using shorter ones. Miłosz [1:26:42]: I don’t like rules for stuff like this, and a similar one is what I actually like from Google style guide is there is no definitive rule for maximum line length. It just says, you know, try to refactor if it’s too long, but it’s not something like you should use 80 characters per line. Because then you have this fictional, artificial rule that everyone tries to apply but doesn’t help anyone. It seems similar with short names. Robert [1:27:19]: I think we covered it in an episode about writing low-quality code. At the end of the day, ask a question. Will it help? Maybe it will not help. Maybe it doesn’t matter. Safety doesn’t matter. Whatever. if it’s good if you have enough time to think about such things you have a lot of time if you Miłosz [1:27:42]: See variables like A, B and C probably something is wrong we can probably improve it, you can also go to the other extreme and call everything by too long names which is also not readable that’s true, I agree that code should be readable And everyone has their own opinion about it, so of course that’s hard to tell. Robert [1:28:09]: Yeah, so I think this readable, it’s something in the middle. So you’re just not going like writing off your state code with one character length of variables. But sometimes you can also just use those short ones and it can really, for example, for receivers. Miłosz [1:28:32]: Yeah makes sense. Robert [1:28:33]: Okay so jim is saying that he appreciate that balance in your commentary too much of go community ime in my experience oh makes sense quite simple with simplistic thank you thank you to jim for your comments thank you Miłosz [1:28:49]: Jim i think that yeah the simple with simplistic comparison is a good one like simplistic might be too much simply is fine. Robert [1:28:56]: Yeah yeah definitely cool so we have no more comments so a couple more seconds for last comments if you have any yeah Miłosz [1:29:07]: So maybe we can mention if you know if someone’s listening to this and doesn’t know go yet or anyone would like to try we have going on evening training on our platform where you can learn by writing code. Robert [1:29:19]: And it’s not our opinion but some people are saying that it’s the best thing that they did online for learning stuff, so I don’t know. Hopefully. And yeah, it can be one evening because Go has simple syntax and we did really a lot of work to compress the most important knowledge that you need. So you don’t need to read the entire Go standard library to learn to Go code because it’s not needed usually. There are some things that are most useful. That we know from the teams that we’re running because a lot of people actually that joined those teams never worked in the Go earlier and they needed like one week to be totally proficient in Go. Yeah, yeah. So it’s cool with Go. Miłosz [1:30:08]: Yeah. Robert [1:30:09]: It’s not perfect language, but no language is perfect. And I think Go is closest to this perfect one. So I recommend it if you don’t know Go yet. Cool. OK, so no more questions. Miłosz [1:30:26]: Yeah, what should people do if they want to not miss the next episode? Robert [1:30:31]: So we recommend to join our newsletter for one reason. So Deadly Algorithm may not send you a notification when a new episode is going out. So yeah, our newsletter is the best way to be notified about that. But we’re still under the rule of Deadly Algorithm, so it would be cool If you can leave thumbs up, leave a five-star review in podcast applications, leave some comments if you have any questions who are answering and reading all the comments. So if you have some extra questions, we’ll be more than happy to help you. What else? Miłosz [1:31:14]: There’s one more comment. Let me read it. So is it too disrespectful to say that Go feels like Node.js? Robert [1:31:24]: It is. Miłosz [1:31:26]: When you start a project with them you start from scratch one index.js or main go and there is no structure from the beginning, yeah guess on how you approach starting out like there are some scripts in node i think that set up this scaffolding for you yeah that’s quite common but the approach yeah i think it’s pretty a sane approach. Start with one file and then see what happens. Start creating packages as the main guy becomes too big to handle. Robert [1:32:04]: If you probably compare it to other languages, like, I don’t know, there are some languages that you need to use some kind of framework to build application and you have a lot of portal plates to start with. In Go, it’s like with You can start with something super simple and extend. So in this case, it’s totally true, but yeah, I think Go is totally different in terms of typing system. So it’s much more strict, let’s say. And while you are maybe building some smaller projects, it’s not a big deal, but when you are working on much more complex projects, when more people are working in, it’s much, much, much easier. Not mentioning about concurrency primitives that, in my opinion, in Node.js and JavaScript are awful. Sorry, in my opinion, promises and actually running everything on one thread is a deal breaker. In Go, it’s better. So you have an abstraction of GoRoutines that is very easy to understand and think about. And it’s also super fast because it can utilize multiple cores. Yeah, it has more threads. Hurray! And it has also nice primitives to talk between threads, let’s say. There are no threads because this is something special in GoRoutines, but communication between them is super nice, especially compared to Python. Miłosz [1:33:33]: Easy to grasp, easy to read and understand. Robert [1:33:37]: Yeah, I think it’s also interesting to actually compare to Python because, again, communication between threads, I have PTSD and I’ve asked about that, not mentioning global interpreter log. Come on. Miłosz [1:33:50]: Yeah, so it’s hard to beat Go’s combination of easy of use and also how powerful it is. Robert [1:33:58]: Yeah, and what’s also nice is how lightweight it is, because under the hood, one goroutine, so it’s not one thread. Without going to the details, it’s using one thread for multiple goroutines, so thanks to that, it’s super lightweight. So, for example, you can spawn like 1 million, 10 million goroutines, and it’s very lightweight, and you can run it easily on laptop. Miłosz [1:34:22]: So there’s a follow-up. Yes, I meant the lack of structure. For example, in .NET Core, Spring Boot, you have an initial structure. So that’s about frameworks, I guess, more than languages. And we had an episode on frameworks as well, where we talked about this. I guess you can find some frameworks in Go too, where it will create some project structure for you. And maybe that’s why people are used to those structures, they start the project in Go, start with a single main Go and they are lost and they know what to do next. Robert [1:35:00]: Okay, if you are looking for some inspiration, so on our blog we have articles about clean architecture, how to dig pragmatically, and there is also this wild workouts project that we created. We’ll add link to this project in the episode description and in the episode summary on our blog. Miłosz [1:35:22]: All right, thank you for the questions and thank you for joining us. Robert [1:35:27]: Thank you Miłosz. Miłosz [1:35:27]: Leave a comment, let us know about your controversial opinions. Robert [1:35:33]: In two weeks we’ll talk about Watermill as we promised a bit earlier. So Watermill, we mentioned this library already a couple times in this episode. So this is an event-driven, not only event-driven, but mostly event-driven library for building event-driven applications in Go. It seems that this is the most popular library for building this kind of architecture in Go, and I think it has now more than 8000 stars on GitHub, and I think it’s pretty popular and pretty successful and we’ll share some magic stories about watermills and why we believe that it’s successful. Miłosz [1:36:14]: So if you plan to start an open source project, it might be interesting for you. Robert [1:36:20]: And after that we’ll talk about sync versus async architecture. So connected to watermills. Miłosz [1:36:30]: Where one even different architecture shines and when it’s better to keep to the standard synchronous operations. Robert [1:36:38]: And it will be over-engineering. See you in two weeks, see you in four weeks. Thank you very much. Miłosz [1:36:43]: See you then. Thanks, everyone. Bye-bye. Robert [1:36:45]: Bye.

Quick takeaways Focus on applying what you learn - reading books or watching videos isn’t enough without practice Build real projects that challenge you - trivial examples don’t expose you to the hidden complexities you’ll face in actual work Expect and embrace frustration - feeling stuck often means you’re learning something valuable Learn timeless concepts over framework-specific details - aim for universal software skills like modularization Mix theory with practice in small chunks - read a bit and code a bit, rather than consuming large amounts of content at once Introduction In this episode, we discuss how to learn effectively as a software engineer. Why some people seem to learn faster than others? What are some practical ways to speed up your learning? Instead of promising magical shortcuts to becoming a principal engineer in months, we focus on a more balanced approach that helps you build skills by mixing theory with practice. Notes We mentioned our learning platform Domain-Driven Design was referenced throughout - check “Implementing Domain-Driven Design” by Vaughn Vernon The Pragmatic Programmer by Andrew Hunt and David Thomas is a great book for timeless software skills Watermill - our open-source library mentioned as an example of a project that taught us while helping others: github.com/ThreeDotsLabs/watermill Event-Driven Architecture traces back to 1950s, but was formalized about 20 years ago The Repository pattern blog post Quotes Just going fast is a trap, because you can go fast in the wrong direction and spend years learning things that are useless in practice or maybe are useful in the short term but not helping you during your entire career. Robert Once you tackle building something more complex that you’ve never built before, or maybe never even seen before, that’s when things get interesting, and that’s where your skills are tested. Miłosz I learned the most with the hardest projects that I worked on. The projects that were easy, greenfield, and everything was known - okay, it was nice, fun, but I wouldn’t say that I learned that much at the end. Robert If you feel frustrated while learning, you are probably onto something. Because you are frustrated that you can’t make it work. And then it also motivates you to push through. Miłosz What helps to learn much faster is trying to focus on things that are timeless. You can get benefit from the compounding knowledge over time. Robert Until you start fighting the obstacles, struggling to make it work, you probably won’t learn much. The hard part is figuring out what to build that’s a bit more advanced and will test your skills. If you pick up an idea and think ‘I have no idea how to do it’ - that is probably a good direction. Miłosz What also helps is sometimes feeling as the dumbest person in the room. Work with people that are more experienced than you. And if you sometimes feel that you’re not the dumbest person in the room, maybe it’s time to try someplace where you will be. Robert If you work alone, you don’t need to care that much about code readability. If you can fit the entire context of what the application does in your memory, you can use a single file that’s 5000 lines long. But as you work with someone else, it starts to get more complicated. Miłosz Timestamps 00:00:00 - Introduction 00:05:13 - What is real learning? 00:12:10 - Theory vs practice 00:15:29 - Learn by building projects 00:27:33 - AI for learning 00:36:19 - Is learning supposed to be fun? 00:42:22 - Learning from social media 00:50:17 - Timeless skills 00:58:11 - Diverse skills 01:04:12 - Making notes and teaching others 01:08:36 - Mentors and communities 01:11:54 - Q&A session Transcript Robert [0:07]: Ever worked with someone who’s been coding for more than 10 years but still makes mistakes? Or have you seen a junior engineer write code that blows your mind? It’s not always about intelligence. The real secret sauce is how you learn. Learning the wrong things can feel productive, but it won’t get you far. In this episode, we’ll explore the art of fast, effective learning. I’m Robert. Miłosz [0:27]: And I am Miłosz, and this is the No Silver Bullet live podcast, where we discuss mindful blackend engineering. We spent almost 20 years working together in different projects and teams. During that time, we learned that following advice like “always do X” or “never do Y” doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices and grow. Robert [0:55]: Yeah so if you have any questions please leave them on the chat we’ll try to address them on demand if not we’ll have Q&A session at the end of the live so we can answer other questions, anynthing to add? Miłosz [1:11]: Good let’s go Robert [1:12]: All right so I think it’s important to start with setting expectations because sometimes you may heard from many people that there are some magic way to learn faster you can be principal engineer in one month or maybe half year but well unfortunately it’s not that simple so everything what we’ll mention in this episode wouldn’t be advice like that it wouldn’t give you any magical power to be principal engineer in one year but definitely it can accelerate your car and it should definitely help you with not staying junior or mid engineer for four years because i think everybody probably met such person during working with software that as we mentioned at the beginning this was mid this mid engineer for forever yeah. Miłosz [2:04]: And magic recipes usually don’t work, and often means someone wants to sell you something, so watch out for silver bullets. Robert [2:16]: But still, sometimes there are people that learn really, really fast. And today we’ll share a couple hints that, from our observations, made them very effective learners. And again, it’s not about being smart, because from my experience, sometimes being smart can be even a trap, because you may try to be too smart and try to maybe dig too deep to some things, but it’s not always the best idea. Miłosz [2:55]: Yeah, it depends on what you are smart about. But you can easily see this in different people you work with, especially in teams. Some people seem to learn much faster than others. And seems there must be some secret behind it or at least some ideas that can help you learn faster Robert [3:22]: I’m not sure if you agree but from my experience it’s often not about what to do but actually the opposite so what you shouldn’t do to actually not be slower so I think we can probably compare it to going somewhere by the car so let’s imagine that you have Lamborghini and, slower car, but you don’t have a map when you’re going with Lamborghini. And in this case, you might get somewhere really fast, but not in the place that you need to go, actually. Miłosz [3:52]: Right. So the destination matters, basically. Robert [3:54]: Yes, yes. Because just going fast is a trap, because you can go fast to the wrong direction and spend years on learning things that are useless in practice or maybe are useful in short term. But they’re not helping you during the entire career because probably you work as an engineer for… Miłosz [4:17]: Yeah, and it’s hard to differentiate because it’s easy to feel like you’re learning something. You can accommodate some trivia or ideas but you never put them in use. So I guess what we want to talk about today is learning skills. Robert [4:36]: Something. Miłosz [4:37]: You can put into practice you can use somehow in your work or just projects in general Robert [4:45]: Yeah so probably it would make sense to start with some kind of maybe high level maybe not definition but idea what learning, means because you can do many activities that may sound like learning but it may not that’s useful for you to use in Rework. Miłosz [5:13]: So maybe the fact that you can use a skill to build something in practice would be the primary one for me. So it’s not only about having the knowledge about some tool, library or programming language, but being able to use it in work to build something actually useful. And probably not a trivial to-do app or something most tutorials teach you. Right, so it’s quite easy to use any programming language or library to spin up a trivial web application, because there are tons of tutorials like this. And nowadays AI can generate this for you right away, because it could learn on so many materials on this. But once you tackle building something more complex that you’ve never built before, or maybe never even seen before, that’s when things get interesting, and that’s where your skills are tested. Robert [6:25]: Yes, and I think maybe it may sound familiar for you, when, for example, you did the opposite. So, for example, you’ve been on a conference, you’ve seen some cool idea, or you read a book or article about this cool idea, and you’re trying to apply it later in work, and for some reason, it doesn’t work as it was promised in… Miłosz [6:49]: Yeah, because you missed some hidden complexity or edge cases that were not mentioned there, or maybe the tutorial didn’t cover it, and now you’re stuck. That’s a pretty usual experience when starting to learn something new in software especially. Robert [7:09]: I think we can probably compare it to playing an instrument or driving a car. Let’s imagine that you are trying to learn driving a car by just reading a book or watching some video tutorial on how it works. Okay, maybe it can give you some useful hints how to do that, but as long as you will not really do that and, put it into practice, you cannot really say that you can play an instrument or drive a car. Because in practice you discover a lot of hidden complexity that, okay, maybe on the movie it looks simple, but in practice there is much, much more and it’s harder than it looks. Miłosz [7:54]: And there is some danger, because you can feel like you actually know how to drive or how to play a guitar, let’s say, after just reading about it. But then in practice you don’t have any experience, so you won’t be able to do it. And it’s quite similar to reading books maybe, on programming especially. And I think we read quite a lot, especially 15 years ago when there weren’t many online resources, videos and so on. The books were primary learning material for us. And sometimes the theory gives you nice ideas how to work with codes or mental models, but you have to actually start writing code to see if you learned anything. Robert [8:48]: And it’s because you can read about many things, but often what’s described in the books it’s applied in some very specific context. And the project that you are working at this time may not really fit this, or it may fit, but it’s a bit different problem to solve. Because there are many similarities between multiple projects, but there are also some small things that are a bit different, but it can make problems much, much more different. Miłosz [9:20]: And because books are focused on just one area, they can’t give you the entire context. And I think we felt this because we came up with many examples to teach people some concepts. And it’s very hard to come up with something that’s not this trivial to the application, that’s not complex at all, and come up with something that’s kind of like in real life, but also can be used to teach someone something. And same issue with books many books try to introduce this idea of a project or something like that But it’s quite hard to do it well. And to cover some real-life scenarios that someone can later put to use in their own projects. Robert [10:09]: Yeah, and there is also one thing from our perspective of people that we’re creating online examples, articles, book. It’s also hard balance because, okay, in theory we can create an application that is as complex as a super complex application built by 100 people. But from the other side, it will be also a bit hard to work with that and learn from that. So it’s a hard balance that we are trying to achieve. The other thing that is also coming to my mind is that it’s connected to the thing that context in the book may be a bit different than yours. For example, when a couple years ago we were switching to Go programming language, we were working in some projects that required some more advanced tactics to write software like domain-driven design. The other side was that there were not all the materials. Actually, I would say that all the materials that were out there about domain-driven design was totally wrong. Miłosz [11:10]: Yeah, it was a very long time ago. Yeah. It was very in the early stages back then. Robert [11:17]: It still didn’t change that much now. There are some materials that are good, but still, most of them are still missing the point. But anyway, I think we spent probably half a year to figure it out, how to do it in a way that is compatible with Go programming language. And you can say that, OK, it’s just using it in a different programming language, but in practice, it requires time. So it’s not that easy. Miłosz [11:42]: Yeah, especially if you do it on a live product. in a team where no one has experience like this. Thankfully, we could use some books, like implementing domain-driven design back then, which was helpful. But still, there was a lot of iteration, and I remember the first approaches were much different than what we ended up with. And probably not that great. Robert [12:10]: Yes, probably we can summarize this first anti-pattern as this theory is not practice, and we’re taught to not be the person that is reading 50 books and think that, okay, I know how to solve many problems. Because just reading about that, it may be not enough. Because on paper, everything looks great, usually, but in practice, it’s totally different stories. And I know some people that were like that, that they were reading about many things, and when it comes to solving some problem, they were like, oh, let’s just do it like that. I wrote about it. It will be simple. But I know already that it’s a bit different in practice. Miłosz [12:50]: Yeah, sometimes it’s fine to read just for fun if you are curious about something. But it doesn’t mean you have to use it in your day work. But it’s much different than learning skills, I would say. Robert [13:05]: So it’s all about extremes. So don’t be a terrorist that is detached from reality. Don’t be like an Envoy Tower architect that knows everything best from theory but doesn’t touch this code and doesn’t need to write it. Miłosz [13:18]: If reading doesn’t work, maybe video cursors. Do you like video lessons? Robert [13:26]: No. No, because I think it has a pretty similar problem. So maybe it can give some more feeling like you’re looking like somebody is implementing something and it’s a bit closer maybe to feeling that you’re really implementing that. But it also has one risk that if you’re looking, if somebody more experienced is writing something, this person is probably skipping a lot of complexities that the person already know or just going straightly forward to the solution because, well, the person already know how to do that and doesn’t struggle about that. Miłosz [14:04]: If they show you a big project on screen, you probably won’t be able to grasp the entire context just by watching the video. Although I think it’s a nice format if you want to give a quick overview of some code, or if you want someone to see how to do something exactly, like refactor something. What comes to my mind is live coding on presentations. That’s quite a nice format because it is like 5 or 10 minute video. Not video, it’s live on presentations of course but it’s a different format that can be useful but what I don’t like is a video course that I open and I see there’s 24 hours of video watching ahead of me and I am already bored I didn’t even start and I know there will be lots of filler content in there and yeah, that doesn’t make me want to learn. Robert [15:14]: Yeah, so what are the solutions here? Miłosz [15:18]: Probably build something. Robert [15:20]: Oh no, I need to get my hands dirty. Touch the keyboard and implement stuff. That’s so bad. Miłosz [15:29]: Yeah, it’s basically what we mentioned about this iteration while learning from a book. Until you start fighting the obstacles struggling to make it work you probably won’t learn much and probably the hard part here is figuring out what to build right because again you can pick up a tutorial and build some easy application some trivial application but it won’t teach you much so you have to pick up something that’s a bit more advanced and that probably will stress you a bit i mean test your skills so if you pick up an idea and you think like i have no idea how to do it that is probably a good direction i would say because you will need to figure it out yeah Robert [16:27]: I think we can compare it to like playing instruments with planning to play a gig. It’s probably similar to how you should approach learning. For example, if you’d like to play a gig, you probably should start with something simple, play guitar yourself, later some rehearsals with your band, and later going to a gig and playing that. Obviously, it’s not easy, it requires a lot of time, but let’s imagine that instead you are watching a video on how to play guitar and you’re going strictly to directly to play a geek it will be probably not the best idea yeah and you’ll be super stressed and probably it will be a disaster but it’s similar thing it’s similar thing with learning programming skills so your geek is well the job that you’re working in and again you can try to learn something from just book from video and try to do your geek directly and okay, it will be a bit maybe faster, but from other side the result will be much worse. Miłosz [17:35]: A parallel that comes to me is in software world it’s like your employer gives you a video course buys it for you you watch some 20 hour video and then you’re expected to apply it in the company’s project which you’re probably not ready to do it yet and you will experiment on a real Miłosz [18:02]: product that’s used by real people. So there’s a risk that you will not apply it correctly and it won’t end up that well. Robert [18:12]: And it’s another risk that I can see. So if you have some concept and the best place that you’ll see to apply that is the project that you’re working on your job, there’s a risk that you may end up with doing the CV-driven development. So basically, you are applying all patterns that you can see in your job project, and it will be probably a disaster for the problem. Miłosz [18:39]: Of course, I applied the saga pattern in my previous year. Robert [18:43]: Yeah, and probably somebody is applying it to the context that it’s not helping. So it’s making the project worse, and also you’re not relearning. Because if you are not applying those patterns for the right problems, you will also not learn how to do it properly, because you just apply it in the wrong context. Miłosz [19:05]: So the ideal scenario, I think, is to have a pet project, but one that solves some real problem. Sometimes it’s hard to come up with something like this, but it doesn’t have to be super complex, but at least if it’s something that does something real, it gives you this kind of illusion of working on something more serious than a simple to-do app that’s free handlers and that’s it. It’s best if you can run in production and treat it as a production-grade deployment in general. So you have a domain and some servers or cloud or setup or whatever, set up some alerts, monitoring and so on. If you treat it like this, then you will start finding all the issues that can happen in a real-life project too. The hard part is finding this idea of what to build. Robert [20:16]: But it’s probably one even harder problem, because you can test it on some pet projects, But I think it’s also essential to, at the end, apply it to some real project with some team. Because I think we’ve been trying multiple patterns in some smaller projects. But after we have used it with teams, it also changed how we’ve seen all that. Because if you’re working on a project alone, it’s a totally different dynamic, let’s say. Because it’s just much simpler. because if you overcomplicate something, you probably know how to go over it. But if you’re working with a team, and if you overcomplicate something, you’ll be not the only person that struggles. Your teammates will struggle much more, and they will also not be super happy about it. Miłosz [21:10]: That’s a good point, since if you work alone, you don’t need to care that much about code readability, for example. If you can fit the entire context of what the application does in your memory you can use a single file that’s 5000 lines long and you don’t need many patterns there because you just know where everything is but as you work with someone else it starts to get more complicated it. So maybe that’s a good advice too, to seek someone else you can share a project with. I think that’s also how we started our more serious projects when we worked together. Robert [22:03]: So there is a question on the chat that, like with playing guitar, so it’s good to have friends to play with, but what if you have no friend to play together or practice with? How you motivate yourself to learn guitar until you join a gig. So we’ll have some solution for that. So we and Milos are probably pretty lucky that we know each other and we’re building projects together for almost 20 years. But yeah, we know that it’s not the thing that everybody, not everybody has this luck, but we’ll have some hints that can help with that definitely. Miłosz [22:41]: I guess the primary one would be you can find a community online and join someone with building a project but Robert [22:51]: We’ll touch on that a bit later. Miłosz [22:56]: Do you want to go over the two comments we have? Robert [23:01]: Yes let me just check what from that we’ll cover later, I think we’ll cover so we can cover them a bit later when we’ll have something in those contexts so let’s wait for a while but yeah, we have in on our list. Miłosz [23:27]: Martin mentions that he feels what I said about video courses and that they are like a torture yeah, I feel the same I think it’s interesting format for shorter things, showcases or screencasts, for example. But, yeah, like 20-hour video courses are… Robert [23:51]: Or maybe for some, let’s say, more soft skill-related… Maybe not really soft skill, but not coding-related skills that are also important, but they’re not about writing the codes. And yeah going maybe into the problems that are there so finding the friends to go together or, maybe you don’t have a time because we know how we know how it’s changing with the time so when you are younger you can code over nights and over the weekends but if you’re getting a bit older you may have family, you can have multiple more duties. And it’s a bit harder because, there are multiple obstacles that you have. So the first one is that you need to find some project to work in. And we already mentioned that this is the hard part of it, because Because it’s sometimes hard to find a good problem to solve that is fitting the pattern that you’re trying to use. Miłosz [25:03]: Not some artificial problem, but something that actually solves something. Robert [25:08]: And it doesn’t oversimplify that. Because if you learn that, okay, get some simple problem and try to apply domain-driven design for that, and you will try to move it to real projects, it will just end up with some over-engineering stuff and it will be not the best. The other thing is probably also related to this torture with videos that you have super big chunk of knowledge to ingest, like 20 hours of video, you are watching that and later trying to apply that. And it’s super hard. And sometimes you maybe don’t have one or two hours to watch this video. It would be better to just have some smaller chunks to work in. And already I’ve seen on the chat that somebody mentioned that my main struggle when learning new skills is finding clear, consistent, practical and consistent material. I hate huge as boring text-only documentations and more visual and neat short tutorials. So this is basically it. It’s not easy to find this. Miłosz [26:22]: Documentation especially. so it varies widely between projects. Some projects have really nice docs that you can get started easily, by the way. Give you some short examples and such. But some, yeah, not so great. You have to dig through the docs yourself and figure it out. So it’s definitely not easy. Robert [26:47]: It’s also sometimes hard to find one place that is going from A to Z and explaining everything. And sometimes you need to go over 100 websites to find how to do that. Some alternative is Ask AI for that, but also you know that sometimes it can give you some hint, but it can lead you into the trouble. Miłosz [27:09]: AI itself is a topic that suffers from this issue exactly. Because it evolves so fast, there’s probably no one place you can go and learn about how to use it. It just depends on which model you use and so on. So you have to go through many documentations and websites probably to learn it well. Robert [27:35]: And there is actually a question. So what’s the AI impact for New West that skip the regular learning process? Do you recommend AI for using it? There are, starting learning. Miłosz [27:48]: Software development? Robert [27:49]: Yeah. Miłosz [27:50]: So basically the question is how about using AI for learning? Robert [27:56]: What do you think? Miłosz [27:57]: I think there’s The same issue as with books on one hand, so it can teach you theory, and maybe even worse, sometimes it can give you ready answers, which you just copy paste or use directly in your IDE and you don’t touch the code, which can work, but you also probably won’t learn as much if you don’t struggle. And the other thing would be hallucinations, which is the primary reason I don’t use them much. For example, learning a library. I tried a few times to use AI to generate some use case for a new library I didn’t know, or an SDK. But very often it makes up function names, or just sort of invents something that doesn’t exist. And in the end, I just prefer to read the docs myself and do it from scratch, because it’s often confusing and I I don’t like the fact that I have to double-check everything it does for me. Robert [29:21]: Yeah so i i think i have pretty similar point of view that i would say that it’s double-edged sword so it can help for many cases but you need to be really careful about what it’s saying and double check that okay because sometimes it’s helping to discover some thing that it may be harder to find somewhere in the internet or in the documentation and it’s helpful but it’s all about, verification that, okay, what it said, it makes sense. And I think it’s probably easier to use for more senior people that already know more context and more understand how, what’s the impact of the things that it’s generating. I can imagine that if I would be more junior developer, it would be the risk that, okay, I’m getting what it’s giving me and putting it to the project, And I’m spending ages to debug that and understand what’s happening. And if I would just follow the way that you mentioned that I would do it from scratch, it will be just much faster. Miłosz [30:26]: If you want to learn the skill, I mean, you can argue that you won’t need the skill in the future because of AI. That’s a separate discussion. For now, no one knows for sure, so let’s leave it. But I also think there’s so many use cases for how to use AI for learning it’s very easy to give one answer because making it write the code for you is probably not the best idea if you want to learn some skills but I can imagine you could use it to give you some roadmap on what to learn you could prompt it with something like I know JavaScript and I want to learn Go recommend some concepts I should read about probably more helpful than just copy-pasting the code it gives you. Robert [31:22]: Yeah, and I think there is one thing that with AI can be pretty helpful, and this is something that we’re actually working for last more or less one month. Because there’s also one problem when you’re learning something, you may be stuck on some stupid typo or some stupid mistakes, and sometimes AI I can help you with being unblocked. So actually, since one month, we are now working on implementing that kind of mentor functionality in our learning platform, because we noticed that some of you were stuck on some, again, some typos or some simple mistakes. And it can be actually pretty nice in finding those typos or mistakes and unblocking you. But from the other side, it can also say some stupid things. And I would say that this is the biggest challenge that we are fighting now, is ensuring that it would not give you some false advice and learn you something false, basically. Miłosz [32:23]: And that’s something that’s very hard to understand if you are not an expert in the field. Because it will sound very confident. And if you have no experience with what it talks about, you want to know better. So I will be careful there. Robert [32:41]: The good news is that we found some ideas what we can make to make it less wrong and pretty moderated on what it’s saying. But again, it will be never like 100% accurate. But let’s imagine that probably you remember some cases like you were building something for fun and you were stuck for 10 hours because of some stupid typo. So it will be just super useful to be unlocked because it’s not something productive let’s say to be stuck because of type of our 10 hours and it’s just frustrating. Miłosz [33:13]: Yeah that’s a good point so also important to not be not go too far to the other extreme and say I won’t use AI ever because it’s not a good idea just use it as a tool as any other tool Robert [33:32]: And I think it’s in general useful to making this tough process of learning by building stuff a bit nicer. Because we’ve been doing that and sometimes we’ve been really frustrated by the structure of building something. So basically we are trying to make this process more nice. So we are also, in our learning platform, giving some projects that you can work on. So you don’t need to find out the problems for the techniques that you are learning. And I think it’s also pretty cool because, again, in our life we built a lot of different projects and it took us some time to find some nice projects to learn stuff. But we know that it’s sometimes hard, especially when you don’t have that much free private time or sometimes you maybe would like to just have five minutes to work on something. And we’re trying to cut this learning to some small steps to give you ability to just spend five minutes a day and progress a bit on this small project that we’re giving. Miłosz [34:48]: I like the fact that we recreated the way we used to work in the beginning. So you read some theory and then you solve some exercise, you write some code and you see it applied live. Miłosz [35:07]: And still to this day, I think, is the best way I know of learning. You need some theory and then some practice. So this is what the other platform gives you as well. Robert [35:19]: I think it’s also nice that it has some path with some proven techniques and you are just following those techniques. And you don’t need to look on hundreds of websites to find how to, for example, do event-driven architecture or something like that. But again, it’s not for everybody. So when we’ve been younger, we have just time to figure it out, go over internet and try to find all these techniques, try multiple different techniques, try to find a project to implement. But yeah, it’s also fun. Yeah, yeah, definitely. Miłosz [35:53]: We can spend the time. Robert [35:55]: It’s not for everybody, but again, if you are looking for some nice learning resources, go to our websites and it’s tab, trainings and courses. Again, it’s not for everybody. We’ve been not using such a platform because it didn’t exist, but again, it’s also not only way to learn, so you can easily replicate it yourself. Okay so the next thing about learning that i think i heard pretty often is that learning needs to be fun and interesting yeah and i think it’s also a bit risky because, it’s easy to start to optimize for fun in learning and in my opinion it’s not the best thing so if you compare it to school so well school is not about making so primary goal of school is not making to make learning fun it’s about learning if you would like if you are optimizing for fun not learning it’s kinder regarded, what do you think about yeah. Miłosz [37:05]: And what comes to my mind is this gamification of you know different learning products so you get some prompts or badges and so on. Robert [37:17]: Like Duolingo, for example. I’m not sure if you’ve been playing with Duolingo. Miłosz [37:21]: A few years ago. Robert [37:23]: I remember that it was actually quite cool that you have badges, you can have some frozen… Miłosz [37:29]: You can invite friends and do it together. It seems like fun. Robert [37:34]: And at some point you’re actually realizing that you’re not learning that much. I mean, you’re clicking some funny stuff every day and you have to take 100, days, it’s great, but it’s also like, okay, it’s very repetitive, and maybe it’s fun, but okay, I don’t feel that I went very far away. Miłosz [37:55]: So I actually have a recent realization I had. So we were working on this AI stuff for our platform, integrating LLMs. And it’s a new technology, you know, there’s many things that have no standards yet, you have to discover many things and so on. And initially, I was very frustrated while working with it, because it’s a different mental model too. For example, you mentioned about hallucinations. We are used to programs being very strict and predictable. But with LLMs it’s just the other way. It’s a bit random at times. And it was initially frustrating. And then we figured out ways to make it work, basically. And it hit me that this frustrating part is usually the sign that you are learning something, you know. Robert [39:07]: But I think it’s interesting because it’s often frustrating from one side, but it’s also a bit exciting. I’m not sure how to describe that, but… Miłosz [39:16]: It is exciting after you struggle. Robert [39:19]: Yeah, yeah, maybe, maybe. And at the end, you don’t remember that much this frustrating part. Miłosz [39:24]: Yeah, yeah. So, I think that’s actually what you should be striving for, maybe? I’m not sure if that’s a good word to put it, but basically if you feel frustrated while learning, you are probably onto something. Because you are frustrated that you can’t make it work. That’s how I feel about it. And then it also motivates you maybe to push through, or maybe demotivates you to give up. But I think it’s a better sign than having fun. You can be having fun vibe coding. It’s a fun thing to do. Robert [40:08]: Or making games. So for example, something that you recommend. Miłosz [40:12]: Yeah, making games is fun too. But also can be frustrating if you don’t know how to do something. It’s a similar thing. Robert [40:19]: And I think it’s similar when you’re working in some projects. So probably we worked sometimes in the same projects. And I’m not sure if you agree, but from my perspective, I learned the most with the hardest project that I worked. So the projects that were easy, greenfield, and everything was known. Okay, it was nice, fun, but I wouldn’t say that I learned at the end that much. I think I learned the most with the projects that, you know, it was sweat, blood, and at the end we succeeded to kind of overcome that. But there were times of this project that it was really hard to just go over it. And well it was tough it was frustrating but at the end we’ve been able to overcome that and i think it only works probably when to also be clear here so it’s not only about projects that it’s hard tough and it’s you’re not doing about anything about that because this is a death march it’s not learning but if you are in the hard project and you have some plan how to improve that and you are implementing this plan, you are learning a lot. For example, you have some super legacy application used by many people with super complex logic, and it will require a lot of skills on multiple levels to just overcome it. Miłosz [41:41]: And you just have to figure it out. That’s the hard part and also the best learning moment. Like when you lose sleep because you fight with some issue you can’t figure out the solution for and then you the next day you wake up or you take a shower and you somehow grasp it unconsciously but that comes after this struggling part it’s not, you know, you are not just happy about it and having fun the entire day you think hard about how to solve this. Robert [42:23]: And I think the opposite, I can sometimes see the opposite in people that think that are learning from social media, let’s say. So you on X are observing multiple programmers that are doing some cool stuff and you read that and you feel that, okay, I know how to overcome some problems that people shared. But I think it’s also dangerous because it’s worth mentioning and reminding that social media is full of shows, let’s say, making grass greener than it is, and a lot of populism. Miłosz [43:06]: Yeah, so basically extremes. Robert [43:09]: Yes, so what works on social media is writing about something that is controversial. Many people will disagree, comment on it, but unfortunately this is what works. And you may think that, okay, I’m reading those comments, ideas, and I’m learning a lot from that. But I think it’s not like this in practice. And if you have more experience, you see that a lot of that is just making grass greener and hiding the bad parts of some approaches. Miłosz [43:44]: Yeah, it’s not really learning according to what we talked about initially. Because you don’t use the skills and you don’t learn any skills actually. Just learn some trivia maybe or some strong techs of some tech influencer. Robert [44:03]: Especially those a bit less experienced that they don’t know that they doesn’t know and they think that, okay, you are overcomplicating stuff. You don’t need to use X. It’s not that, just go simple. Miłosz [44:17]: Yeah, it may also seem like you contact only the extreme views on some topic. Because that’s what attracts most comments and trolls and discussion in general online. You are very extreme on one end of the spectrum about something. In reality, there are probably not so many easy answers, and it depends on the context. Our last episode about clean architecture is probably a good example of this, where you can find as many people who love it as those who don’t want to work with it ever again. Robert [45:10]: But again, it’s a risk, because it’s easier to just scroll through social media and have something of learning, but again, it’s not really learning. It’s probably also similar to watching some fun programming streams and people doing some fun on coding. So it’s probably easier to ingest and it may be fun. But again, I would separate the fun from learning. Miłosz [45:42]: It’s entertainment. Robert [45:44]: Exactly. Miłosz [45:45]: For example, my YouTube feed is now full of drummers doing some white tricks. But I’m not even playing drums. It’s something that’s cool to watch, but it’s not learning. Robert [46:03]: So again, what’s our tactic here? I would say that entertainment is also important, but it’s also good to separate some time for some deep dive content that is not making grass greener and making ideal world in places where it’s not. It’s better to sometimes find some deep deep dive videos or articles, basically something that we believe that is teaching you a lot. And this is actually something that we’re doing on our blog. So for example, we have multiple blog posts that are super long and they’re probably not as easy to read as some five minutes articles. But from the other side, the alternative is reading some simplified articles and later struggling to implement it on production, because it’s usually how it’s ending up. Miłosz [46:59]: And even if you read this deep dive article, you still need to apply it somewhere to truly grasp it. Right so it gives you a more more more tools more context but still it’s best if you just use it yourself Robert [47:16]: Yeah and don’t take us wrong so it all doesn’t mean that learning doesn’t need to be fun it’s just about putting the priorities so the first is learning because if you spend time on learning wrong stuff you just lose a lot of time and for example if you’re prior to prioritize, fun but i would maybe say that you know you should not you should not optimize for fun but for excitement rather so find some exciting project that you can work on that will excite you to go forward and even if you’ll be frustrated this excitement will try to you. Miłosz [47:58]: Want to see it live Robert [47:59]: Yeah that’s the motivation. Miłosz [48:01]: Here that’s very nice Robert [48:03]: And i think that it’s for example what which helps me often to overcome this hard part. Miłosz [48:12]: When the frustration hits, you need to use this excitement to look forward and push through. Robert [48:19]: And I think it also helps at the end when you spend a lot of time on this frustrating project, but at the end somebody’s using that and you see that it’s useful for people and it’s like, oh, okay, it was better. And I think, for example, I have sometimes like this with Watermule, with the library that we created, that sometimes working on fixing some flaky tests or merging some PRs or spending time on doing some reviews, it’s a bit sometimes boring. Maybe it’s a bit sometimes tough to do. But at the end, when we see that we’re making a new release and people are happy about it and people are using this, it’s giving some kind of fuel to progress with it. Miłosz [49:02]: And it also started as a small library that solved our specific use case maybe also a good example of how we can start working on something that is useful for someone else and can help you learn something because we also learned a lot about maintaining an open source project over the years thanks to it definitely Definitely. Robert [49:30]: So, okay. Now we, I think, covered pretty deeply what is the good direction, what you should basically learn. So, let’s imagine that you have your map now, but let’s now discuss for a while how you can get faster to this place that map is showing you to go. Robert [50:01]: So, the first thing that I have in mind in this is, again, what you should learn. Or, okay, actually what you should learn. So, I think what helps a lot is focusing on things that won’t get out of day pretty quickly. Because to take some example, let’s imagine that you are a front-end developer and you are some framework specialist. And the risk is that your knowledge may become obsolete much faster if we compare to learning it some universal skills. So in other words, I think what helps to learn much faster is trying to focus on things that are timeless. And basically, you can get benefit from the compounding knowledge over the time. Miłosz [51:10]: Maybe also not stick too long to one technology, if you see something else that you could use. Which on the other extreme is also not that nice if you just keep jumping from one programming language to another. Even take frameworks, it may be useful to know more than one, even if you primarily use one, even to just learn some different concepts. Maybe that’s actually a good one about programming languages, but you can learn some just for for some different point of view even if you don’t do it on production But it can also teach you those timeless concepts you mentioned. Robert [52:03]: That’s true. That’s true. But I think one help that can help you evaluate… Because you are totally right. So it’s important to not go into extremes like, okay, I’m not learning any framework, I’m just learning the things that are timeless. But it’s more about not going in an extreme and trying to mix it and not being a framework specialist. I think we already mentioned it in previous episodes multiple times. And in terms of AI, you will be never better than AI in learning some framework quirks. Because it’s just a limitation of your memory and it will be more updates than you can ingest. and you will be out of date sooner. But there are things that, well, AI will be never as good as you maybe. And like modularization or splitting bigger problems into smaller things like that. And it doesn’t matter actually what programming language or what stack you are using. For example, if you learn how to modularize things or split some problems into smaller. It will always help you, independently what technology you will use. Miłosz [53:23]: It’s still relevant sometimes after decades since the invention. Robert [53:29]: For example, I tried to check event-driven architecture that we are maybe not advocates, but we see many use cases where it’s useful. So from what I found, the first origins was in 1915, so pretty long time ago. Current form, it was formalized around 20 years ago, so a pretty long time ago, but again, the idea is pretty old. And the cool thing is that probably when you learned in this eight years ago something about it, it will be highly relevant still. And if you learn that many of the things that are universal and timeless, it will just help you over the entire career. Miłosz [54:22]: Yeah, you can apply it to other languages or frameworks or whatever. Sometimes it’s fine to see how the solutions are reinvented in a different way. And this is like something fresh. Robert [54:38]: Yeah, we’ve been touching this in an episode with clean architecture, I think, at times. Miłosz [54:45]: Or, let’s say, like, Go channels, which seem quite fresh when coming from some language that doesn’t have these concurrency primitives built-in, but they are, too, not some new concepts, right? It’s just a concurrency model applied to a new programming language. Robert [55:12]: I think it’s also pretty interesting that Vibe coding is a pretty hot topic, let’s say, now. And let’s even imagine, so I think it will not happen soon or ever, but let’s imagine a scenario when LLMs and Vibe coding totally replace programming. And still, in this world, skills like modularization or splitting problems into smaller, it will be still pretty useful. So you can try to hint this LLM a bit to how to split this problem into smaller, just to not have 5,000 lines of code, file, with everything. Because it will just struggle to do that. And it’s visible now, when you’re vibe-coding something and you’re trying to put everything into one file, it’s just a disaster. And it’s not that obvious how to split it into smaller chunks, let’s say. But if you have already this skill, it’s much easier because you can hint this model how to split that and its performance will be just much better. Miłosz [56:17]: And modularization is probably a good example of this topic you can’t really learn that fast. And there are many extreme takes on it, like use microservices. And once you work with software for a while you know it’s not that simple and it’s also not something you can learn from a Twitter thread you learn in 30 seconds where someone says use this directory or whatever you just have to work through many different projects and learn some patterns learn some mental model that will help you later apply this approach. Robert [57:02]: And I think it’s a similar case with testing. So, again, let’s imagine this work when we are no longer writing code, we are vibe-coding everything, fine. But it’s still useful to know what kind of tests you would like to have, and, how they are made. Because even if your model is great, it’s good to ensure that the next prompt will not do some regression in the logic that you had. And I would say that if this word will come, a bit won’t, but never say never. I think it will be critical to have proper tests for that, to ensure that does other prompt will not break things that some previous prompt did? Miłosz [57:48]: Yeah, we can see it with the work we did recently. But similar testing principles still apply. Exactly. Even if you use new tools. Robert [58:00]: Don’t be afraid. If AI will take your job, learn testing and modularization and techniques that you’ll feel that will be timeless. Miłosz [58:13]: Maybe it’s a more general suggestion to have a diverse skill set. Probably everyone specializes in one area and it’s fine, but it’s useful to know a bit more about some fields similar to what you do. Like if you specialize in backend programming, you can still grasp some front-line ideas. So it makes it easier to communicate with teammates and solve more high-level problems than just looking at code. Robert [58:50]: Yeah, and I think it’s something especially good for some more senior people, let’s say. So if you’re a junior, okay, let’s maybe try, you probably should try to master one area, let’s say. What more senior you are becoming, Link. More you should learn some other areas because i think what’s also useful is the fact that you can borrow from different areas let’s say so if you are even doing back-end you can borrow some ideas from front-end maybe from data science maybe from i don’t know ux or something like that so there are many nice ideas in different areas that can help you solve problems, or maybe even some areas that are totally not related to programming, like accounting, for example. You worked with some accounting domains. Actually, I think accountants have some cool ideas that you can apply to different domains, and it can help you to solve those problems nicely. Miłosz [59:56]: Yeah, especially since the more senior you get, the more you work with people outside engineering. And you solve more high-level problems than just fixing bugs in the code. You have to figure out the entire process of a feature, let’s say. So the more you know about some other fields, the better you can do it. Robert [1:00:21]: And what I think helps here is, again, doing some pet projects that… It’s probably our example. It’s not that pet project because now we’re doing it full-time, but before doing it full-time, we’ve been creating those pet projects and we needed to learn how to do front-end, how to do some basic data science, how to do some basic UX, and I think it helped us over the years when we were working on just back-end engineering, mostly in a daily job. And also there’s one thing that it’s sometimes good to, learn is about some management basics. So if you’re a software engineer, it’s sometimes nice to work for a while as a manager, because you can also understand the wider picture and maybe understand how the component dynamics are working. Because often we can go into some extremes like, why those stupid people want something from us? And if you work as a manager for a while, you know that there is sometimes some reason for that and it actually makes sense. Miłosz [1:01:38]: The dangerous extreme here is thinking of yourself as, I’m just a developer. Tell me what to do and Don’t make me figure out your issues. So, yeah, that’s probably what you don’t want to do, because then you become, well, easy to replace, basically, because you are just typing code. Robert [1:02:13]: All right so we’ve already said that uh from where you can to recap maybe from where you can learn so books so that this is the typical place from where you can uh learn from and it’s good direction but remember to put the things that you learned in practice because you will just forget you learn pretty quickly and it will not give you that much uh that you read this book basically. Miłosz [1:02:44]: What I like about books here is if they stood the time of test of time and became classics like the Pragmatic Programmer for example it’s an easy way to figure out that there’s something valuable in them but you have to be careful because the quality varies there are some books that probably could be could have been blog posts but there are also some timeless classics and it can also teach you some theory that’s very nice to know. Robert [1:03:17]: Yeah, so videos, as I said, we are not big fans of videos. There are some for programming knowledge. Maybe it’s nice for some soft skills, but as somebody mentioned, it’s sometimes a torture. And I think it’s also a hard process of learning like that, because you cannot spend five keynote on watching that, switching to code and later watching, I think it’s not working best, basically. Or for example, watching something for one hour and going to code. Miłosz [1:03:50]: It’s easy to get distracted or bored. Robert [1:03:54]: Yeah, so I think we more recommend the way that, for example, we are promoting in our learning platform that you are learning some small part of knowledge, applying that learning piece of knowledge and applying that. But you can also do often similar things with the book, basically. What’s also helped to remember stuff, at least for me, but I think probably for most people, is making the notes of the book. I think it may sound obvious, but sometimes you may not have enough time to do that. But I highly recommend that because without those notes, the chance that you forget what you just read is pretty big. What also works for me pretty nice is doing mind map for some more complex content for more complex or more complex concept content yes, because it’s also helping to navigate through them easier and basically. Miłosz [1:04:56]: Writing down in your own words often helps you think differently about the concept and also teaching others is very useful. Because then you try to figure out how to describe this in a simple way. It’s often when I write about some topics on our blog that I start to get a deeper understanding of it. Because you have to construct this series of thoughts that someone can read and understand. So having your own blog or even writing in private is super useful for having a more deeper understanding of those ideas you learn. And you can go back to it later in the review. Even if you don’t, it’s worth the effort, I think, because it just helps you think better. Robert [1:06:00]: And again, writing notes with your own words, I think it’s very important because it’s forcing you to do some thinking process and just remember it better. Miłosz [1:06:14]: And something I really like to do is mixing theory with practice. We mentioned it before as well. Miłosz [1:06:25]: So, you know, it’s easy to go to the extremes here. So on one hand, you can be the person who just reads the entire documentation first and then starts applying it, which can take a long time and probably you will forget some parts of it. Or you can just try to blindly write it yourself and never consult the docs. It’s also probably not a good idea because you can miss some concepts. So usually what I like to do is, let’s say I use some new technology or library or something, to get the example from the documentation and have it running on my computer. That’s the first contact you have with it. You can see it compiles, you see it running, you can play with it. That’s how you learn how it works. And then you will probably run into some edge cases you don’t understand. This is where the frustration begins. And this is when it’s time to read the manual. Just go back to the docs, find maybe some theory of how it works and you just go back and forth between practice and reading. I think that’s what works best. Robert [1:07:52]: One thing that sometimes people are mentioning that it’s great to learn from mistakes and That’s true, but it’s also not the most optimal way. So often you can lose a lot of time going… Around some simple solution that you don’t know that exists and in perfect word it will be cool if you can have some mentor so sometimes you can have some mentor in the work that you’re working on if not find some communities so for example like our discord that we have or there are some communities around our trainings maybe reddit and x may be tricky part because a lot of online communities are extreme and if you are less experienced, it may be tough. Some asking AI, it can be also sometimes. Miłosz [1:08:47]: Yeah, it’s a bit like this. If you don’t know what you don’t know yet, describing this to someone, maybe AI, maybe not, can be helpful. I remember how I was a junior programmer, it was super useful to have some seniors around that I could nag with questions during the first few months of work. I think it’s a super fast way to learn. If you have some problems at hand, you need to solve, but you don’t know how, so you ask someone who is an expert in this area, and they can explain why and how it works. Robert [1:09:36]: Yeah, and I think from my experience, probably sometimes we missed this mentor, this person that could say us that just do it in this way, because we spend sometimes a lot of time to figure out some stuff, just to learn later that… This is the typical way of doing this, and if just somebody said to us, it would save a lot of time. On the other side, it’s also a nice process sometimes to find a way to solve some problem and, it was also useful. Miłosz [1:10:09]: But i Robert [1:10:10]: Think it’s important to have some balance basically here. Miłosz [1:10:16]: Yeah but yeah maybe i i use for this can be a good idea sometimes because you not everyone can afford to have a more senior person sitting next to them and teaching them about some concepts so if you can have the the model explain some things or explain why something is done this way or another. Coming back to the previous discussion we had, if AI is good for learning, that might be one good use case. Robert [1:10:47]: Watch out for that, because it can sometimes say some stupid stuff. What also helps? I think it’s also good to sometimes feel as the dumbest person in the So basically, work with people that are more experienced with you. And if you sometimes feel that you’re not the dumbest person in the room, and maybe it’s time to try someplace where you will be the best person, Robert [1:11:22]: and you have some people to learn from. Because this can really help you to accelerate your learning. Miłosz [1:11:35]: So maybe online communities are a good example of that. If you don’t get caught in a flame war. Robert [1:11:44]: Yep. So watch out for that. All right. So something to add, Miłosz, or we can go to Q&A session. Miłosz [1:11:51]: Yeah, I think we can wrap up. Robert [1:11:54]: So yeah, just a reminder. So if you have any questions now, please just leave on the chat, and we can answer that. Okay. Okay, so there is a question about our event-driven training. So it’s one typical question that we have. So yeah, the plan about event-driven training is that, as we mentioned a bit earlier, so now we are working on our AI mentor that is integrated with our training platform. After that, we are planning to do big update to event-driven training before releasing that, and after that we will release that. So this is our plan. If somebody who is listening to that, if you already have event-driven if you already own go event-driven training, no worries you also have access to this because, as we promised you have lifetime access and lifetime updates of it basically this is the plan, we don’t have exact dates because what is the reality of estimation so. Miłosz [1:13:00]: We received many suggestions and feedback from trainees on the training. Robert [1:13:09]: We also had a lot of feedback like this is the best training that I ever had but we all are aiming even higher. Miłosz [1:13:16]: We also received some issues that we could solve and we wanted to build another training first but we probably will first update, go event driven before the next sale. So you will hopefully be able to learn even faster and better with it. Robert [1:13:39]: If you are not in our newsletter, the best way to know about this is obviously joining that, so you’ll be the first person to know. Again, we don’t want to commit on any date because we don’t know. When it’s done, it’s done, but, trust us it will be really cool update together with this uh ai mentor that we’re working now so we are at the last mile let’s say and it’s really really helpful with many cases when you are basically stuck or you need some hint what’s about what this exercise is it’s it’s cool okay the. Miłosz [1:14:19]: Next one from ryan hi self-taught two years in career changer i know i need to build and learn as i go i’ve been studying tdd bdd any tips to navigate the design of my data modeling user stories i’m so noob don’t worry and so this is about the modeling of Yeah, modeling of the design of the application, I guess. It’s a bit like DDD, but maybe more general. Robert [1:14:54]: Yeah, so I think we already have some blog posts about domain-driven design. They’re not that big about modeling, I think, on our blog. So you can have some hints, but this is definitely the topic that we’d like to explore deeper because it’s useful for more complex domains. And we had some experience with that, so we’ll be happy to share that. So we have somewhere the plans to write some new blog posts about that and have also a training that is around it but well as we said for the previous question so we have pretty long backlog before before that but. Miłosz [1:15:31]: Maybe the follow-up is relevant because then royans says learning to deal with malleability of software and wanting to design things as from the beginning as possible is crippling me. So I think that’s the core issue. You need to let it go. This is something we talked about in the episode about architecture. And maybe in some others as well, that you can’t figure out the perfect design from the ground up. It never happens, because the software will change with time. So instead you need to adapt, try to start with something simple enough that works well, and see what the issues are, and then iterate. Robert [1:16:18]: Yeah, and I think what’s helpful is the approach that model will be never perfect. It needs to be useful. That’s it. It’s a bit like a map. So when you have map, we’re talking about map multiple times during this episode, but when you have map, it’s not a picture from satellite or it’s not very, it’s just detailed enough to know how to go from point A to point B. But it doesn’t have all details, but again, it’s useful. So I think thinking about models in a similar way is useful. that you can try to make it perfect but it will be never perfect as you said you change over time or won’t. Miłosz [1:16:58]: Be complete probably Robert [1:17:00]: Yeah yeah but it also doesn’t mean that you should just do YOLO and you’ll improve it later because it’s also not the best approach because if you just, take the old approach it will probably hurt yeah. Miłosz [1:17:15]: This is the the hard balance between making software that’s hard to extend and going to the other extreme where you try to foresee all possible future use cases and try to adapt your software to be universal so it’s it’s quite hard but it’s like i think it’s a good philosophy to start with that you know you want something good enough for now and something that is easy to change in the future and Robert [1:17:44]: Make it as simple as possible but not simpler. Miłosz [1:17:48]: Yeah that’s that’s hard to do i mean we touched on on this in the first episode about writing low quality code it is a bit similar topic um where to you know strive for this super high quality where everything is perfect but the reality is you probably won’t get it and I think we’ve seen it best in one system where we tried to model this accounting domain. And I remember, I anticipated this moment when we will finally cover all the edge cases and it will be finished. But then it turned out, even our stakeholders who were accountants didn’t know how to do some things. And we just kept iterating and struggling for months. Turns out that’s the reality of them. Robert [1:18:46]: And don’t be afraid that it will be not perfect because it will be not to set expectations. Also, when we are building some models that are far from being perfect, they are sometimes wrong. But again, as long as they are useful, it’s good. And practice helps with time. So you’ll have more idea how to get this pass. But, well, practice, practice, practice. Miłosz [1:19:15]: The next comment is about this, exactly. I switched six years ago, completely self-thought. Plus about two years of theory courses before the actual switch. The only way for me is to get as much real-life experience as possible. Robert [1:19:31]: Yeah that’s it i see that also martin mentioned so yeah without. Miłosz [1:19:37]: Practice no theory will stick Robert [1:19:40]: Oh yeah i think i can confirm that it so you have the same feeling after 20 years thank. Miłosz [1:19:46]: You for summarizing our episodes in one sentence that’s exactly Robert [1:19:51]: This. Miłosz [1:19:54]: And as soon as you feel comfortable at something, it’s time to learn something new. Good point. I’ve switched to Go plus DDD a few months ago, and it’s still frustrating to learn new things. I’ll buy it a bit faster. Yeah, that’s a good point as well. I shared before that I felt a bit like this, working with LLMs recently. Because for a long time we didn’t touch much new technology i think we just know because we we’ve seen that for many product issues very often technology is not the solution you can you can stick to very boring and stable technology and basically achieve what you need but yeah learning about something new is exactly this is exciting at the same time but also can be frustrating if you don’t know how to move forward. The good part is, once you do satisfaction and you actually know you learned something. Robert [1:21:04]: I think it’s also really that something really new appears that can solve some new problems. I think LLMs are pretty different because it has some use cases that it wasn’t possible to cover earlier, but from another side also a lot of companies are trying to find a problem for the solution. And it’s also pretty problematic because it should be the opposite. So you have the problem and you’re finding a solution for that. Miłosz [1:21:35]: Okay and ryan wrap up props up um i also go back and review material frequently when i can remember something without looking it up um yeah Robert [1:21:48]: I remember that when we worked in one company we have the book about ddd on our desks all the time and we were all the time opening the book and looking for stuff and it was like ah yeah it was something like that. But I think we learned during the time a lot. So it was just, okay, we had something in mind that it was solving similar problem. We were looking to this block. Unfortunately, the example was in Java. So it was like, okay, this is how it’s done in Java. Let’s try to figure out how to do it in Go. Miłosz [1:22:23]: Yeah, but it sure helps to have this material Or documentation. Another way to use AI maybe is to have it summarize some parts of it for you. Robert [1:22:34]: Or maybe put everything to RAG and query. Miłosz [1:22:39]: That’s another good use case probably. Because you won’t host it as much if you provide the full context to it. Robert [1:22:47]: It’s a very fancy search, but it can do more things than search. Because it’s not text search, it’s more like meaning search. Miłosz [1:22:55]: But when I think about it now, you know, going back to this project where we started using DDD, imagine we had an AI fed with this entire book and we could just ask questions about something specific or make it change the examples to go. That could be quite cool. But also it probably wouldn’t be exactly what we arrived at. So I’m not sure if that’s a good or bad. Robert [1:23:23]: Yeah, yeah. But it was definitely a value in figuring it out yourself. And, well, I think it may be hard to be junior now, basically, because, as Samadhi mentioned, it’s probably trying to avoid LMs in some cases, but it’s not always possible, let’s say. Okay and the last question on the chat so well why how did you decide to teach it’s more rewarding for you than just programming i want to teach and do presentations but i feel like i have no enough experience how would i start uh so yeah um so how we decided to do that i don’t know how I. Miłosz [1:24:20]: Remember that there was some New Year’s Eve party, and I told you I thought about writing a blog, and you said, yeah, me too. Okay, let’s do this. Sorry, created a blog. Robert [1:24:36]: I don’t have that good memory, but it sounds like a good beginning story. Miłosz [1:24:42]: There was no grand plan, I think. For me, writing about stuff is, on one hand, satisfying and on the other, it also helps me think better about the subject. So that’s what we mentioned about teaching others. So I think if you feel like it, definitely go do it. Robert [1:25:04]: And I think you don’t need to be super experienced to do that. So i think earlier you begin then better and don’t be that much afraid that okay i will write something stupid or something like that it’s not that big deal to be honest i mean it’s hard to overcome that but at some point you will forget about that so if you start but so i think the good thing about is that it’s as you said it’s helping you to structurize the knowledge that you have and also i believe that skill of writing clearly it’s useful in multiple cases so as we mentioned a bit earlier so it’s helping you to convince for example your teammates to go in some direction write some design documents for some problems that you are doing and you can benefit in many fields from this skill, basically. Miłosz [1:26:05]: And especially presentations are also not exactly about what you invented or discovered. There’s also how you present, how entertaining it is, and so on. So you don’t need to invent something great to be a good presenter or to be a good writer and yeah there’s probably still like space for for new articles good articles because especially now I think we are you know scattered with this AI-generated trivial posts on topics and spend time to write something great. Robert [1:26:57]: And yeah, I think it’s again no silly bullet but what we recommend is creating the deep dive articles that are five times longer than average article in the topic and I think that this is some niche. And obviously, it requires more time to write that. Less people will read it. But I think in long term, it will just work nicely. And I think it’s also good to write about timeless things. So we mentioned those timeless things multiple times. But to give you a good example, so the blog post that we wrote about repository pattern six years ago, I think, probably last month we have like 50,000 visits because it was just reposted on Reddit and it went over the entire internet. And again, it’s probably since the start of the blog, I don’t know how many people read that, probably, I don’t know, 200,000 people or something like that. But it wasn’t in this one year. It was over six years and it’s still pretty valid and it will be valid probably in 10 years. So, but again, it took us some time to write that, but from other side, probably there are no other articles on repository pattern in Go that are that deep. So it’s some niche, obviously. And you also asked if it’s more rewarding than just programming. Robert [1:28:22]: Oh yeah, definitely. And I think it’s also worth mentioning that probably you are still programming more than teaching. Well, we love to write code and I think it’s allowing us to have more ideas what we can teach, but it’s about some balance. And I think it’s just making it all more fun. Miłosz [1:28:46]: Maybe we still do program what we teach through the platform. And maybe just one part of the story here is that we wrote a blog post, we had some example projects, and we wanted to recreate this idea. We talked about in this episode about being able to read some theory, write some code, and do it back and forth. To help people learn. That’s how it happened. Robert [1:29:24]: So TLDR, I recommend it. Miłosz [1:29:28]: And share with us once you do. Robert [1:29:30]: Yeah, definitely. If you would like to also have some feedback, we can also help a bit. Right, so it seems it was the last question that we had. So last minute to… To ask us questions. Robert [1:29:57]: Alright, and so while we are waiting for the last questions, just in case if they will arrive, so just a reminder. So if you are not part of our newsletter, I recommend to join that, because we recommend to subscribe us on all platforms where you are watching us, listening to us but newsletter is the best way because we cannot control magic algorithm of other platforms but newsletter you will be sure that will notify you about new episodes, so remember to do that uh if you have any comments questions leave us comment on youtube we’ll uh we are answering all the comments and we can also improve the and or touch it in the future episodes please don’t forget to leave us thumbs up five star review because again deadly argo algorithms are are ruling the world and without that more people will not hear about about our live podcast so yeah that would be it for today i think so thank you miłosz for today’s episode and see you in two weeks where we’ll talk about the popular Go opinions we have. Miłosz [1:31:16]: Exactly. And why are we controversial? Robert [1:31:18]: Hopefully. Miłosz [1:31:19]: Can do it. Robert [1:31:20]: And if you are not writing in Go, join us as well. Maybe you’ll learn something new. Miłosz [1:31:26]: Or you can learn Go in two weeks using our learning platform. Robert [1:31:30]: Or maybe in one evening. Miłosz [1:31:33]: Okay. Thank you everyone for joining us today. Thank you, Robert. See you in two weeks.

Quick takeaways Clean Architecture is most beneficial for complex projects with larger teams - for small teams or simple projects, it can become overengineering. Separation of concerns is the core benefit - keeping domain logic separate from implementation details makes code more maintainable. It’s easy to go too far - using too many interfaces or too many layers without a clear reason creates unnecessary complexity. Start simple and evolve your architecture - don’t force Clean Architecture from the start if your project doesn’t need it yet. Understanding the “why” behind the pattern is crucial - blindly following it without understanding leads to poor implementations. Introduction In this episode of the No Silver Bullet podcast, we discuss Clean Architecture and whether it’s overengineering or a best practice for organizing code. We talk about why the pattern is often controversial, when it makes sense to use it, and how to implement it effectively. We share our experiences using Clean Architecture across different projects and teams, including the common concerns developers have when first seeing it. Notes Clean Architecture: a software design pattern that separates the code into layers, each with its own responsibility. Introduced by Robert C. Martin (Uncle Bob) Related patterns: Hexagonal Architecture / Ports and Adapters (Alistair Cockburn), Onion Architecture (Jeffrey Palermo). While slightly different, they share the core idea of separating domain from infrastructure. The Dependency Inversion Principle: one of SOLID principles. Blog post: Introducing Clean Architecture Blog post: Microservices test architecture. Blog post: Repository Pattern in Go Blog post: Combining DDD, CQRS, and Clean Architecture in Go (Mentioned as available) Example Go Project: Wild Workouts go-cleanarch Linter Quotes In general, if you ask many people about clean architecture they will say that it’s overcomplicating things and making your project worse. Robert One of the main points of Clean Arch is to separate the interesting code from the implementation details. Miłosz The goal is not using any kind of architecture. But the goal is to use something that is useful for you. It supports your velocity because at the end of the day, every technique that we’re using, it should allow you to code faster and that’s it basically. So if you are using some architecture and it’s slowing your down, you cannot iterate faster, it’s probably the bad one. Robert If you have been in a team that spent hours discussing in which directory to put some code, then you might assume you don’t want to work with Clean Arch anymore because it’s just it’s too much hassle to keep discussing how to use it. Miłosz If you are building more complex applications and you are starting to mix all this stuff it’s super hard to maintain that. And this inversion of control helps you to cut this logic in multiple parts and make it easier to maintain. Robert Don’t do it because you want your code to be clean, but to better organize work in a complex project that will grow over time. Miłosz Clean architecture is not that complex pattern. You have three, four layers depending on the complexity of your application. You’re ensuring that layer from inside doesn’t know about layer outside, and you’re using interfaces to do inversion of control. That’s it. Robert The pattern is not exactly about project structure, which is also easy to fixate on. All the layers are more concepts, not exactly directories, although this is often the case. Miłosz Timestamps 00:00:00 - Introduction 00:01:28 - What is Clean Architecture? 00:15:32 - Early Audience Comments 00:23:30 - Controversies 00:29:56 - Common Pitfalls 01:02:47 - The Benefits of Using Clean Architecture 01:21:20 - Team Collaboration & Splitting Work 01:28:29 - Can we Just Keep It Simple? 01:36:35 - Alternatives 01:38:20 - Go-Specific Implementation Tips 01:43:20 - Summary 01:46:51 - Q&A Session Transcript Miłosz [00:05]: Clean Architecture is an overengineering nightmare or the best way to organize projects. It all depends on who you ask. To make it work, you need to understand the “why” behind it and avoid the common anti-patterns. Today, we talk about why the pattern is so controversial and how it can help you in your projects. I’m Miłosz. Robert [00:25]: And I’m Robert. And this is the No Silver Bullet live podcast where we discuss mindful backend engineering. We spent almost twenty years working together in different projects and teams. During that time, we learned that following advice like “always do X” or “never do Y” doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices that will help you to grow into principal engineer level. Miłosz [00:55]: If you have any follow-up questions, you can ask them in the chat. We’ll have a Q&A session at the end as usual. And we might also pick up your comments during the discussion. Robert [01:07]: This time, we’ll try harder to look on the chat and pick up your questions, especially that I think the today’s topic is pretty controversial one. And, yeah, I think it would be great to see some different point of views on that and also hear your experience with that. So we can also learn something maybe. How do you see that? And we can also create better content. Miłosz [01:32]: Alright. So the title says Clean Architecture for today’s topic but the pattern actually has a few other names like Hexagonal, Ports and Adapters, or Onion. They are not exactly the same thing. There are some differences in the implementation. But I think we can assume that the main idea, the core idea behind them is similar. Right? So similar that we can treat them as pretty much the same pattern. And we won’t dive deep into the differences today, And we’ll just stick to the clean architecture just because that’s how we started calling it and I think it’s quite popular anyway. Robert [02:18]: Probably the most common term because, yeah, you could probably heard some terms like hexagonal architecture, ports and adapters, onion architecture. But, yeah, the idea on a high level is quite similar. Miłosz [02:33]: Yeah, and while it’s popular I think everyone understands it a bit differently which is common with complex patterns. Robert [02:41]: And probably this is the reason why there are also a lot of misunderstanding about that because many people actually are messing clean architecture with domain driven design. So some people don’t have a good idea how to connect that because, okay, in many clean architecture implementations you have the domain layer. So for some reason it’s like, okay, you have domain layer so we are doing domain driven design. Or some reason also some people also think that okay, clean architecture is some Java or it’s some enterprise. It’s not something that you can use. Miłosz [03:14]: That’s only for enterprise super complex projects. Robert [03:17]: Yeah, yeah. And in general, if you ask many people about clean architecture they will say that it’s overcomplicating things and making your project worse. Miłosz [03:29]: Which is problematic because if you don’t understand something, chances are you won’t implement it properly in your project. So that’s risky. And I think often it’s also a very controversial pattern. If you mention it to someone and they had bad experiences from the past usually they won’t react well, like, they won’t be happy about you using the pattern, let’s say. Robert [03:58]: Yeah, and I think it’s interesting that we worked in multiple teams and usually we have some people that were not happy about using clean architecture but at the end we’re kind of able to convince them that it’s nice. Probably we can have one person confirm that. So Firmino said that he’s missing working with Milosz. So it’s a good sign. So at least one person is missing working with you. Maybe clean architecture helped a bit. Miłosz [04:28]: Thank you Firmino, at least one person misses me. You can mention if you like clean architecture or not. So it might be one example for today’s discussion. Maybe it’s not that bad. Robert [04:41]: What I think is interesting about that the thing that I already mentioned that you know, we were leading, multiple teams in our lives and it wasn’t uncommon for us to have some people coming from other projects. And, well, we had pretty good experience with clean architecture and they were coming to our project and most of the people were like, oh, like, okay, it’s cool. Please tell me how it’s working. But some people, I think usually some people who have already done have more experience or thought, oh, no, clean architecture or over engineering stuff. But in most cases, it was like, okay, it took some time, like, it wasn’t weeks. It was like sitting with this person for a half hour showing how everything works. And at the end, most of it people were saying, yeah, it’s actually cool. I’m using it in my projects after the job. Actually, I remember one person even said that he was building a game and he used the architecture in that because he could exchange, I think, engine or something like that. It was quite cool. Miłosz [05:43]: So does it mean we should use it everywhere? Would you say it’s a silver bullet? Robert [05:52]: No. But, you know, I think pet projects after work, it’s probably a pretty good place to experiment and do some crazy experiments. But, yeah, long story short, I think we’re pretty surprised to later hear that it’s not everywhere like that, like in our team. So there are people that try to clean architecture and at the end, they were saying that it’s always bad. And we started to wonder what is the reason why people hate architecture in all projects. Miłosz [06:24]: Maybe we can answer it today. But maybe let’s start with a very short definition. Let’s try to boil it down to the very core ideas because there are many flavors and, you know, even if you use one of the flavors, you can use it in a different way than someone else. So let’s just try to agree on what’s behind the idea. For me, the first one is separation of concerns, and it often comes with the layers idea in those architectures. But after years of using it, I think you could just summarize it as, you know, you agree that your- project has code of two categories, The boring code, which is the implementation details that you don’t care that much about. Robert [07:17]: Or you should not care. Because, unfortunately, often it’s the opposite. Miłosz [07:21]: Yeah, like the SQL queries you use or what database engine you use. It is important, of course, but probably doesn’t make your business win automatically. And on the other hand, there is the the interesting code, the domain code, the logic of what your application does, sometimes called business logic. But I think this is a bit unfortunate term because it, you know, it again has those enterprise vibes. And I think people writing usual web applications maybe don’t think of business code when they write the logic they do. But the fact is it is some kind of domain code anyway. Right? Robert [08:05]: Yeah. But it’s worth mentioning that it depends heavily on what kind of application you’re working on. So just a kind reminder. So this is no silver bullet podcast, live podcast, and, yeah, it can suggest that it depends on different projects. And so as you said, so sometimes you have business logic, but sometimes it’s as simple that you can have it in controllers mixed with SQL queries. But sometimes it’s not that simple. I mean, you can go this way, but and we did it in our lives but it’s a nightmare a bit later. Miłosz [08:37]: But I think we can agree that the main point of one of the main points of CleanArch is to separate those two, right? To separate the interesting code from the implementation details. Robert [08:51]: Yes, and probably it’s worth mentioning how it’s done. So it’s all about the well, I think it’s not the first time when we’re mentioning that there are some old patterns that are still valid and the same is for, inversion of control. And the idea is that you can implement, you can implement so you can define interface on some behaviors that you expect for example, from the database, from external providers. So and thanks to having interface for that your implementation doesn’t need to know really how it’s implemented under the hood. And, again, if you’re creating a simple application it’s not a big deal because the logic is not that complicated needed there. But if you are building more complex applications and you are starting to mix all this stuff it’s super hard to maintain that. And this inversion of control helps you to cut this logic in multiple parts and, well, make it easier to maintain. Miłosz [09:52]: And it’s not that complicated, right? Inversion of control sounds a bit complex but in reality this is accept interfaces instead of implementations. Robert [10:03]: Yes. But we’ll come into that because, well, like always the idea is simple but in practice people unfortunately are overcomplicating that and there are multiple ways on how you can overcomplicate that but we’ll touch on that in a while. Miłosz [10:19]: So we have, separation of concerns of the domain and implementation details and second inversion of control of dependencies. Often there is some concept of layers, which are basically groups of code you use that are dedicated to something. So often we have the Adapters, which are for, you know, external communication of external things like databases or API clients or even for entry points to your applications which we like to separate often. Robert [10:58]: So in other words, it can be calling other microservice or other module. So it it can be database, it can be external API, external by not your company but well, if you are building real microservices, you know, other microservice should be kind of like separate, something totally separate for you. Miłosz [11:17]: And this layer is quite boring usually. It just maps your code onto some external code and that’s it. It should be boring. Then there is some kind of application or use cases or controllers maybe, layer where the logic is or orchestration of your application. And sometimes also domain layer which is this pure business logic layer, which does not always have to be. Robert [11:49]: So the second layer is so again we’ll use multiple names for that because as we said that at the beginning so there are multiple names for let’s say similar idea like clean architecture, ports and adapters etc. So we’ll just use multiple names to be compatible with more. So the second layer is interfaces or entry points or sometimes ports. And the idea is that this is the place where you’re defining your HTTP handlers, your gRPC handlers, your message handlers, even CLI. So any entry point to your applications. The idea is that any operation that is coming into your application, it should go through this layer. And what’s super cool about this, layer is that it’s allowing you to have different entry point for the same operation. So, for example, let’s imagine that you have application that allows you to create a customer and for some reason you need to create customers through message and also through HTTP API. It’s nice because at the end you can do the same operation share a bit less code and support multiple different ways of interacting with your application from external world. Miłosz [13:00]: I really like this approach because, you know, when when you take a look at the project, it’s very easy to see what are the entry points. So it’s very clear where is the API if you want to interact with it. Robert [13:13]: And it works pretty nicely with the pattern that we touched in the previous episode. So MVC. So this is basically your controller from your MVC. Miłosz [13:24]: But, given all those layers, it’s also important to keep in mind that the pattern is not exactly about project structure, which is also easy to, you know, fixate on. So all the layers are more concepts, not exactly doesn’t have to be directories, although this is often the case. But we probably will touch on it a bit later. Robert [13:57]: So the next layer. I think this is pretty problematic layer because I think many people don’t understand what this layer is but this is application layer. So this is the layer that is doing the orchestration of everything. So I think it’s not probably the best place to describe how it can look in practice. At the end, we’ll give some materials that we believe are covering how to implement application layer nicely. But, yeah, long story short, this is the orchestration of what you are doing in your application. And this should be called by your ports. Miłosz [14:31]: Ports or interfaces or entry points. Robert [14:35]: And it should call your domain layer if you have this layer, and your adapter’s layer, long story short. Miłosz [14:44]: So that’s the parts of the puzzle, I guess. The separation of concerns, and inversion of dependencies and layers, right? Robert [14:57]: And I think we forgot about one layer or maybe you mentioned it. So the domain layer. The optional one because I think it’s worth mentioning that it’s sometimes useful but not always. And but it yeah. It’s the layer with the logic. So the layer that should be free of any database logic of any adapters, it should be, relevant. Again, it’s probably something for I think it’s better to read code than talk about that to show how domain layer works. But again, we’ll give you some material. Miłosz [15:28]: It’s probably also not the place to go into details on application versus domain, but you may find some of it on our blog. Robert [15:37]: Let’s maybe take a look on the chat so I’ve seen a lot of messages there so we can catch on it. So, yeah, there are some voices that, yeah, I like it often and use it often and like it. So, yeah, we have some positive things about clean architecture. The second comment, I love that clean architecture and your articles helped me to show how to code is not about language but about structure and functionality. Thank you. Robert [16:05]: We will later link to some articles that we have on our blog. So, yeah, I think it will be just explaining better how those layers work because, again, it’s pretty hard probably to just talk about that without showing the code. What else? Nice topic. I believe that we shouldn’t face architectural choices in dogmatic sense. Definitely. We’ll touch on that a bit later. Miłosz [16:28]: Yeah, we’ll touch on it. Robert [16:30]: Oh, another person missing working with Miłosz. Miłosz [16:34]: Even though we didn’t work together. Yeah, maybe in the future Oleg, who knows. Robert [16:40]: Greetings. Yes, so there’s one comment about overengineering. I see when using interfaces a lot. A lot of projects are created with a single interface each struct. We’ll touch on that also. Miłosz [16:57]: Yeah, I think that’s a very misunderstood part of the pattern like how to use interfaces. Robert [17:04]: And it’s actually, I think, pretty easy to overcome and also to have some way of understanding when needed problem. But no worries, we’ll touch on that. It’s like using abstraction without being abstract. Miłosz [17:19]: Yeah, that’s a good point. Robert [17:20]: I think it’s exactly that. But again, we’ll go into that. As I understand clean architecture, you can’t depend on layers that are deeper one level. So you have to duplicate structures every time that makes me sad. Yes, but not always. I think we’ll also touch on it. Okay, so hi, I work at a big e-commerce company with 10k+ IT area with 4,000 microservices. That’s the scale. Okay, to give some context, there is an area which is a clean architecture for most of their microservices. So this is probably around what we said a bit earlier so Or maybe not. Miłosz [18:12]: And there is a follow-up, I think you need to include this. Robert [18:15]: You can touch on it. Miłosz [18:18]: Yeah, it’s implemented across apps and libs, all Java of microservices. This fact implies that all the areas have an important learning curve to understand and add code here. Yeah. So you mean that there’s, bigger learning curve to understand the code because of it? Which can happen, and we will also cover this. Robert [18:51]: But I think it depends a lot on how you’re implementing that. So again, I would say that it should be some hint like if it’s taking a week to explain a person how can architecture clean architecture work in your company probably something went wrong and it’s maybe not really clean architecture but something around. Because, again, explaining clean architecture and showing it in the code, it should be quite quick. Miłosz [19:14]: Should be, yeah, but not always. I think people always start with this big chart of different circles and shapes and it can be confusing at first. Yeah. I think we will talk about controversies in a bit. So let’s start there. Robert [19:35]: So maybe let’s finish with the comments. So, people write the interfaces next to concrete type when it should belong to the client code. Only including the methods itself depends on common mistake. Miłosz [19:48]: So this is exactly this misuse of interfaces. Right? So if you do it this way, then there is no inversion of control, basically. Or maybe there is but gives you nothing in return. Robert [20:06]: So another thing that we’ll also cover, so no worries, is starting a new project with this architecture. We’ll cover that. How to explain to my colleague that it’s worth using this approach? I think we’ll also cover that. And again, later I will also recommend checking the articles that we’ll link a bit later. So we are also in those articles giving some context where when it works and why it’s, makes sense. Okay. And to catch on the last comment so far, I’m really proud of you guys. Oh, thank you. Because you have decided to run your own company as your main. Now, we’re also pretty happy about that. Alright, cool. So, now, we’re finished with the comments so far. Again, if you have anything to comment in, we’ll be happy to go over it. Alright. So layers. We talked about layers. I already mentioned that it’s not necessary to always use all layers and I think it’s one reason of reason of one common mistake that people are always using the all layers and because of that it’s making it more complicated. So somebody in the chat mentioned that it sometimes hurts because you need to copy structs between layers and it’s a waste of time and it’s true. But it’s mostly true in cases when it’s not needed because you’re just using all layers when its project is not complex enough. Miłosz [21:39]: You start with the big idea instead of seeing where the project leads you. Especially if you start small. Probably, it’s better to let the architecture evolve over time than to force some complex ideas in the beginning. Robert [21:55]: I think we’re always mentioning that that, you know, you need to be pragmatic at the end of the day. So if you see that pattern is creating more mess and require more effort that it’s giving you back probably it’s something wrong because at the end of the day it should help you and, well, that’s it. And it’s connected to the dogmatic approach of that. I think dogmatic approach is pretty often the case why it’s over complicated because often often there are some people that are saying no we need to do it in this way. It needs to be consistent with everything else. Miłosz [22:29]: We like consistency. And I think the topic that we talk about in every episode just keeps coming back. Robert [22:36]: But, if you didn’t have chance to check our previous episode about the low quality code highly recommended because it’s also connected to that. Right, but let’s go back to the architecture. Miłosz [22:52]: So maybe just to summarize, I think you can it’s very easy to to say what will be the the opposite of the architecture. I think that will be having all your code in an HTTP handler. Your logic, your validation, and, your database code is all mixed in the same handler or controller. I think that will be the opposite, basically. Robert [23:20]: And it’s a better approach? No. Miłosz [23:23]: It depends. Robert [23:23]: We’ve been doing that. Sometimes it was good enough. Sometimes, well, I hope nobody needs to maintain this codebase any longer. Miłosz [23:37]: Somehow clean architecture is quite controversial, right? We already mentioned this. And I think for us it was quite surprising because when we started using it, we were quite happy about it and people in our teams were also quite happy. We even heard of them spreading the pattern to other companies as they left our team. Yeah. So why do you think it happens? Robert [24:10]: I have one theory. So I think that the theory may be that we are trying to be extremely programmatic. So I heard that I heard it multiple times. So if we have seen that somewhere it was making our lives harder than more helping us it was probably we’re not using this pattern in this case for you and we’re not blindly following that and I know that it’s opposite compared to other companies. Miłosz [24:39]: You mean if someone forces you to use it? Or maybe someone someone has seen it implemented in a way that wasn’t the best to work with? Robert [24:50]: Or even worse, somebody is forcing you to use that without understanding it. And this is the ultimate combo. Miłosz [24:57]: For the sake of consistency, for example. All microservices in our company will follow Clean Architecture. So, yeah, that might be controversial and that might scare people, I guess. And for some reason, this is tends to divide people. And often I hear something like, you must be Uncle Bob cultist because you use those patterns. So it starts to get almost political maybe, I don’t know. It’s funny. Robert [25:38]: But for what I can see in the chat? I can see that some people actually also see it pretty pragmatic. So, I don’t consider hexagonal architecture over engineering at all. When building a bigger than just couple of weeks project, first, it allows to register stubs for our own implementations, details, and tests. Yep, it’s true. The second comment saying that I believe clean architectural learning is not that hard. Yep, that’s true. And we believe that it’s true. I have seen people moving to similar way of development without knowing those concepts exist. Yeah. But I think we can compare that. And actually, one team will have ultimate combo because we’ve been building a project in Go that and that time Go wasn’t that popular language and 90% of the team was learning Go and Clean architecture at the same moment and it wasn’t an issue. Maybe we’ve just had a great team. It was probably also important reason there and very open minded and great team. But, yeah, at the end, it was cool. Miłosz [26:57]: I think you skipped one. I think you skipped one comment. Robert [27:02]: Did I? Yeah. Hi folks, I’m wondering how well clean architecture matches the lang’s initial idea. From the little that I’ve read about it, I had feeling that Go apps were supposed to be a little simpler. Oh, yeah. Miłosz [27:24]: My favorite topic of keeping it simple and we will touch on this and on Go later. So stay with us, please. Robert [27:36]: Right. Miłosz [27:41]: Those comments are very much about the architecture not being that complex in the first place. I think maybe it is clear when when you learn it. But I think if someone doesn’t know it yet and sees a codebase, then it can become quite confusing. Right? I remember once when our team was using it, but other team in the company was not, and people there were not aware how it works. And for some reason, they had to debug something in our codebase one night. And that was quite confusing for them because if you don’t know the rules, know where the dependencies are, and so on, it can be quite confusing at first. Robert [28:31]: Yeah. But I think important part of this is being kind of open minded because if somebody is like, oh, clean architecture is bad over engineering. I don’t want to look on that. So obviously, if somebody have approach like that, well, it will not work. But if somebody’s open minded and he’s open to learn how it works, it helps a lot. Yeah. But And I think that that was part of the problem that other teams were like, no, I don’t like this idea and why we are doing that? Miłosz [29:03]: We like consistency, as we said. So, you know, if we use the simple Go mindset, for example, let’s say, it can be it can add to this controversy of using it. I mean, that might be one of the reasons. Robert [29:19]: And it’s also worth mentioning that often within a company you have multiple teams solving different problems. So some of the teams may be solving some simpler problems that doesn’t require some advanced techniques and some teams in the company may be working on very complicated domains. And it’s not a matter of using any tool that it’s complicated. It’s just complicated by design and you can help to tackle that, but it’s just helping you to make it simpler. But it has some base complexity, let’s say, that you can try to tackle, but it will be always complex because just logic is super complex. Miłosz [29:56]: So should we talk about some common issues maybe? How it fails basically? Robert [30:08]: Yeah. Because we were discussing this topic a bit earlier with people on some social media or in real life. Yeah, it was a very common topic to understand why it’s not working in multiple projects. So we gather multiple reasons that we know that it’s leading to making the architecture overcomplicated or even project failing. By the way, if you have any, any stories from you, also please leave it on the chat so it will be also cool to hear about that. Right. So the first thing is, over engineering architecture. So as we said multiple times and it was also mentioned multiple times on the chat, clean architecture is not that complex pattern. So you’ll have three, four layers. It depends on the complexity of your application. You’re ensuring that layer from inside doesn’t know about layer of in the outside, and you’re using interfaces to do inversion of control. That’s it. But, unfortunately, people are going sometimes a bit too far with that. Miłosz [31:14]: Yes. Because what you said doesn’t sound like proper architecture. It reminds me of this, what we discussed in previous episodes about this belief that we can figure out the architecture now, and in the future, we will just cookie cut the services or applications. It sounds a bit similar to me that we try to figure out the perfect architecture first, all the rows in place and layers and so on, and then just apply it. It sounds much much better than just following some simple rules of using interfaces. And it’s easy to go too far, right, and become a purist and just force this idea everywhere. Robert [32:06]: So, overconsistency, let’s say. Miłosz [32:09]: Yeah. This is the hard part, when you want to be consistent and when there is too much consistency. Robert [32:19]: It’s always a cost. so the second thing that I have in mind about clean architecture and I think this is probably the most common one so it’s and I think somebody mentioned it on the chat so it’s using too many interfaces. So in general, the interfaces are needed if you are kind of communicating between layers so those layers inside will not know how what’s implemented then. But that’s it. But very often I’ve seen that within the layers before putting more interfaces. For example, you have domain layer and under the hood it has multiple interfaces. And what’s even more funny often it’s under the hood have only one implementation. And it’s pretty problematic because one, it’s harder to inject that because your dependency injection is starting to be super hard. And this is something that is also probably common thing common thing between between multiple episodes because people were saying that they have complex of complex problem with complex dependency injection and from my experience, it’s often the problem is that your dependency tree is too complex. It’s not a problem with some dependency injection framework or whatever you are using. So if you’re not able to inject your dependencies by hand, probably something over complicated. Miłosz [33:37]: It’s part of the over engineering, I guess. You know, just forcing the ideas because there’s a rule that says accept interfaces because you might want to change them with something else, and then you create some domain service, let’s say, that accepts some some structure. You might you might go and think, oh, maybe I will use an interface here because I am supposed to use interfaces, right? Maybe in the future we will want to change the implementation. And it’s exactly this trap that you take the base ideas and just use them everywhere even when it where it doesn’t make sense. So maybe I no. A good a good a good heuristic here is to think of, what what you want to use mocks for in your tests. That’s probably what you want to use an interface for. Robert [34:30]: Yeah, and obviously between layers but this is a different case. And I think it’s very important to mention that, okay, if you are mocking things you should probably mock only things that are IO in general. So it’s database, some external API. You should mock not mock your domain logic because you have no benefit of that. So your test will be worse because you will be not checking the, the real code plus well, it will not speed up your tests basically. So mocking makes sense in most cases when you need to speed up your tests or or simulate some external logic. In your domain code, you should be able to do it through your public API. Miłosz [35:13]: Exactly. You can tell you went too far if you’re, you know, if you use too many mocks and all your testers do are checking some function calls on the mocks you used. It probably means you went too far and you don’t need to mock that much. Robert [35:30]: Yeah, but there are some exceptions. So there’s no silver bullet. So exceptions are very very very complex domains. So sometimes you may have some complex domain and within the domain layer, you may have interfaces to hide this complexity somewhere. But you never should start with that because it will go in the wrong way. So I guess, I don’t know, within one year probably it will be not possible to have such the complex domain code to, need to have really layers within layers. A couple years old codebase that is super complex, maybe, but I would watch out for that. Miłosz [36:07]: Layers within layers actually sound like the Great idea. Yeah. Like the You’re so smart. The next pitfall, right? It’s again over engineering in the layers area. I remember this Reddit post. You know which one? The other day at the Golang sub subreddit there was a Reddit post where the OP said something like, Don’t use pin architecture, it’s over engineering. And of course it, you know, invited me to click. And in the content they said something like, you should have at most three layers in clean architecture. And I thought, yes, of course, what’s the problem? Again, this is how different the implementations can look like. I guess what this person found in the code was, I don’t know, six layers? I can’t even imagine what was there. But that’s another issue, you know, if you just take the idea of layers too far. Robert [37:24]: Another surprise. I mean, if I would work in projects where you have six layers at least, I would also hate using that and would love to go with a simple approach, obviously. Miłosz [37:36]: Yeah. So maybe it’s good to start with the basics. So you want to separate the implementation details from the logic, from the domain code. And that’s two layers for starters. It doesn’t have to be much more. You can separate the entry points if you want. That’s kind of a hard layer, but I would call it part of the adapters as well. It’s a bit similar. I’m not sure why would you use something beyond the domain layer. What’s the fifth one? I don’t know. Robert [38:16]: Maybe you know somebody is using layers from hexagonal and onion architecture. Miłosz [38:23]: Yeah, maybe. So if you see more than four, I would be very, very careful. I would ask some questions. Robert [38:33]: Definitely. The next thing that I have in mind about the common problems about clean architecture is using it everywhere. So one point that we’d like to, you know, give you with this live podcast is that there’s no one single way to use something as we’re mentioning, in the intro. So if somebody is saying you always do x or never do y, It’s pretty big chance that this person just work in some specific type of projects, but it’s not universal advice. Miłosz [39:09]: Or they want to sell you something. Robert [39:11]: Or this. Some shortcuts as usual. But what is the message? So the message is that, you know, clean architecture is nice, but it doesn’t matter that you need to use it in an entire company. So the thing that I mentioned a bit earlier. So often in company, you have teams and services that are complex and simple and using the same approach for everything is just crazy because in some places it will be over engineering and sometimes it will just suffer from from the complexity that is not handled properly and it will be also the problem. Miłosz [39:51]: Sometimes even in the same project in different modules, it also has some trivial domain that doesn’t need it. Just what we have in the White Workouts, our example project, the Users module or service is basically a Scrub domain with no layers at all. Yeah, and it’s different for your past projects because if Robert [40:14]: You are working on something after work, it doesn’t matter that much. I think it’s worth trying multiple approaches to have your opinion on how it works. But in a job, it well, it’s worth to be pretty mindful. And if your project is not complex maybe clean architecture may be not needed but sometimes it may be worth with starting with some simple one without domain layer, for example. It doesn’t cost that much. But if you know that your team is bigger, then it can make sense to start with that. Miłosz [40:46]: Yeah. And and some apps are are may most of them are not be a good fit or maybe if you create a library, for example, maybe it’s not the best idea because the users might not be familiar with it or for things like CLI apps maybe or something infrastructure related. Robert [41:06]: And you if you think about that, so libraries are usually living in inside of some other layers. They are not covering multiple of them. So usually, it’s probably adapters or ports. Like, for example, Watermill library that we’re maintainers are also It’s living within ports and it’s not going to other layers basically. Miłosz [41:26]: Yeah, exactly. So you don’t want to tie it with your application logic or the domain. A similar one is Project Structure, which we mentioned before. It also keeps coming up in our previous episodes that it’s easy to focus too much on the project structure in the beginning. Again, it’s this idea of, you know, let’s prepare this great environment, great architecture, and then we can just keep adding features. Most of the time it’s not that easy and the project will evolve anyway, in some way you don’t know yet. And you can always refactor, right, if it changes. So I wouldn’t start with thinking of the, you know, what’s the perfect project structure or what’s the perfect number of layers. Just start with the basics, something simple and then refactor as you go. Robert [42:40]: And assume that you may be wrong. I mean, we have almost twenty years experience with coding in general. Also pretty much years, you know, with architecture and still we are not always right. But I I think it’s important to know how you can be not right and how to handle that and not go too deep into the wrong architecture, basically. Miłosz [43:05]: It’s probably not possible to invent the perfect structure in the beginning anyway. Robert [43:16]: If somebody is saying that he can do that, don’t trust this person. Miłosz [43:21]: They want to sell you something. Robert [43:25]: And overall I think it’s if you’re thinking about some pitfalls of architecture or problems with that. So the most important thing that I believe made our teams happy about using this is keeping in mind that it should be helpful for you. So the goal is not using any kind of architecture. Miłosz [43:44]: Oh, it’s a big one. Robert [43:47]: But the goal is to use something that is useful for you. It supports your velocity because at the end of the day, every technique that we’re using, it should allow you to code faster and that’s it basically. So if you are using some architecture and it’s slowing your down, you cannot iterate faster, it’s probably the bad one. Miłosz [44:12]: I mean, it’s very easy to start being dogmatic about it, to become like a purist. And it is tricky because we have a bigger team and it helps to have this consistency around the project. So Clean Architecture gives you this consistency in supports. But then there might be places or features where it’s not a great fit. But if you want to break the rules, then someone needs to decide if it’s fine to break the rules or not. And it often starts those endless discussions about it doesn’t follow clean architecture or it breaks. Robert [44:57]: Which layer it should be in? Is it domain logic or it should be in domain or application layer? Miłosz [45:03]: Okay. Let’s call it a meeting. A meeting to discuss is super important. And sometimes it is important but sometimes it’s not. It’s not. It’s just, you know, a detail. Robert [45:14]: And, unfortunately, this is the topic that we cannot give cannot give you a simple ask for like always do x. The best way, unfortunately, here is to just do it over time and learn by doing and probably this is the best advice that we can give here. But, well, it’s hard to have any shortcuts here because there are just many different kinds of applications. And, again, it’s important to be open minded. Also listen to your teammates because it’s also easy to be, you know, the architect or the person that gives architecture. And if you are going to holidays and going back, you see that it was all removed. It’s probably the sign that. Miłosz [45:54]: Have to change everything. Robert [45:55]: So, you know, it’s also probably the sign that, okay, maybe something that you invented is not that helpful for people because if it would be if it would be helpful for them, they would not remove it during your holidays. Miłosz [46:08]: I think this this factor is for sure part of the controversy around it. Because if you if you have been in a team that, you know, spent hours discussing in which directory to put some some codes, then you you might assume you don’t want to work with Cleanarch anymore because it’s just it’s too much hassle to to keep discussing how to use it instead of this right code. Yeah. And especially if you didn’t have the proper onboarding for new people Robert [46:39]: In your team because often, you know, people are changing teams, changing companies. And this is also important thing to ensure that, okay, the new person is joining the team, the person will know how it works in your company and how to navigate about that. Because if you will not do this, the person may be not happy about it. Miłosz [47:00]: Yeah, it’s probably best to show them instead of having some documentation because I think it’s easier to show it as an example maybe than reading about something. Definitely. But on the other hand, I think clean atch also enables some nice onboarding because you can have those you can have new joiners implement something quite simple because of the separation of layers. Like, you can give them an HTTP handler to implement and it will be a quite simple task. They can probably do it in one day and get started with the codebase. Robert [47:41]: And again, we’ve mentioned it already a couple of times but if explaining to some person your clean architecture takes more than half hour, probably it’s something wrong there. So we can say probably it’s some not very accurate metric but more or less like this. Miłosz [48:06]: So overall, don’t ignore developer experience. Think this would be the summary. Yes. I think it’s very important because it’s basically what everyone will be working with daily. So you just don’t make a religion out of it at the end of the day and don’t try to use arguments like this is not clean. It’s the worst worst kind of discussions you want to have. You don’t want to have them. Robert [48:39]: It’s probably worth mentioning to also set some expectations. So if you have everything in your HTTP handler your dependency injection can be as simple as it doesn’t exist. If you have clean architecture, it’s more complex because you need to inject more things. But again, it should be manageable by doing it by hand and okay, it depends on the language a lot. So with Milosz, we are mostly working with Go where there is it’s compiled language. So it’s a bit harder to do some magical dependency injection. But it’s actually quite good because because you need to do it. So in our opinion, it’s easiest to do it by hand. And if you’re in the position when it’s too complex to do it by hand, probably it’s showing you the problem with your maybe clean architecture, maybe architecture in general, but it should be doable by hand. In other languages, we know that it depends. So we also worked a lot in PHP where it was working automatically. Sometimes it was nice but it was harder to debug with more complex dependency tree. Miłosz [49:48]: There are some libraries usually. I’ve seen someone in chat mentioned fx. We haven’t used this one, but we used wire from Google. It was okay. But currently, we just wired wired it up manually in in the main function, basically. And it it grows. It’s big. But most of the time, that’s not an issue, really. You just fix it and if it compiles you’re good to go. Robert [50:18]: And I think it’s one risk maybe that some people maybe don’t see at the first sight that but, you know, it’s something making your it’s something making your super complex dependency tree easier to handle. It’s actually opening some kind of gate of hell because you can do we can make this dependency tree more complex if something is helping you with that. Are going deeper and deeper into. Miłosz [50:42]: Those layers within layers. Interfaces within interfaces. Robert [50:46]: If you are doing it by hand probably you sooner see that something is wrong there. Miłosz [50:53]: It’s also a good general rule to start fixing the pain once you start feeling it because if there is no issue maybe you don’t need all these solutions. Robert [51:03]: And if you are starting to blame dependency injection framework probably the problem may be not in the dependency injection framework but between laptop and wall. Miłosz [51:17]: Tough pill to swallow, but sometimes, yeah. Robert [51:20]: Yes. Firmino is actually mentioning that even in Java, dependency injection can be easily done by hand but developers could use using auto wired from Sprint. Yep. That’s true. But again, we’re also not that against using such frameworks but again, you need to watch out. It’s like with every tool. Okay, not every tool but there are some tools that are encouraging you to do bad things and probably dependency injection framework is on this list. But again, it all have its use cases like ORMs also, I would say. So many people also doesn’t like ORMs because it can also lead to some issues. But, in my opinion, it’s like with knife in the kitchen. I mean, you can hurt yourself with a sharp knife. But from other side, it’s also pretty useful. I like sharp knives. I think you also like You need to just watch out. So that’s it. It’s much better than using blunt knife. And, yeah, the last thing that can lead to problems with clean architecture and we have some solution for that fortunately is using the wrong resources. So many resources that we’ve seen in the internet are unfortunately missing the point of clean architecture and it’s encouraging to use it in the wrong way that is far from being pragmatic. Probably doesn’t help that many of those clear architecture hexagonal onion architecture they don’t have super strict rules on how to implement that and it’s nice because it’s giving flexibility but it’s not nice because it’s giving flexibility on what you Miłosz [52:58]: Have to decide for yourself. And it’s exactly where this starts getting difficult because you’re just starting out and there are no, definitive rules. And this is where you start thinking, okay, what’s my perfect project structure? Which can be a trap. And I think many of those open source examples often miss the point or they show some very trivial domains like, you know, to do apps that have two entities and two endpoints. And if you take a look at this as such a simple domain implemented in the architecture, it will look like over engineering, of course, because it makes no sense if you can fit this into files to split it across the project. So you have to be careful of this. Robert [53:53]: But there is also one, tricky part about this because so we have run repository in Go where we are showing how to you can connect everything. And I think it’s, you know, fully worked application. It’s not that complex still because it’s still some more pet project even if it’s fully functional. But also, if project is too complicated, it’s also hard to grasp the most important parts of that. So it’s it’s all about the balance. So as mentioned multiple times, at the end of the we’ll give you links to the useful resources. But, yeah, it’s important to just use those resources. Miłosz [54:35]: Yeah. I see an interesting question in the chat. So does clean architecture fit in projects with lots of tiny and simple microservices or better fits in a monolith application? Robert [54:51]: It depends. There is no silver bullet. Miłosz [54:54]: Yeah, what’s a simple microservice? I guess that’s that’s the question. I I hope if you have, a monolith, it is split into modules just like those microservices, and then it’s pretty much the same answer. If if the module or service is complex enough, to benefit from Cleanark, maybe a good idea to implement it. If it’s truly trivial and crowd domain, probably not. Robert [55:24]: Yeah. But I think there is one risk that I’ve seen multiple in my life. You’ve seen it as well. So because microservices are cool for splitting complex problems into smaller but the problem is that often people are splitting it to granular. So you have multiple so instead of having one service that can handle some, use case you have five, ten microservices that are handling that. It’s okay. Maybe every service is pretty simple and you don’t need to have clean architecture inside of that. But each time when you’re implementing a new feature it takes changes in five or 10 microservices. Miłosz [56:03]: Yeah, because because it’s different entities, I guess. Yes. Yes. My favorite from the past is the Avatars microservice. Stored Avatars. Robert [56:14]: And, you know, if you have sub granular microservices, okay, and then you split at your complexity but if implementing one functionality takes changes in 10 microservices, I’m quite sure that it will be much easier to implement it within one service that is not maybe microservice, whatever microservice is. But doing it in one service will be easier. And okay, this one service will be more complex but Clean Architecture can help you with, handling this complexity because you don’t need to split this complexity in multiple services. You can instead split it by layers. And it’s making your so this is the thing that we’ve done multiple times. So it was 10 microservicization operations. So moving multiple microservices into one service and obviously this one service was more complex but clean architecture was helpful to handle this complexity because we didn’t have one big sum of everything but it was data in multiple layers and at the end implementing features was much easier. So, yeah, I hope it’s useful. Miłosz [57:21]: Yeah. Looking at the chat, I’m gonna read the next one. So, Tony says, For me, the most difficult part of clean architecture is when implemented in a team and different people diverge in some details and the mess appears when little by little the project becomes less consistent. This is exactly the developer experience issues we talked about previously. I think it’s very hard and probably what can help is having some team leader in place who can just cut the discussions and make a decision. Because otherwise, yeah, you can take long hours discussing, and it’s natural that some people in your team will be like consistency more. There are many developers who strive for this consistency and like to have everything neat and tidy. And if someone starts doing something in a different way they will be annoyed and you want it to be the same way. And I think it’s probably worth mentioning that, Robert [58:33]: At least, I believe that democracy doesn’t work in programming teams, unfortunately. Miłosz [58:38]: Yeah, unless you have infinite time and can spend days discussing and voting. Robert [58:42]: So probably sometimes usually something closer to totalitarianism works a bit better. But better to not overuse that obviously. But it sometimes helps with these discussions. Miłosz [58:55]: It’s a tough balance, I would say. You want some discussion with the team but you also need someone to say, ‘Guys, let’s just focus on important stuff.’ But it’s very hard to make those know, exceptions because if everything is consistent, why would you use a different way in this in this case? You know, maybe it’s just more easy this way. Maybe it’s fine. But if you do advanced then, no, people exactly what what the comment says people might think that, oh, okay. It doesn’t need to be consistent so let’s just do whatever. Robert [59:38]: And then also, if you’re not a team leader but team leader is doing a bit of totalitarianism when you are going too deep into some discussions also don’t be mad at this person because we’ve been in this position and, you know, it’s also not that hard. But well, after you are in this position, sometimes you see that it’s easy to overdo some discussions and that’s it. So sometimes at the end of the day, it’s better to have something that works, but it’s not perfect, but it works because sometimes you can throw it out. Probably you can listen more in the previous episode about code quality. Miłosz [01:00:13]: Yeah. I think it’s one of the most difficult parts of implementing clean arch. We don’t know definitive answers here. Robert [01:00:24]: Yeah, there’s also a nice comment about frameworks. Frameworks like Rails that define a clean directory structure seems to be harder to apply that architecture and I would say it’s still possible. So I don’t have that much experience with Ruby on Rails but I worked more in PHP with PHP frameworks like Symfony or whatever and those frameworks usually okay, it requires some effort but usually you can hide them in some layers. So mostly those frameworks are about your ports but it’s also sometimes a bit application layer but it’s doable and it’s often worth doing but often it’s also not worth doing. So it’s getting probably hard to give some universal advice here but you can do that in general. And sometimes it may be beneficial for more complex domains because those frameworks are nice and we’re discussing it in the previous episode also about frameworks. But long story short is that they can give you some velocity at the beginning but later it may be harder to do some more context scenarios. And if you went with this framework structure all in, it’s much harder to customize that as and sometimes, you know, the product after a couple years, it’s still within the framework and we work sometimes in such companies and it’s even impossible to get rid of this framework because it’s so integrated with your code. And if somebody will split that in the screen architecture way, it will be maybe possible to replace that and maybe just not use it at all. And it will be possible to develop this, product further. But often, unfortunately, it’s something like that. And it’s super hard to develop the product further because it’s so limited. But it’s probably worth mentioning that often those projects were successful because at the beginning they’re they’re managed to have free fast velocity and win the market. Miłosz [01:02:22]: Yeah. That’s that’s the benefit of using them. But if you can introduce even some, you know, thin domain layer or something that’s separate from from the framework, that might be helpful if you want to move on. Not to use everything it gives you. But, of course, then then it’s kind of, you know, defeats the purpose of of using the framework in the first place. Robert [01:02:52]: Right. So we talked about, what are the common issues of clean architecture, what are the contraversions, but I think there are also some upsides that we’ve seen multiple projects and some reasons why this maybe was invented because it’s like with many techniques like it wasn’t invented for no reason. Miłosz [01:03:14]: Yeah, exactly. So the why behind it is important. So maybe let’s start like we did, yeah, with the burning it down with separation of concerns. And, you know, it’s the reason we we want to separate those concerns is this this logic independent of implementation details. Right? And, basically, the what it gives you is you’re you’re able to easily replace one implementation with another, which we talked about about mocking. That’s one use case. Super useful. But also you can replace it with something completely different like changing the database. Robert [01:04:16]: It’s quite interesting because you can do that, but how often are you doing that? Miłosz [01:04:19]: Yeah, that’s a good question. It’s often a counterargument I hear about clean architecture or interfaces in general. Like, yeah, yeah, you talk about how it’s easy to change from one database to another, but in reality, it never happens. But it happened to us recently, so that’s one example when it was useful. So we switched from Firestore to Postgres, and using Zaprouted was quite easy. Another nice example we did, in the past was starting with in memory implementation. So creating a domain domain code with some very simple, in memory implementation of the database. And then later when we know this is what we want we can just replace it with SQL or whatever implementation. It’s also quite nice. Robert [01:05:22]: Yes, but I think it’s also worth mentioning that this is not the end of the architecture. So the end goal so it’s designed. So if you can replace your database or port easily. It means that, okay, you did a good job with implementing architecture, but it’s only showing you that it’s done properly. But it’s not the end goal in general. Miłosz [01:05:51]: Yeah. In reality, you’re not going to be changing the database every few weeks. So it’s it’s not why you do it. But I think but it gives you this nice separation of, you know, the implementation details from from the interesting code in your in your project. Robert [01:06:11]: Yeah. And why it matters? So it’s mostly because of, cognitive overload. So as I mentioned a bit earlier, so when you have multiple microservices that were too granular, you are joining it into one service. If you would be all in your handlers probably it will be very complicated. But if you’re splitting it in multiple layers, it’s allowing you to so as long as you know how the clean architecture works, it’s it’s making you a bit much easier to reason about the code base because you see what are the entry points of your applications, what use cases are supported, and how you’re interacting with the database. If it’s everything here in your HTTP controllers, for example, it’s super hard to understand that because you can have some query that is called in some weird way, but it’s hard to backtrack how it’s called. Miłosz [01:07:03]: Yeah. And and usually it spills over somehow to your to your logic. Robert [01:07:07]: And it’s nice if you have some transaction, but often it’s not. Miłosz [01:07:15]: And if you don’t have the separation of any kind, it’s also very easy to lock yourself in with some framework or library because, you know, you just scatter the library method calls all over your domain code. And if you want to migrate later you have all those places you need to change which is much more complex. Robert [01:07:39]: And there is one interesting comment about abstracting the database layer, that it makes sense when you need to support different databases. So you need to choose what to deploy. And it’s a very good point. And once in our life, we actually did something one level higher. Long story short, we worked, with the product that needed to be deployed in two ways, as microservices and as monolith. And it wasn’t some it wasn’t some crazy architectural idea. It was a pragmatic thing. So it would have, two deployments. So one in physical, servers and physical device and one in the cloud with the kind of simulated device. And because of that, it needed to be deployed in this way. And architecture was quite cool because of that because it was a matter of having two set of ports and adapters. One for the cloud deployment and one for the more bare metal deployment. And it worked Mhmm. Pretty nicely. And it thanks to that, it was manageable to do this. Miłosz [01:08:39]: Yeah. That’s quite quite a specific approach, but that’s a good fit for for clean architecture in general. Robert [01:08:46]: Don’t do that. You probably don’t need to do that, but, yeah, sometimes it’s useful. Alright, so the first thing was the cognitive overload. But the second thing is that help is much easier with clean architecture is testing. And like cognitive overload which is important for a more complex project with bigger team, etcetera etcetera. The same goes for testing because when your, project service or whatever is small, probably doesn’t matter what you test. I mean, you can probably just test it end to end and you can do that. But if you are working in the application that have thousand of use thousands of use cases, a lot of complex interaction with your database, testing it end to end is impossible because probably most of the use cases you cannot trigger with the public interfaces and it will be super super slow. It’s especially, true for applications where the domain layer is very complicated because you can have even thousands of use cases and test cases and running it with the database, it will be just super slow. So mocking the extra level is the must because in other, I mean, obviously, you can wait a week for running test suite and I know that such application exists but probably it’s not something that you would like to do. Miłosz [01:10:09]: Yeah, and often having parallel tests in the same database is not often trivial to do. You can have conflicts between the use cases and so on. Robert [01:10:19]: At some point it will be slow. I mean, obviously you can scale it horizontally but Yeah. Miłosz [01:10:24]: There’s often this argument that you can you have Docker, right, or test containers. You can just spin it and run all on this infrastructure. And it’s true. That tests are so critical, you know, path of local development and CI that you probably want them the fastest you can. Robert [01:10:45]: Exactly. So for example, if you’re changing some important functionality, you don’t want to then wait ten minutes every time when you change and change that. You would like to wait at most one second and see. Okay. It works. Oh, no. And thousands of unit tests are running and checking that. But it’s the thing that it’s again important for bigger projects, not all projects. If your project can just have end to end tests, it’s fine. I mean, we worked with some projects. The main project that we’re working on now in Tridos Labs, the Academy, is at the at the beginning we just have end to end tests for a pretty long time. It was just enough. So Miłosz [01:11:25]: Yeah. I mean, if we were if we had unit tests, it would be even better. But sometimes you have to optimize for delivery speed. Exactly. So that’s what we did. Yeah. Speaking about tests, maybe we can just quickly talk about how we speed them. If you have multiple layers, you you probably don’t want the same testing approach for for all of them. So, for example, as we mentioned, for the domain layer, unit tests are are great fit because it is purely logic. There is no usually little dependencies that you can mock out, and you can have many many test cases that run fast using just viewing tests. That’s that’s pretty nice. And on the other hand, the adapters usually are a good fit for integration tests. So this is where we run some Docker database, for example, and just test the repository or the client or whatever it is in in separation from the domain. Right. And for the entry points and the application layer, we usually have little tests here, if any, because if all it does is translates one layer to the other and orchestrates things, it’s probably not no big benefit in testing it. Robert [01:12:52]: And I think it’s actually one common issue that we’ve seen multiple times and I think we forgot about mentioning that. But, yeah, if you have domain layer and application layer, running a lot of unique tests for your application layer probably doesn’t make sense, especially if you’re mocking everything around because it’s Miłosz [01:13:10]: Yeah. There are all kinds of tests. Especially if you have those mocking libraries that let you assert whether a function has been called, then your your test becomes become basically an application of your code. And every time you change anything, you have to go to a test and adjust it and basically gives you zero value. Robert [01:13:31]: Yeah. I think it’s actually great symptom, let’s say, of why you have the bad tests. So when you are copying your logic of your logic in application from application error, for example, literally to the test. It’s it means that it’s something wrong with your tests. Yeah. And especially if everything around is mocked. But I think there is one nice, type of tests that are solving this problem nicely. Miłosz [01:14:01]: Which is? Robert [01:14:02]: Components tests that I think at least from my experience not many people knows of but long story short, idea of component tests is that you’re mocking everything from external word and usually you’re using so you’re running your application in for example, Docker, you’re mocking all external words or external APIs, everything. You have just your database in the Docker and maybe and everything else is mocked. And you’re calling public APIs of your application and that’s it actually. So, it’s different from end to end tests because you’re not integrating entire application. You are just having actually stops for the external words and it’s nice balance because you don’t need to, have super big environment to integrate everything with the downside that obviously you may mock some wrongs in the wrong way. But this is this standard problem with mocks but it doesn’t mean that you shouldn’t have end to end tests that are integrating everything. But you can cover much more cases than in end to end tests because you can define behavior of, the external APIs plus you’re also checking the integration of everything. So it’s covering a bit. Miłosz [01:15:18]: Yeah. Probably the biggest benefit, Maybe the benefit, but the characteristic is that you don’t call other services in inside those tests. So you have to somehow stop them or use numerators or something like that. Because you you want to test this one service end to end basically basically not the entire infrastructure around it. So it’s also quite fast thanks to this. Robert [01:15:46]: Yes. And it’s much easier to also run local because if you are running end to end tests with entire infrastructure, it’s not possible to run it on your laptop. And we’re big fans of being able to run as much as you can locally and test against that because it’s much easier to and faster to iterate over that. Miłosz [01:16:05]: But no one enjoys, like, another team service breaking the test URL locally. It’s the worst experience. It’s not productive. So if you want to learn more about, testing this architecture, we have also a blog post about it, and we will link it in the description. Maybe another, like, like, less technical benefit I like about clean arch is, how it changes your mental model over time. So it gives you this kind of consistency in the project, so it’s easy to find what you look for. For example, if you if you want to add a new API entry point, API method, or message handler, you know, to which directory to, you know, to go and which files to edit. Robert [01:17:10]: And this is actually the often the counterpoint of many people that’s, yeah, I’m looking on this clean architecture project and I have no idea where everything is. Come on, spend fifteen minutes to understand where it is. Miłosz [01:17:23]: Yeah, if the project is big enough and you work with multiple people on it, and this investment in learning, you know, the structure probably pays off, especially if it’s consistent with how you do other projects in the team. And I think it also gives you this if this mental model spreads among everyone in the team, it gives you this nice vocabulary to to use. So, you know, when you discuss how to implement on some feature and someone says, okay. We need SQL adapter. Then everyone knows what is this, and it’s quite clear how to implement it. So it’s it’s very easy to, to discuss the the project. Robert [01:18:14]: And then what’s useful in this because I know that it may take some time what layer you can import to what layer and works forbidden. The good, the good news is that there are leaders that can do that usually. So, for Go, there is go clean arch linter I written a couple years ago and it seems that it’s pretty useful for many people. I guess it have probably around thousand stars in GitHub so probably it’s useful. Often you have also linters I think in Java like, ArchLint or something like that that you can define some, I think it’s in Java and PHP. So you can define what are the modules interaction. So, yeah, it’s recommended to check for that, and it’s pretty useful to because you can ensure that the dependency between, layers is properly handled. Miłosz [01:19:10]: Yeah. In in Go, it’s especially useful to have this clear direction because you you can have import cycles. Your code won’t compile otherwise. So if you have some, you know, some clear way to to import between packages, it becomes just easier to stick to one way and don’t worry about it again. So that’s that’s quite nice. Maybe maybe the next one would be what I also like about cleanarch is how it becomes easy to extend the application over time. And it’s a it’s a bit similar to this idea of having, you know, great architecture from the start. But I think it’s a bit different because you don’t really need to have all the use cases. But thanks to this clear structure, for example, it’s easy to to add new SQL implementation or or or, for example, to to change one database query with another. That’s that’s faster. And the impact is very low because I mean, it’s the the plus radius is small because you you don’t touch multiple files and areas. You just change this one little part and it works. So you don’t need to think of all use cases in the beginning, you just adapt over time with new adapters, with more application use cases, And this way you avoid side effects. In contrast, if you have this huge HTTP handler with everything in it and you make some changes to it, you don’t really know what can happen. That’s, you know, the worst scenario you can have when you work with a project you don’t know well, and you see this huge method of thousands of lines, let’s say, and you make some edits but, yeah, who knows what happens next. We don’t really know this. It can affect many, many areas. Robert [01:21:25]: And, yeah, the last upside that it’s coming into my mind is the one that I think many people doesn’t think about, but we used it pretty heavy in previous teams. So it’s how how it helps you to split work. Miłosz [01:21:41]: Oh, yeah. Robert [01:21:42]: Basically, the idea is that you can plan some work so you have some feature to implement. You know that you need to implement it pretty quickly as usual. It will be great to parallelize your work as much as possible. And how we’re usually approaching that was, agreeing on the interfaces between layers and just splitting the work. So, for example, we had some feature to implement. So we agreed that, okay, for example, we need such interface to integrate interact with database. This is more or less what we need to have in the domain layer and we could split this work for four people, for example, and each person was responsible of implementing one layer and it was quite simple after they agreed on one interface. And thanks to that, implementing one feature instead of taking, I don’t know, three weeks, it was possible to finish it within one week including deploying that. And it was most important for features that we needed to just implement quickly. And this created some nice separation for each person and they were sure that we’re not, not conflicting that much. And, obviously, sometimes there were some changes into the interfaces because, well, you will not be always sure always right from the beginning. But assume that you are talking to each other. Miłosz [01:22:58]: Yeah. But I can just it gives you this nice vocabulary to use. So, for example, when you when you split, the feature into smaller tasks, someone can say, okay, I will take care of the HTTP layer. And it’s super easy for them to, you know, create this API definition that will just do nothing in the beginning but will be merged, can even be deployed, and everyone else can integrate with it. Robert [01:23:24]: And you can also split by complexity often. So, for example, you have some junior person, some principal person, and some mid developer. And more or less you know how much complexity will be in each layer. So usually, for example, your HTTP layer shouldn’t be that complex usually because sometimes you have. Sometimes your domain logic will be very complex. Sometimes it will be quite simple. But for example, adapters will be complex or vice versa. And it’s also very helpful to kind of assign work properly. Miłosz [01:23:56]: Yeah. And it helps helps you create smaller tasks and PRs which is also a great thing. If you have, you know, a one file change in SQL adapter, I can review it in two minutes probably or something like that versus this whole feature where you need to focus on everything else. I think it’s quite unique overall, this approach. Mhmm. Because I think in most teams, you would expect to see something like, Okay, Robert, you’re supposed to work on this feature now. So see you in two weeks and you will show us what you have. And then you will come up with a PR that’s, like, I don’t know, 4,000 lines long. And then everyone spends reviewing it two days. Robert [01:24:47]: And you have 100 comments like why you did like that or I don’t like this. Exactly. I think it’s even a good situation because often you will also struggle to find somebody who will do your reviews because this is the problem often when you are not working as a team but a set of individual stuff. Yeah. Your, goal is to your goal is to implement feature. It’s not a team goal. And if you are working at it as a team of four, for example, or five, you have always people to review that and work is just going much, much, much faster. Yeah. But important disclaimer again, it’s for, as we mentioned earlier, it’s for more complex projects with bigger team. Again, for a full team, it’s not a big team but it’s not two or one person team, obviously. And we assume that it the features are complex enough to take one week of four people. So Yeah. Also important to keep in mind because, you know, sometimes there are features that one person can implement within one week. And using four people to implement that with green architecture, it will be just over engineering probably, and it will be just waste of time. Miłosz [01:25:55]: Yeah. But it’s good to keep in mind it’s possible to to speed work, actually. I think it’s not the the default for sure in the industry and somehow we accept that, you know, it’s normal to just have someone sit on a picture for two weeks and then review what they did. Or other two weeks? Yeah. It’s really not that productive. And sometimes if you have the layers especially, you can have three people or four or two work on the same thing and you know, do it in parallel on different areas. It’s super effective because you have those small small PRs, smaller, bus radios of of the changes. You can deploy early often and all those good things. Robert [01:26:42]: Yeah. And I’m not sure if you remember but I remember this feeling like just work flowing. Miłosz [01:26:48]: Yeah. Robert [01:26:50]: Yeah. It was just pleasure because it was super effective and even very complex features were able to, you know, deploy it within one week. So it was cool. Miłosz [01:27:01]: Another benefit that’s kind of not relevant to Clean Arch, but you have those small subtasks in each feature and you can see how it moves on the board every day, you don’t really need stuff like estimations maybe because, you know, whoever is responsible for the product or any stakeholder can just take a look at the board and see, okay, five out of eight subtasks are done in two days. So probably in two days they will be done with this. Okay. Robert [01:27:33]: It requires the team to have a pretty big, let’s say, engineering culture to achieve that. Miłosz [01:27:39]: Yeah. Robert [01:27:41]: It’s very highly recommended to at some day work in such a team. Miłosz [01:27:46]: Yeah. I think that goes a bit beyond clean arch. But I think clean arch is what really enables it. If you don’t have this clean separation in the code of different layers, then it’s very hard to split work because you will just keep getting, you know, in like, merge conflicts are the probably the least issue, but just conflict of who who works and what and how it impacts each other’s work. Robert [01:28:19]: Yeah. Yeah. Probably it’s a topic for a separate episode how to do this splitting because it’s also not obvious how to agree on this, let’s say, one week work and everything like that. But definitely we’ll touch on it at some point. Miłosz [01:28:34]: So we talked about the benefits, talk about talk about the issues. The the question that’s asked very often when someone asks about architecture is Can we just do it simple? Robert [01:28:49]: And I think it was even mentioned on the chat. Miłosz [01:28:51]: Yeah. Especially on on the chat, I think it was related to Go, which is a very good point because Go is seen as this simple language, you know, in contrast to Java, I guess, or C++ or whatever you want. And this is true. But does it mean that clean architecture has no place in Go? So, I think, you know, believing that if you keep things simple, it will save you is kind of lazy thinking, and it can work well if the problems you solve are simple. But if you start modeling something complex, it can it can be worse than having some more complex architecture in place. Robert [01:29:47]: Yeah, and I think we’re lucky enough to work in different kind of projects and I think it was nice to try different approaches. Try to do as simple as possible as over complicated as we can to try some techniques and, yeah, I think it’s nice to showing that this balance is not simple, but it’s not like you can always do simple way. It would be great always to go with the simple way. But, well, unfortunately, most of the applications are okay. Maybe not most, but many applications are not that easy. To make it even worse, I think probably we may end up in the situation when those simplest applications can be handled by AI because it’s not complex. I mean, it it can already handle some simple scenarios, but the deeper problems, well, it’s a bit hard to reason let’s say reason AI, but you know what I mean. Miłosz [01:30:47]: And it’s hard to define what simple code is too. And, of course, everyone wants to create simple code. I I don’t think anyone will tell you that they don’t want to do it. But, for example, in Go, in the beginning especially, it was, I think, a seamless Flat structure of a project was considered simple. So probably you don’t want to create any packages at all or just a few and then use these files in them. Works well for a standard library. That’s a good idea if you write a standard library. But good luck managing this project with 10 people. When everyone creates files they want, I don’t know. How do you create some consistency then? That’s the hard part. Robert [01:31:38]: How do you manage conflicts? So when you have more people working on one service and you have everything in one package, it’s a much bigger chance that you will conflict on something. Miłosz [01:31:48]: Yeah. So it’s very, very hard to not be naive about simplicity. But if your team is two people, we are just alone, maybe three people, manageable. Or if you can just fit the entire code in your head. Robert [01:32:05]: Or you have 10 times more microservices than people in your team. You laugh but probably you remember some projects where it was the case. Miłosz [01:32:16]: I see the solution. You just need to have one file microservices and then it’s not an issue anymore. Robert [01:32:23]: Very recommend checking but, yeah, long story short, unfortunately, it’s it can give you some impression that it’s simplifying things. But again, implementing one change in 10 services, it’s no way that you will be able to work efficiently. Miłosz [01:32:42]: So basically, complex projects needs need proper modeling and can be approached with this primitive structure. But of course if you use some very simple domain then it makes sense to keep it super simple. Robert [01:32:59]: Yeah, and I also noticed that many people think about clean architecture in terms of product structure like you have the structure, multiple directories, but I think it’s not the most important part about clean architecture. So it’s more about the mental model. So I would say that interior you can probably have clean architecture with one package because it’s more it’s harder to manage obviously because you probably will have more conflicts. It’s harder to ensure that you are not interacting between the layers. But, well, in theory, probably, you can do that. But I don’t see a really point in doing that. But, again, it’s not about the structure of the product project. It’s more about how you’re splitting, functionalities in your application. And I think it’s also pretty consistent with the Go philosophy where you should have one package with one responsibility. And in some way, it’s compatible because you can have HTTP package that is responsible for handling HTTP. You have domain layer that is responsible for the domain layer. You have adapters that are interacting with the outside world. So I would say that it’s very compatible with the approach of the language. Miłosz [01:34:15]: eah. Yeah. If you put it like that, it’s clearly this. Maybe it’s the issue of being dogmatic but the other way around. So if you are too dogmatic about being simple, it can also hurt you. Robert [01:34:33]: It’s quite ironic. I mean, both sides of the barricade are doing the same issue. So they’re dogmatic but just in the opposite direction. But they’re making exactly the same issue. Miłosz [01:34:50]: So it’s probably good to remember the the basics and to understand why you do it in the first place. So don’t don’t do it because you want your code to be clean, but to, you know, better organize work in in a complex project that will grow over time will be there in a few months or years, and you have many people working on it. That’s probably good good idea to use in architecture or some flavor. Robert [01:35:21]: Yeah. But if your team are two people, it’s easy to go in the wrong direction of the spectrum and over original things and make it just too complicated. So not see any benefits of clean architecture and just pay the price. That is usually not that big if you will not over engineer that because, again, it’s not adding that much overhead if you know what you are doing, but still worth worth keeping in mind. Miłosz [01:35:52]: Similar to the, question of should I write high quality or low quality code that we discussed in the first episode. Right? So if you if you have two person team, not, you know, developers who work together for a long time, and they trust each other, know how they work, and so on, you probably don’t need that many roles in place. But if you have a fresh team that just started working together of 10 people, you probably want more more guidelines. So there is some consistency in the project, so they can just work better together. So pick also depending on the team and how how much mature it is. What about some alternatives to clean architecture? Robert [01:36:45]: Well, so we already touched a bit on microservices. So as we talked earlier, clean architecture is useful with the handling complexity, making testing things easier. So it sounds like what microservices are promising you. But please keep in mind one thing that we touched multiple times because this is very often the issue that we have seen that people just went too far with microservices and you have more microservices than, people in your team much more. Miłosz [01:37:19]: And it’s usually Pretty common. Robert [01:37:21]: Yeah. Yeah. And usually, it’s ending up in the situation when implementing one functionality takes changes in 10 microservices. So I’m not saying that microservices are bad, but they are bad if you are splitting them too granularity. And I think it’s one of the sign that it’s done with how many services you need to touch to implement our functionality or, for example, how many services you need to test one entire functionality. So, for example, if testing one use case that is handled requires 10 microservices, it probably may also mean that, there are two granular. Miłosz [01:37:59]: Mhmm. Yeah. And there’s also the vertical slices architecture, but we didn’t use it as much yet. We probably have nothing smart to say about it. Yeah. So maybe next time, the next episodes. Robert [01:38:15]: Or maybe somebody has some experience with that, so you can write in the chat. Miłosz [01:38:26]: Okay, since we work with Go most of the time maybe we can just give some hints about Go specifically using clean arch. I think my favorite part is the implicit interfaces in Go. If you don’t use Go, in contrast to languages like Java, for example, you don’t need to specify that your your structure implements an interface. You just need to define the methods with the same signature as the interface, has. And while some approaches like ports and adapters, such as these separate ports layers for interfaces because you have to keep them somewhere to implement them, in Go you can implement them just close to where they are used, which is, I think, super convenient and also in line with the idea. So you can just define an interface next to the constructor with the dependency it needs. And basically, it’s the only place that knows about the interface in your entire project. Robert [01:39:42]: And I think it’s also nice that you can make it work with interface segregation principles. So you can just use some subset of the interface that you’re implementing in the consumer, let’s say. So you don’t need to always use entire interface everywhere. You can just use some subsets of the metal. And it’s useful when you are doing some refactoring and, for example, changing some external adapter and you it’s nicely visible where it’s needed. Miłosz [01:40:12]: I think someone mentioned it in the in the chat before. But this anti-pattern of creating, interface with exactly the same methods as the implementation. So you often see something like, you know, interface and then an implementation called the name of interface implementation, something like that. It makes no sense. And on the other hand, if you have the smaller interfaces, many things become easier like testing because your mock just becomes one method instead of 10. That’s a super nice feature of Go that plays well with clean arch. Robert [01:40:51]: The second problem that is pretty high a big problem is the fact that many sources are a missing point of clean architecture. It probably doesn’t help that it’s not defined that strictly, let’s say. So it’s adding a big, big, interpretation area. Let’s see. So, yeah, look for good sources. So if you’re writing in Go or you would like to write in Go, we recommend checking our blog obviously because we covered it already a couple years ago and it’s still totally relevant. I think even if you’re writing in a different language, it’s the many ideas are common so it might be worth checking, what we’re writing there. But remember that different languages have a bit different, let’s say, idioms and some things are done a bit differently. So we know it from Go because many people try to port, clean architecture or other techniques directly to go from Java, for example, and it was terrible idea because of what you mentioned a bit earlier. Not, for example, in Go, we have duck typing. And because of that, it’s implemented a bit differently. And it’s worth keeping it in mind. Miłosz [01:42:11]: Yeah. And probably the biggest anti pattern you can see in the examples are the single single model used for everything. So So you can have a structure that’s used for, you know, the domain logic, but also it it has database and JSON tags on them, which probably means, you know, it’s also used for storage and for API responses. Robert [01:42:35]: And it probably means that we’re getting into this problem that implementing an architecture in such a way will lead to over engineering. Because if you are able to use the same model everywhere, it means that probably you don’t need such architecture and you could just use it everywhere and you don’t need to have all this strange mapping, etcetera. You can just simplify it a lot. But if so if you don’t see benefit so you’ll not see benefit from using this approach. But if you have more complex models and your domain model is not exactly what you’re returning from the API that works in multiple domains like, for example, financial, domain. The benefit of having different models in different places is is there. Miłosz [01:43:29]: I think it’s time to summarize and Yeah. Go to the Q&A. So remember about the basics. Right? You want to separate domain logic, whatever it means for your app, from implementation details. That’s the that’s the primary idea of of clean arch or and similar patterns. And try to focus on the on this idea, on the why behind it, not on the how or or directory structure for for starters. Robert [01:44:10]: Yeah. And there are a couple things that the architecture should give you and if those things are not met, probably it may mean that it’s not maybe you don’t need to use Clear Architecture like managing complexity with the bigger project, easier to test and write faster tests, you can parallelize your development, you can insulate your changes, and improve navigation. If you don’t have those problems, maybe you don’t need clean architecture, maybe your application is not that complex. Maybe, you know, just having everything in your HTTP controller it’s fine. It’s totally fine. We’re doing it in some cases when because even within one application you can have Twin architecture in some places and doesn’t have it in other places and it’s totally, totally fine. And No use approach. Will save you a lot of time. Miłosz [01:45:07]: Yeah. On the other hand, there are some pitfalls. So it might be confusing at first for people who don’t know it yet. So take time to explain it to your colleagues so that you are on the same page. And watch out for common issues like using too many interfaces or too many layers or just over engineering the the basic idea, and watch out for using it everywhere, as Robert just said. So if you if you use modular monolith, I hope you do, not all not all modules need to follow the same architectural patterns. You can use something different in different modules. It’s fine. Robert [01:45:52]: And, yeah, to analyze your benefits and pitfalls of your application, it’s worth putting to this equation the team size, your experience, and, well, watching out to not be too dogmatic when you’re doing that. Miłosz [01:46:09]: A good tip for whatever you do. So we’ll link some materials in the description. Check out our blog post on clean architecture, on testing strategies as well. And there’s one about integrating clean architecture with DDD and CQRS. In Go. And also the one about the repository pattern is probably quite relevant to clean architecture in general. And all those posts are just a series that describe the white workers repository, the example project. So you can check it out too. Robert [01:46:51]: Let’s take a look on comments. So we already covered some of them but let’s check what we have missed. Where are comments? Oh here, okay, so let’s start from the beginning. Yeah, so again a couple comments that people are missing working with me, Osh. Good sign. So usually we’re using clean architecture so, you Miłosz [01:47:28]: Know. Yeah. I think it’s the only pattern that I’ve heard of people using beyond, the teams we worked on. So, you know, we met one year later and they told us, oh, you know, I spread it in my team and now we also follow Clean Architecture’. Robert [01:47:47]: Or I missed the project that we worked together. The current one is so. Miłosz [01:47:53]: But I mean, the the architecture is a pattern I’ve heard mentioned multiple times, but it’s probably also the the only one mentioned so often. So that must be a sign. Right? Looking at the bottom, there’s a question about how clean architecture differs from hexagonal. And I would boil it down to naming with a different approach. I would say we use a combination of both. So it’s not important. I would say it’s implementation details. Robert [01:48:36]: But, yeah, I think it’s all about you have some inner layers and outside layers and inner layers doesn’t know how outside layers are implemented. Miłosz [01:48:47]: Yeah. And all those variants call them a bit differently but the core core idea is the same. Robert [01:48:56]: Yeah. And, you know, my theory about this is that, you know, the multiple techniques are reinvented every 15 years because, you know, there is a hype cycle of every technique that at the beginning it’s, hype time. Oh, yeah. It’s solving all of our problems. So everybody is going into that without understanding how to do it. Multiple teams are implementing that. Some people in very dogmatic way. So people are starting to hate this technique and the word is spreading that, this is a bad technique. Don’t use it ever. It’s super bad. It’s overcomplicating stuff, etcetera. And less and less people are using that. And after some time, somebody is thinking, oh, okay. It’s so nice idea but let’s it’s the name is already so spoiled that we need and we need to name it a bit differently and so people will be ederabad about that. And, you know, it sounds absurd, but it works. Miłosz [01:49:58]: So are are you suggesting we need a new name? Yes. And your standards too? Robert [01:50:04]: Yeah. So if you have any idea how to name next, iteration of clean architecture, listed on the chat. Miłosz [01:50:16]: There’s a question about tests. We kind of mentioned this before. So do you ever add tests on the application layer and how do you go about it? Mocks or fakes in memory by implementing the interfaces? So it depends how many logic do we have there. If we have a separate domain layer, we usually have almost no test in the application layer because it becomes mostly orchestration of things. Yeah. Robert [01:50:45]: And it should be covered by the component tests, basically. So you are calling the component tests and it’s going over all layers and checking that everything is integrated with that problem. Miłosz [01:50:55]: But sometimes it makes sense if there’s some some logic you can test. Robert [01:50:59]: Especially when you don’t have domain layer, for example, because sometimes you may not have enough logic to have a separate domain layer. So you may have more logic in application layer. But in this case, what will for example, what you can do can you can have functions that you can just, test independently without entire words. Miłosz [01:51:19]: Yeah. And in this case, in memory fakes work well. And this this is where having small interfaces is super important. Robert [01:51:27]: Yeah, and what’s important there is ensuring that you are mocking this IO stuff that I mentioned a bit earlier. So don’t mock your, I don’t know, logic, let’s say, because it your test will be just mocking some subset of logic and they will not give you a lot. So, you know, mock your database, mock your mock your external dependencies, don’t mock your logic. Miłosz [01:51:52]: And we are big fans of writing the fakes, manually. I know many people use mocking libraries but we find them usually not that helpful. Robert [01:52:04]: At least in Go. I think it may be, let’s say, problem of Go because of its compiled language and often those libraries are using a lot of reflection. For example, I remember that in PHP there were some nice libraries for testing but in PHP it’s a bit easier due to not being compiled language but, well, upsides and downsides. Miłosz [01:52:29]: And sometimes it also creates, like, brittle tests. If you you start testing the function calls, for example, then you often need to change them if you change anything in the code. Robert [01:52:46]: And, yeah, the problem with mocking libraries so it’s another thing in the bucket of things opening the gate of hell so it’s making easier to mock too much. In other words, it should be painful to mock things because you can mock too much. Miłosz [01:53:03]: Yeah. That’s what Todd in the comments mentioned before that if you need a mocking library, you are mocking too much. Robert [01:53:09]: Exactly. Miłosz [01:53:10]: Yeah. That’s kind of the idea. I think that Yeah. And Firmino had some experiences with architect and his Gomock lib. I don’t remember which one gomock is. Is it the one using reflection or the one using generation? Robert [01:53:29]: I guess so. I already forgot it intentionally. Miłosz [01:53:36]: Yeah. I just found another one from Todd. The first step I use when trying to introduce layers to people who don’t use them is to explain that the application shouldn’t be tightly coupled to the API. So, yeah, it’s a great approach, and also shouldn’t be coupled to the adapters, the database, for example. I guess the hard part of explaining it is explaining someone why it is good to decouple them. Robert [01:54:05]: And if you don’t see any reason again maybe you don’t need the architecture because if you have a CRUD application and you are mapping your CRUD structure before your at least at least four layers, in worst case six layers or how many, it’s just a waste of time. Miłosz [01:54:28]: Yeah. And sometimes it’s also not that trivial to explain the benefits because you can use the typical reasons like if you ever want to change the database it will be easy. But it’s easy to counter this with. We won’t. Which is also true. Robert [01:54:54]: Don’t use this argument. Miłosz [01:54:57]: Yeah, I mean, it’s it’s hard to have a discussion that that it doesn’t start to be about this dogmatic approach. Our code needs to be clean. Our code needs to be simple. So watch out for that. Yeah, but this is definitely a nice idea. Robert [01:55:17]: It’s also probably worth mentioning that, you know, you won’t be able to always work in the teams following this approach, which is fine, but also don’t be a rebel that is just using clear architecture in your small project. I mean, if you’re working the team that, okay, you’re not following this, don’t be the rebel. I mean, sometimes disagree and commit, it’s sad, but well, maybe it’s not sad. It’s something that you need to use in you should work in a team, not alone. Miłosz [01:55:48]: It, I guess, depends on how well the project is doing. You know, if I know it’s hard to, you know, disagree and commit if you see all the issues that following some pattern could solve. That’s that’s the hard part, you know. Robert [01:56:05]: But you will be not always able to just convince everybody. It’s different. Miłosz [01:56:10]: Yeah. Just you need to know why you you want to use some pattern. Right? We discussed this in the first episode that we had this one project where everyone was super happy to work with because it it used, many cool patterns, and we kept talking about it years after it. And the project was shut down because it had no paying users. Robert [01:56:40]: Well, at least it was great. Miłosz [01:56:42]: So that’s the hard part. You need to be productive and avoid getting into this big ball of mud issue. But sometimes it doesn’t matter that much as well. Robert [01:57:00]: Yeah. So at the beginning, we have a couple of comments that somebody is missing working with you. We have also one comment that please come to my team and make team follow architecture and solid. I use elixir, and I don’t know if it’s the language or the community, but, oh my, everything is coupled. Miłosz [01:57:21]: I don’t think it’s the language. I think it’s just it is what it is. Robert [01:57:27]: I think it’s also something connected. I mean, the mindset and language. I will not mention language names started with r or Miłosz [01:57:38]: But Yeah. But, you know, even if in Go, which is supposed to be a simple language and, I don’t know, somehow make it magically better or your applications easier to develop, We know the reality is it’s just as easy to create something coupled in Go than in other languages. Maybe with less magic. That’s the good part. Robert [01:58:01]: That’s true. Miłosz [01:58:02]: But, for example, you know, the struct tags in Go make it super easy to couple layers and it’s not that easy to see if you don’t know what you’re doing. Robert [01:58:12]: But I think it’s something in that, you know, a lot of Go people are very open minded because it’s, you know, fresher language and in general, you you know, if somebody was eager to learn some new language and try something different, it’s at least from my experience, it many cases, it was real open minded person and it was somewhere there. But, yeah, about about the team, you know, if you are not able to change the team, change the team. It’s sometimes like it and it’s just sometimes not possible to change what you’re doing in the team. So, you know, it might be worth, you know, leveling you up in your skill set and trying to find a better place. Unfortunately, there are a lot of great places and hard bad places to work in. It’s also a bit unfortunate that we know that it’s hard to join good teams that are, you know, following some good practices. But from other side, you know, also grass in your neighbor’s garden is always, more greener than yours. But yeah. It you can always find a better team. But for that, you often need to Miłosz [01:59:20]: There’s also there’s a risk there that you start thinking that the team using clean architecture is better than yours. Yeah. Who knows? Maybe they over engineer it so much you would hate it even more. Robert [01:59:34]: Yes. Yes. But yeah, about the quality again, I recommend checking the episode about how to write bad code qualities. We’re also touching a bit on how to set expectations because well, none of the team is perfect. There are always problems and but, yeah, there are better and worse teams. Miłosz [01:59:57]: Okay. Are we out of questions? Robert [02:00:01]: I think so. Miłosz [02:00:04]: I think we are just on the two hour mark. Robert [02:00:08]: Perfect. Miłosz [02:00:14]: Cool. So thank you everyone for joining us. If you want to not miss the next episode, you can subscribe to our newsletter on our blog, and we will let you know. You can, of course, subscribe to YouTube, our YouTube channel, but we’re not sure if they will notify you, so our newsletter is the best one, the best place to get notified. And you will also not miss posts, our notifications of our our our posts, so that’s another reason. Yeah. Leave us any comments. Tell us if you liked it and what you can improve. If you are listening to this after the live happened, you can just comment under the video. And if you listen to this on any podcast application, give us five star rating. Robert [02:01:17]: Yeah. Unfortunately, we’re ruled by the algorithm so your, thumb up or flight style review can help us a lot and I think this is the best way how you can support us and well, help us to record more episodes. Miłosz [02:01:37]: Which, will happen in two weeks. Robert [02:01:39]: Yeah. About what we are talking in two weeks? I don’t remember. Miłosz [02:01:42]: I think how to learn quickly. Robert [02:01:45]: So I think it will be another episode that is nicely connected to what we were talking earlier. And, yeah, if, Nicholas, you were mentioning that you would like to join your team, We are not looking for a job, but what can help is, well, learn more and being able to join better teams. So this is the tactic that we definitely recommend. Miłosz [02:02:09]: Or spread some ideas within your your team. Robert [02:02:13]: Yep. That’s true. So, yeah, the next episode is about that. Hope you see. Miłosz [02:02:19]: Thank you everyone for joining us today. Thank you, Robert. Robert [02:02:22]: Thank you, Miłosz. And see you in two weeks. Bye bye. Miłosz [02:02:25]: Bye bye.

Quick takeaways Frameworks promise productivity but often lead to issues as projects get larger and more complex. The Go community prefers small, focused libraries over frameworks due to Go’s design philosophy influenced by Unix principles. Watch out for risks using frameworks like vendor lock-in, deprecation, and costly migrations that can take months. Explicit code is more maintainable than magic framework abstractions. Choose your approach based on project size and maturity - frameworks might work for prototypes, while modular libraries are better for long-term projects. Introduction In this episode of the No Silver Bullet podcast, we discuss frameworks in Go and when they’re useful or problematic. We talk about why the Go community generally avoids frameworks compared to other languages, and how small, modular libraries are often preferred in Go development. We share our experiences with frameworks across different projects, including tradeoffs between productivity and long-term maintenance. Notes Model-View-Controller (MVC): Pattern first described in the 1970s for Smalltalk, still widely used today. Unix Philosophy: design concept created by Ken Thompson (also a Go creator) promoting small programs that do one thing well and work together. When to avoid DRY in Go Watermill: Our event-driven library for Go designed to not be a framework. Repository Pattern: Our blog post that is still relevant and frequently referenced. tdl and pq - the CLI tools we mentioned. Clean Architecture: The topic of our next podcast episode, a design approach that helps maintain separation of concerns. Wild Workouts : Our example Go codebase demonstrating clean architecture. The Best Go framework: no framework? Quotes The happy path is easy enough, but the happy path is usually not the hard part of software. We often overvalue how much effort the boilerplate requires. Miłosz Framework knowledge tends to become out of date. You can spend days or weeks learning something about a framework, but it can be outdated. And if you switch to another programming language or company, a lot of effort that you spent to learn stuff will be just wasted. Robert It’s more important to learn even-driven architecture because you learn the theory behind it and how it works in general - it transfers better to whatever you will do later. Focus on timeless skills like how to split modules in your application, how to make it decoupled, how to write business logic so it’s easy to read and modify. Miłosz The Go language is heavily influenced by Unix philosophy - write programs that do one thing and do it well, write programs that work together. It’s visible in Go’s standard library. This is why Go promotes building independent components that you can connect together. Robert You need to be careful not to go too far with foundations. It’s better to start with some modular libraries, have some reasonable setup in place, but don’t go too crazy with it. Most of the time you’ll need to refactor the project anyway, whatever you do, because it can change drastically. Miłosz One big decision at the beginning may cost you six months of work later. Understanding if something is tightly coupled to your application is simple - just think about how easy it would be to remove it. Robert Timestamps 00:00:00 - Introduction 00:01:14 - What is a framework? 00:06:01 - The hidden costs of frameworks 00:10:43 - Go's approach to frameworks 00:12:45 - Unix Philosophy and Go 00:15:53 - Framework risks 00:18:50 - Early experiences with frameworks 00:24:59 - Framework limits 00:34:20 - Migrating out of a framework 00:37:58 - Should you use a framework? 00:42:38 - Simple vs. complex applications 00:51:23 - Alternatives 01:00:32 - Framework developers vs. universal knowledge 01:06:13 - Summary 01:10:29 - Q&A session Transcript Robert [00:00]: What if framework you choose today costs your team months extra work next year? Choosing a framework at the beginning of the project can greatly impact the project in the future. We have been in projects where relatively small decisions in the beginning led to many months of work a couple works later. Have you ever considered why the Go community does not favor using frameworks in go? Does it mean that using frameworks is always bad in Go? I’m Robert. Miłosz [00:31]: And I’m Miłosz And this is No Silver Bullet podcast where we discuss mindful backend engineering. We spent almost 20 years working together in different projects and teams, and during that time we’ve seen that following advice like “always do X” or “never do Y” rarely works and can limit your growth. And in this show, we share multiple perspectives that will help you make smart choices and level up into the principal engineer level. Robert [01:01]: If you have any follow up questions, leave them on the chat. So we will try to answer them at the end or maybe even during discussion if we will have enough time. Miłosz [01:14]: So we talk about frameworks today in Go, but also in general, maybe let’s start with what we consider a framework. So we are all on the same level. First of all we focus on backend frameworks. So we don’t talk about the frontend world today. Just the backend side of things. And in this episode we specifically focus on the Model View Controller framework or MVC, which is usually what most people consider a framework during the discussion of “should I use a framework at all?”, or “which framework do I choose?” So maybe, Robert, could you give us a TL;DR what MVC is? Robert [02:02]: So MVC. So it’s an interesting pattern that it’s already pretty old, but well, I’m kind of fan of some old patterns because I think often they’re still pretty relevant. And the same thing is with MVC. I think for the first time MVC was let’s say invented or written down in 1970s. So yeah, it’s more than 50 years ago. So pretty long time, but it’s still pretty valid. Miłosz [02:29]: Like SQL. Robert [02:31]: Yep, yep. And I think it was for the first time in Smalltalk language. That was also pretty interesting one. But we’ll not go today about talk about Smalltalk. But yeah, and MVC. So it’s acronym for Model View Controller. And the interesting thing is that the chance that you are using MVC is pretty high. If you are building an application that has a database and a user interface, and you are allowing to modify this data. So it’s pretty high chance that you are using MVC. And it’s because it’s the industry standard on how you are building applications now. And very often it’s actually MVC pattern. And the idea here in MVC is to split your model, view and controller. So model, this is your domain logic. Your entities that are handling data. View is we can say the user interface often and controller is responsible for orchestrating everything between this. I’m not sure if you agree Miłosz, but probably it may be obvious for some single page applications, but I guess we can also say that even if you are building some react application, it’s some form of MVC also, even if it’s a bit different than how it was built. Miłosz [03:56]: On a different level, but a similar idea. With the one model that is the source of truth of data, and then some parts of the application that react to it in some way. So that’s similar concept I guess. Robert [04:13]: And yeah, today we’ll discuss MVC frameworks. So if you think about frameworks like Ruby on Rails, Django, Laravel, Symfony, Spring, all of them are MVC and are MVC frameworks because they have building blocks for building MVC parts. So you have some building blocks for models. Usually you have some building block for controller and you have some building block for view. So the idea is that they should simplify your life when you are building such kind of application. Miłosz [04:45]: And there are some other libraries that we probably shouldn’t consider MVC frameworks for this discussion. Like router libraries and in Go there is many of them, but you can also think of something like maybe flask in Python or I guess chi or echo in Go. So they give you the tools for working with HTTP messages and you get the views, but there is no model. Or you could do it yourself with some ORM framework maybe, but it’s yeah, it’s not out of the box compared to, for example, Django that gives you all of it. Robert [05:31]: Django or anything what I mentioned earlier, like Ruby on Rails or Laravel, Symfony, etc. etc. Miłosz [05:38]: From what you said so far, frameworks seem like a good idea, like the separation between model, view and controller seems like a separation of concerns. What we want most of the time and they do many things for us, they come with batteries included. So usually if something sounds so great, there must be some hidden cost, right? Robert [06:09]: Yep. And frameworks are no different. So I went over websites of couple of frameworks to figure out actually what they are promising. And I have a list actually about things that are usually mentioned there like authentication and authorization, logging, observability, queues, database access or some sort of ORMs, dependency injection, sending emails, payment provider integration. And the idea is that it’s all integrated. So for example, if you are using any component, it should be integrated with observability with queues. And the idea is that it should work together out of the box. I also checked some promo materials for those frameworks and some sound pretty nice. Like a “complete ecosystem for web artisans” or “build and ship software with tools crafted for productivity”. Sounds good. This one I like pretty much “compress the complexity of modern web applications”, or the last one “optimized for happiness”. It sounds a bit like some religious cult, you can say, but to be fair, well, spoiler alert it’s not false. I mean, in some contexts it’s meeting those requirements moments. Maybe happiness sometimes. Miłosz [07:34]: It sounds good on paper. I think especially if you just start, if you bootstrap a project, especially before we had AI tools to bootstrap projects with. It could be quite cool to just call a couple of functions and have a website ready. Robert [07:57]: And it’s kind of “wow” effect. So you just spend five minutes on something and it’s working. You can show it to your mom, “Mom, look what I built, oh”. Miłosz [08:08]: Especially if you are a beginner, right? Robert [08:12]: Yeah. So you don’t need to have all this knowledge about how to connect stuff. And you can focus on things that matter at the end and are the most important. So building the useful things. And you are productive when you are doing that. So that’s cool. Miłosz [08:29]: And what I also like about it is that you can focus on writing the core domain parts of your software because for many web apps, there are probably, I don’t know, 80% of the code that is written is very generic, right? It’s what all web applications use, like database handling or HTTP routing and stuff like that. So it makes a lot of sense to focus on what’s different in your software and not waste time on writing what someone solved already some time ago. But I think there’s also this trap here that I’ve seen many companies and projects have this dream of, you know, we will build all the foundations first. So we have this super custom framework or platform or whatever this means, and then we can just create new applications or microservices or whatever else, like a cookie cutter. So it virtually should take no effort in theory. And it sounds nice, but I think it’s also a trap because it ignores the fact that most of the complexity in software is in the details, in the weird edge cases, which is, by the way, also probably the reason why AI tools are not that great yet. I mean, they are very, very nice for the generic cases, maybe for similar stuff for what frameworks are nice. But when it comes to those weird edge cases you need to handle in the real world, yeah, that’s a bit harder. Robert [10:18]: It’s actually quite interesting that we can say in some way that AI is competition for frameworks, because, yeah, that’s true. Miłosz [10:26]: They all fall into this category of this dream of “code with no code” or “software, with no code” to create applications with no code like UML was supposed to be at some point maybe. Robert [10:42]: But let’s go back for a second to Go, because the title of this live podcast is “When you shouldn’t use frameworks in Go”. So let’s go for a second for Go, because now we covered a bit how it looks like in the different technologies. But it’s worth mentioning that in Go it’s quite different situation. So like in different languages that we mentioned you have some kind of well-established frameworks, Go is different. So there is no leading framework. And generally the community is against using frameworks. And this is interesting situation and I think one of the reasons for this situation is that well, Go is pretty new technology compared to other technologies. And well, building one big framework, it requires much more effort. If you compare it to just building small libraries that do one thing and yeah, it’s just faster. So it may be probably one reason for that. Miłosz [11:46]: And maybe also the, the libraries that we have now evolved over years. I remember like when we started eight years ago, there weren’t that many libraries for anything but they started appearing over time and they grew and became more polished. But there was nothing that connected them all. Well, there were some attempts for sure, but I think there is no one standard framework right now that everyone uses. Maybe it’s because also of how the standard library looks like and the Go philosophy of small packages that do one thing well. That kind of resonates if you think about the current ecosystem of open source libraries in Go. There are many smaller libraries that focus on doing just one thing. Robert [12:44]: I think a good reason for that may be also the design of the language, so how the language was designed. So I think, I’m not sure if anybody mentioned it explicitly, but I think it’s visible that it’s heavily influenced by Unix philosophy. So I suspect that for at least one reason that one of the creators of Go language is Ken Thompson, who is also author of Unix Philosophy and Unix Philosophy tells us to write programs that do one thing and do it well. To write programs that work together. And it’s visible in Go standard library. If you look on how, for example, HTTP package is built, how io.Reader or io.Writer are built. So, you know, io.Reader or io.Writer is an excellent example of a small program, let’s say, or interface doing one thing and you don’t really care about how it’s working under the hood. It’s pretty similar. So if I would like to explain Unix philosophy for you, probably the best way will be mentioning how your terminal works. So you have things like cat, you have things like grep and you can connect it together and pipe that. And this is similar. This is actually this idea. And this is moved to Go heavily. And yeah, I think it may be also the reason why Go is kind of promoting the building independent components that you can connect together. And it’s nice because for example, if you have http package, you can have multiple libraries that can understand HTTP request or HTTP response writer and interact with that. So it’s promoting this, creating independent libraries. And for example you can have a library that is just giving you one HTTP middleware, and you can integrate it with the rest of the system. Miłosz [14:38]: It’s a great API design rule overall, no matter the programming language really. And it’s nice that Go gives you those interfaces out of the box. So most libraries try to stick to them, which makes it easy to connect them together. Robert [14:53]: And it’s not something that I’ve seen in different languages. So usually in different languages you have library that is doing HTTP, and standard library is not giving you that much. Or sometimes those frameworks or libraries are doing something totally incompatible. So it’s just hard to have multiple libraries that are working together. And in Go giving these interfaces, it makes this much easier. Miłosz [15:15]: That’s actually a good point. I think Go is the first language I’ve used that gives you this really powerful HTTP server. That’s yeah, it wasn’t ideal from the start, now it’s a bit better. We usually use routing libraries anyway, but it’s quite solid if you want to start with anything. I think most other languages, at least at the time Go was released, weren’t that great. You had to just, you know, include some custom library. So why not use a framework if you have to do it anyway? And there were some attempts at creating Go frameworks, I think. And one risk of using them is that if there is no one big default framework like let’s say Spring in Java, you have this risk of this framework becoming deprecated, and then you are stuck with some project that uses a framework that’s hard to migrate out of. And I think there was go-kit. We’ve seen used some projects. Robert [16:28]: No, it was go-micro. Miłosz [16:29]: Oh, true. Yeah. And I think there was also a very popular Gorilla router library that also became deprecated over time. So you can still use this of course, but it doesn’t receive updates. So there is some risk that you might need to migrate out of it in the future. And you know, migrations are fun. Robert [16:55]: And you know, if you are using frameworks, you have this framework that is integrating all MVC layers. So model, view and controller, it’s much harder to migrate out because if you just have a library that is handling routing for you, it’s kind of simple to get rid of this library. But if it’s integrating everything more, I think model is probably the worst part, because if your model is integrated with views and controllers, it’s very hard to get it out of your application. And yeah, and okay, somebody may say, okay, it’s not happening, that libraries or frameworks are deprecated. You can check history of them, that some of them are quite dead, we can say. And there is also one example of licensing change. So go-micro from what I checked actually changed license and is no longer open-source license. So it’s business source license. So the same license as Terraform is using and it’s not open source. So you can read the code, but you can cannot use it like open source thing. Miłosz [18:02]: Time to fork. Robert [18:04]: Yeah. But remember fork the tag before changing license. So the thing what happened with with Terraform. But it’s risky obviously. And in some cases it may just cost you a costly migration. And this is just one example. Miłosz [18:20]: So someone could say it can happen with any library. Right. So why bother? But I think the difference is the smaller libraries are easier to to replace if it impacts just one area of your code. Using a different library here is quite easy, probably. Depends on how you do it, but hopefully. But if you do it as a core of your project, like a framework, then it becomes much, much harder to do it. Robert [18:47]: Definitely. And if you go back for a while to discuss the promises of frameworks because, you know, we’ve been there also a couple of years ago when we were starting programming and, and yeah, at this time we’re working, to give you some context. So it was, I think almost 20 years ago. And at this point, the landscape of programming looked totally different. So, for example, if you know Laravel framework from PHP, it didn’t exist yet. jQuery, that already sounds like super old technology, it was almost not adopted yet. And if you are talking about other PHP frameworks. So it was Symfony, but it was also not widely adopted. It was something pretty new. And when I remember that we use Symfony for the first time, it was actually quite great at the beginning because it made a lot of things that we were doing earlier by hand, and it just allowed us to save a lot of time. So I already mentioned authorization, logging, nice observability. It just worked of the box, and it was great. It also generated for us database models. So again, we were able to deliver stuff faster. But again we were also a bit less experienced. So for us it was like magic at that point. And I remember that before that we were just crafting some SQL queries by hand and it was super dirty. Miłosz [20:22]: But this magic was also a bit unexpected for me when I first started using it. So I remember you tried to onboard me to Symfony. And you know, I was looking through the project and I couldn’t find a SQL migration files and I asked you, where are the SQL files? You told me what? You don’t use it. You just you just use models in PHP. And that felt weird for me at the time because you had to change the mindset. You have to just trust that framework, that it will do its job well. You don’t really. I mean, you can dump the schema, of course, and inspect it, but usually just accept that there’s some annotation you have to do and that’s it. So if you would like to know how it works, the internals, that’s a bit of a mindset change you have to do. And also, on the other hand, if you have someone who is used to working with Spring, let’s say, we had developers who were Java developers and joined us to write in Go, and I remember they were confused about how we don’t have a dependency injection library, and they were just mind blown similarity of, you know, how why do we do it manually? Why not just put a single annotation and it works? So it’s a bit different concepts. You have to adjust. Robert [22:07]: But I think it’s also one downside that, you know, it’s hiding this complexity from you. And sometimes those people were coming from Spring. They struggle because, okay, they need to write some query by hand. And it’s like writing a SQL query is so hard. I didn’t write any SQL query in my life earlier. Miłosz [22:29]: That’s maybe why, you know, “no silver bullet” is so difficult because we tend to adopt to one way of doing things when we started. Robert [22:42]: But I think it’s also showing that it’s, you know, often frameworks are showing you that, okay, you’re doing it in one way and people are not really thinking about doing some exceptions because for many cases, it’s just enough to have generated model that does queries for you and ORM is doing for you, but probably 10, 20% of queries can be written by hand and it can be much more optimal. But people are often going in the direction that, okay, we need to do it with ORM and are going with some crazy stuff and also to ensure that it’s properly shown somewhere with the view layer. Miłosz [23:18]: Yeah, we like consistency. What can I tell you? I had the same experience as I just just mentioned. It’s nice to work in a way you already know. And if you have to change it’s problematic. Robert [23:34]: But probably we can recap that as okay, frameworks obviously they have some downsides but well they are giving us a lot. So you may ask us okay why we are no longer PHP developers writing in Symfony. So it may be some reason that, well, it wasn’t the best technology in our life and now we are a Go developers instead. So it’s a question what went wrong. So I already mentioned that, you know, 20, 10 years ago the landscape was totally different. So it was much less open source tooling like for observability, logging, tracing and routing so you didn’t have much choice because you could use Symfony or some magic PHP frameworks, or Spring maybe, or maybe some Python tools. So actually frameworks were giving this a lot for free, but now it’s quite different because you have much more libraries, especially in Go that are open source. And they are using some open standards. Miłosz [24:38]: Or even standard library like with HTTP in Go, you don’t need a separate library for this and many other similar for stuff like, you know, it wasn’t possible back in PHP. You know the standard library there was quite poor at the time. Robert [24:55]: Oh yeah, especially compared to the Go one. Miłosz [24:57]: So basically we learned that there is no free lunch. There is no silver bullet, you have to consider what you need to give up to use a framework. And I think it’s more relevant if you think about the project in the long term. Because if you just start the project, as we said, you know, we just throw in a couple of function calls and you have a working website that’s pretty cool. It feels like magic initially. But over time it becomes less cool because you start getting into those edge cases that the framework doesn’t handle for some reason. And maybe what you mentioned about SQL queries is a good example of this. So often it’s it just makes more sense to use a custom query to get some data you want, some complex query, instead of trying to use the models. I especially like how you know in Go the ORMs often have a separate language in the struct tags. So you have to learn a separate syntax that is not validated by the compiler. And I started to wonder, you know, why is it really that that helpful to use it. Robert [26:34]: And you need to also do a PhD from how those magical notation works and it’s not great. Miłosz [26:42]: Yeah, exactly. So at some point it just becomes less flexible. It’s easy to start. But then you need to add some extensions, maybe, you know, even in several components that use the framework and it’s starts to get a bit tricky. Robert [27:01]: Model is one thing, the flexibility of models. And if you are talking about MVC frameworks because, as we said at the beginning. So in most cases when we are talking about frameworks, it’s MVC frameworks. So if it’s MVC, it needs to support not only model and controller, but also view. And from my experience, very often the biggest problem is in the view because okay, if you are starting to build your simple application, just having a form and a form with some fields there with simple inputs, it’s enough. But probably very soon you will find out that what frameworks are giving you out of the box, it’s not really allowing you to express, let’s say, every UX and requirement that you are receiving earlier. So yeah, it’s also obviously often you can do workarounds or for example, not use the view part from framework. Just return your models from Rest API or from other site, you are losing a lot of advantages that the frameworks is giving you. Miłosz [28:10]: You need to find this balance. Because if you find yourself spending more time on extending this, those missing edge cases and writing those workarounds, it’s probably a good idea to do something else. Those queries for example. Right. So if you start fighting with those with this weird syntax of an ORM or a library, and maybe just give up and use a standard SQL query and should be good enough. Robert [28:44]: And I think it’s especially a problem when your project is becoming more successful and team is bigger, because if you are two people, team whatever, and your product is not that successful or widely used, it probably doesn’t matter. Actually, you can take shortcuts and it will be everything fine. But well, if your product is becoming successful, all the problems are starting to pop up there. Miłosz [29:08]: As usual, the happy path is usually easy enough, but the happy path is usually not the hard part of software. And we often overvalue how much effort the boilerplate requires. Like, it often it just seems boring to to write all this, you know, routing and database database queries and all this stuff that you do in every application. It’s not really different. It doesn’t differentiate you from other applications, but it’s not really a big effort. Right? It’s just boring. And now LLMs will happily do it for you. So it’s even better. Robert [30:02]: But from one side, writing it by hand is boring. But I would not say that it’s super interesting to do workarounds for that. And I would say that it’s much, much worse than. Miłosz [30:14]: Oh, but sometimes, sometimes it’s actually fun to do it. I have a story maybe I can share. So you remember in one project we had this entities stored in a database in, and we used one model for them in Go. And yeah, we had this issue that we didn’t want to return the exact values we stored in the database to the front end side. So we had one model and we wanted something different in the views than just the database model. And we could just create a different model and map those fields between them. But that would be the boring part, right? You don’t want to write this mapping. Robert [31:11]: We are smarter. Miłosz [31:13]: Yeah. So it’s the usual question we get asked, right? Like, okay, I did clean architecture and I separated my models between database and HTTP, let’s say. But what now? How do I automatically map those structures. Yeah. You don’t. Like in this project. We exactly thought we are too smart to do it by hand. So we wrote a library that could do it for us using reflect and custom struct tags. And please don’t do it. Robert [31:13]: It was great. Miłosz [31:52]: And since then, my rule is if you use a statically typed language and use reflection think twice. Is this really what you want to be doing? Robert [32:05]: Yeah, probably second outcome is that don’t be smart. Usually being smart as programmer, it’s not the best way because you may become too smart. And your solution may be that smart that nobody will understand that. And it will take a lot of time to maintain it. Miłosz [32:26]: This is actually my favorite part of this story is the why behind it. We just thought we are smarter than writing boring code and that led to more trouble. So you have to be careful around this. Robert [32:48]: All right. One important thing. So I think in general, the discussion about frameworks that we have today, it’s nicely connected to the previous episode that we had two weeks ago. So if you didn’t have a chance to listen to that, I recommend it because in general it’s often well, choosing framework is big architectural decision. So often if you do it at the beginning, it’s quite hard to remove it later because as we said earlier. So it’s a matter of coupling your models to rest of the application. And it’s not like replacing your HTTP router or something like that. It’s usually very tightly coupled, and it’s just hard to remove that. And yeah, in previous episode we were discussing pretty deeply that it depends a lot in which stage of project you are or if you think this project will be successful in long term, because probably if you are doing some pet project or you have some small team and it’s proof of concept, probably doesn’t matter that much. But if you see that your project is taking traction and team will be bigger, it may be problematic to go out of this decision. Miłosz [34:20]: So this assumes someone would need to remove a framework, right. Which usually doesn’t happen if it’s a big and proven framework that many people use. But with smaller frameworks it’s a bit more risky. So some time ago we worked with this go-micro framework. You remember, for microservices. Robert [34:46]: I remember. Miłosz [34:47]: Across many teams, tens of microservices, and there was some issue with the protobuf version there. There was something custom. Robert [34:57]: Yes, I think it was some custom implementation of protobuf that supported something extra, but at some point it was no longer maintained. Miłosz [35:06]: Was it related to gogo/proto or not? Doesn’t matter. But somehow there was a decision to remove it. And that’s a long story short. Robert [35:18]: Yeah, I think so. It wasn’t like, okay, we need to remove that. I think it was leftover. Somebody decided to use it a couple years earlier. A year and couple years later we were in the project and it was some leftovers from that. And I think it was not maintained and it didn’t support something. And it also have some weird edge cases. At some point we needed to get rid of it because it blocked something, something else. Miłosz [35:44]: Possibly also back then Go modules weren’t as great as is it now. And I think there were also some issues with this, but basically there was decision to get it removed and it went exactly how you would expect a framework removal to go. Robert [36:04]: It to be exact. So it wasn’t removing entire framework, it was just leftover from something that was there since a couple of years. Miłosz [36:13]: I think it took man-months of, I don’t know, half a year, probably in total effort from people from different teams because this spanned across teams, because it was related to communication between services. So obviously it’s what connects the project together. Yeah. I remember when we finally got rid of it, there was a big maybe not event, but I remember celebration where everyone was super happy. I mean, I were too, but then you start to think about how dumb effort this was, you know, spending time on removing some obscure library that caused some issues, but, you know, shouldn’t be there in the first place probably. Robert [37:07]: I think it’s an excellent example that some decision that happened a couple years earlier that have that big impact later. And yeah, it’s not the only time when we’ve seen that. So obviously it’s hard to do all the decisions fine all the time. But yeah, the thing that I mentioned. So it’s worth thinking like, okay, I’m using this framework library or something like that, just in case if it will be deprecated, how I can remove that. And if the answer is okay, It will be super hard. Maybe it’s worth thinking. Okay, maybe we should choose something different or how we can mitigate that. Or again, maybe again. It’s a bit more in the previous episode about that, but maybe it’s fine because maybe it’s proof of concept. Miłosz [37:58]: Probably hard to decide what is a good, you know, or maybe maybe a different way. What is what are some red flags that tell you stay away of this framework? Because I guess if someone uses a proven framework like Spring or Ruby on Rails or Symfony or whatever… Robert [38:20]: Probably it would be not deprecated. Miłosz [38:22]: Probably. Yeah. Because the community is big enough. That’s one thing. If the company behind it would drop it, I guess you could expect the community to just fork it and maintain it over time. Robert [38:36]: Or maybe not, because it’s just not adopted enough and because I think it’s the thing in Go. So in Go we have pretty, it’s pretty big segregation of frameworks and there is no one leading framework. And maybe some people will do some updates, for example in the fork. But I would not expect that it will be super big updates. So it’s risky in Go, I would say. Miłosz [39:04]: Especially if the framework is created by one person or a small team. Robert [39:09]: Or for example, some other risk that people are often not thinking. So if it’s created by some VC backed company. So, you know VC capital is not free, it’s not charity. So probably they will need to make money at some point. And we’ve seen recently when it was big shift in trying to get money from multiple products and some licenses changed, like with Terraform, for example, and it’s definitely a risk. Again, especially with things like framework that it’s just coupling everything and it’s hard to remove. So I can imagine that in many cases it’s like, okay, let’s better pay for that and forget. Miłosz [39:58]: Or maybe let’s just pay for the first year and then we’ll migrate. And you know, this migration task sits at the bottom of your backlog, and one month before the next invoice, you realize, oh, we should migrate. Robert [40:13]: Task: remove framework. Miłosz [40:15]: Oh, let’s do it next year. And then the next year, the company shuts down because they run out of funding. So yeah, there’s a risk. Robert [40:29]: If you are thinking about framework, it’s also worth looking on the features list and assess if you need everything from this list. Because frameworks are usually providing many things out of the box, but very often you don’t need to have everything out of that. For example, do you really need, I don’t know, payment provider integration? Do you need ORM or many things from that? And it’s a question if it’s worth paying the cost of coupling everything. Or maybe it’s just better to use something independent. Miłosz [41:08]: And often frameworks have to be that, have to cover all those edge use cases because otherwise they are not useful because people use a framework and they have all kinds of requirements. I think it’s even for authentication is a good example. I think there is this authboss in Go for authentication kind of framework. I don’t know if maybe it’s more a library, but I think it nicely shows how we can have so many approaches to authentication. If you want to create a library that’s popular, you have to support all of them. Otherwise people will just skip it because it is not supported. But as a user, in contrast, you only probably want 10% of what it offers. Robert [42:02]: And I think it’s also making the complexity rising exponentially, because when you have multiple things like database models, logging, observability, payments, view controller and you need to have everything integrated. Adding one case, it requires you to add it very often everywhere. So obviously it’s much harder to make it flexible because you are often focused on providing one path that works. But if you need some flexibility around it, just maybe not compatible with the rest of the components. Miłosz [42:39]: And this one is often very, very useful. If you have a proof of concept app or just a trivial CRUD domain. Then having this happy path scenario and you know, one model for everything that’s actually quite useful because you don’t waste time on this mapping and so on. Robert [42:57]: That’s true. But from other sides. Well, experience tells me that often applications are not CRUD. I mean, I like to think about that. Okay. If your application is simple CRUD, maybe spreadsheet is enough. Miłosz [43:13]: Exactly. That’s that’s the usual case, right? Like, why would you build custom software if it’s that trivial? For stuff like this, this no-code solutions should be ideal. Robert [43:27]: Or spreadsheets. I love spreadsheets. Miłosz [43:29]: Or some of the more, more complex spreadsheets that are now quite popular. Robert [43:36]: Oh, come on, you’ve seen what you can do with Google spreadsheets and that entire startups can live on Google Spreadsheets, and it’s fine for a while. Miłosz [43:46]: Yeah, I think the biggest trap is if you think you have a CRUD domain, but you have not, and you start with this single model for everything, and then you realize it when it’s too late. And we have a lot of articles about this on our blog, so you can take a look. There’s this one to watch out for the DRY, don’t repeat yourself principle, which is about this issue. So we will link this in the summary later. So you can take a look. And it’s also about the mapping issue I mentioned before. And I see that Grzegorz in the chat asked about this mapping. So there is a mapping library in Java mapstruct and it is common to use it. So why can’t we do it in Go. And how did we overcome the mapping problem. So the issue with the magic mapping we created was that it was just hard to maintain because there was a lot of reflect code, which is hard to grasp and test. And, you know, long term it will be just it would be easier to just map the structs manually. And that’s what we did later in later projects. So that’s how we overcame the problem. We didn’t. We just kept separate structs and mapped one to the other when we had to. And I don’t know, maybe it sounds boring or bad. It is quite boring, but there is a big plus of boring code. It’s trivial to understand and test and change. Robert [45:40]: And it’s also compile time checked for that. So if you break something, you written it properly. Compiler will tell you that it’s something wrong here. Miłosz [45:51]: And today we have so many, you know, copilot tools that can generate it for you that it’s even better than it was a few years ago. It’s basically free. So that’s you know, that’s our approach right now. We prefer explicit to magic. Robert [46:10]: And I think what I can recommend is just trying it because again, it may sound boring, but yeah, I would recommend trying that and just feel that it’s not that scary. And actually it works quite nicely. And again, don’t be afraid of boring code or boilerplate because again, we tried different languages also where it’s more promoting some magic, some elegant code, but at the end later it’s super hard to understand. Like Python is probably a great example that okay, it’s in general allowing you to create nice interfaces that look really nice, like some decorators, but at the end you have no idea what’s happening under the hood, and in Go, you just cannot do that. And it’s very explicit. And you can navigate in the code up to system calls to understand what’s happening. And it’s very linear I would say. So there are just navigating and you see what’s happening. Miłosz [47:12]: But maybe there’s also a good idea now to remind of the no silver bullet principle that now what we just said about this manual mapping and all of this is especially nice if you have if you work with a team in a bigger project and you know, we have you have to have some consistency and maintainability for the long term. And we talk about this in the previous episode. But again, if you have a proof of concept application or something you work on just yourself or want to try to build some MVP, then of course just doing this single model might be a better idea, maybe just simpler and easier in the short run. Robert [48:02]: Yeah, I think it’s a really good point. And usually where the problem is, it’s when people are going into extremes, like always do it like this and it’s it’s usually the bad idea. And yes, you said so. It’s also not like where you always using separate model for everything. No actually we often using one model for multiple things. But it’s in case when the model is the same everywhere. So for example, or sometimes and sometimes we’re also wrong. I mean, we’re creating some two models for something. And we later see that okay, we’re always mapping it 1 to 1. Maybe let’s just make it one model and whatever. So it’s important to be kind of mindful probably in that. And if you see that you’re doing some stupid mapping and it’s always the same. Because what we just said a couple seconds ago about this mapping manually, etc., it’s good for the models that are different. But if you are always have the same fields in two models, maybe you don’t need to have two models. Maybe it’s fine to have one model. And it’s also a matter of encapsulation, because sometimes you need to have DTO when you are doing an encapsulation like in some of our projects that. So for example in Wild Workouts project that we have. So in this case it’s we are introducing some transport DTOs to have encapsulation in domain layer, but again, it’s not always needed. If you sometimes have some pretty simple models encapsulation, whatever you have, small team, whatever, you have 100 people team, you know, it’s totally different thing. And spoiler alert. So I think it’s also in the topic of clean architecture that also a lot of people are saying like, oh, clean architecture is overengineering or whatever, but okay, how complex is the project that you are mentioning? How many people were there? Was the person that was using clean architecture aware how to do that? Maybe it was overengineered, but maybe it was not the proper implementation. Miłosz [50:02]: We’ll talk about it in two weeks. About clean architecture. I think what helps is knowing both extremes, because then you can just consciously choose one method. It’s much worse if you don’t know about it. Just stick to one and use it for everything. But what’s also tricky is consistency, especially in a team. If you map structs for the entire project and then someone wants to skip it in one place, it usually rises long discussions about should we do it? Should we not do it? But I think if you work in a team, it helps to have more strict guidelines. So you just get rid of those discussions or you know, someone has to decide. Robert [50:53]: All right. So time to go back to frameworks. So we already know where more or less we should avoid that. Let’s name it vendor lock in. So we should watch out for frameworks that can become deprecated. Or there are some problems with flexibility. And especially in Go when we don’t have the frameworks that are standard one. So we may be more afraid that something will be deprecated or maybe non-flexible. So what do you think are the alternatives in in this case. Miłosz [51:30]: What worked very well for me before is just modular libraries, you know, just connecting libraries together. And it can also mean internal libraries in your company or in your project, the so-called common package. That’s also considered bad practice sometime. But I think you need some consistency usually. Or you want some consistency across things like, you know API communication, messaging, maybe database as well across projects, especially in a team or across company, usually makes sense to have some standards around this. So you don’t have just wild west, everyone doing whatever they like. But the difference is, instead of this one framework that you have to fit into the project, you just you use the libraries for where they make sense and sort of like building blocks or puzzles. You just pick what you need and leave the rest. And if something changes or breaks or you want to remove one thing, it’s quite easy to do. And I think what’s also very nice is you hide the complexity from like whoever uses the API. I had this experience of setting up a code base in a company in Go, and I took this approach. And for example, we used Watermill for messaging, and I hid it behind internal library. And there were all the bootstrapping, all the settings and so on, which can be quite complicated if you use some Pub/Subs. And the nice part is, you know, my colleagues didn’t need to know exactly how it worked, although it was documented and quite easy to grasp. But as a user of a library, you usually want very small API, right? So for example, if you use an HTTP router, you don’t want to, you know, work with the internals of the protocol. You just want to say /users GET and that’s my function. And that’s it. You don’t want to configure it further unless you want to do something more specific. So then you can. But I think it’s a nice design philosophy in general to have small APIs. Robert [54:16]: Yeah, and I think in practice it’s not that hard to usually do that. Again, we’re mostly talking about Go now, but I think often it may sound like rocket science. And I think a lot of people are really promoting frameworks like it’s helping me to create stuff so fast. And yeah, you can also if you go to the frameworks websites, you can also see some big logos using those frameworks. But you know, don’t be fooled by that that much because I’m not sure if you know how often getting some logos on your website work, but you know, if you have some colleague working at some big company, you can ask him to use your library or framework in this company, and later you can put this logo. So, you know, it’s often more like that instead of, you know, I don’t know, some Airbnb using some big framework, so you know, it can be just used in some back office application used by couple people in the company, and it’s fine. Miłosz [55:20]: Sometimes if you read those stories, it seems like the framework is the success behind the company or something. And when I read this I feel envy because, you know, I would like to be as a successful as they are. So maybe I will use this framework. But most of the time it just gives you this boring boilerplate. That’s quite nice, but also not life changing usually. Robert [55:51]: And sometimes even opposite from my experience. I mean, I’ve seen some projects where again, it wasn’t that it was couple months to remove the framework. It was rather impossible to do that even. I think I would find some projects that I built. I use a framework and sorry for everybody who is maintaining it now, but I was young and I was optimistic, but I’m pretty sure that if they was able to get rid of that, that’s great, but I’m pretty sure that they didn’t. And it’s probably a mess to. Miłosz [56:25]: What we didn’t cover yet is in-house frameworks, which can also be a trap because then you have no one maintaining it if you don’t do it. Robert [56:38]: So I think it’s probably some balance between some internal libraries and that are using something that is pre-built versus frameworks. Miłosz [56:46]: I really like the internal libraries approach, and especially to hide some open source libraries behind it, behind some API. And one example might be observability. So I think for a logger, you know, it’s I like to expose my own logger type as just a small interface and then use some implementation behind it. And I think you replaced logrus recently with slog in our project. And we used the same approach. And it was quite easy. Robert [57:25]: Yeah definitely. And probably the same goes for metrics and for tracing. So usually metrics and tracing the interface is not very big. So most of the interface is matter of setup. And but the function to do trace to report metrics. It’s pretty small. So you can and well it looks that those libraries are also changing pretty often. And if you look in the longer time span it it may change again. Miłosz [57:55]: And the metrics and tracing calls will be scattered all over your your code base. So if you stick to the. You just import the library directly everywhere. You will need to redo it everywhere the same way. And maybe not a big deal. You know you can just do some search and replace or whatever, but it’s much easier if you stick to your own interface and then just replace the implementation behind it. And that’s quite a nice outcome. And you don’t need a framework for this separate package. Robert [58:31]: And what’s also nice is that you can also hide a lot of flexibility that those libraries are often supporting. So you can expose a simple interface without a lot of complexity flexibility. And it can also standardize how, for example, you are doing metrics. Miłosz [58:50]: And maybe one counterexample for frameworks or we can have from our own is Watermill. Right. So for people who don’t know Watermill is our event driven library, and we especially had this idea in mind in the beginning to make it not frameworky at all. So the idea is that you should be able to replace it with anything else you want. And of course, it depends on how tight you couple your application with it. Hopefully you don’t. But yeah, that was the idea from the beginning. And I think it it works pretty well right now in our projects we could just, you know, throw it away and use something custom. Robert [59:42]: And yes, I think it’s nicely answered the question that I mentioned earlier. So how that you can understand if something is coupled to your application by understanding how easy you can remove that. And for example, if Watermill. Well, you can do it pretty easily because you can still keep some application using Watermill and consume messages by any programming language practically. And it will be just compatible because under the hood you have messages on some message broker of your choice and it’s compatible with everything and vice versa. For example, if you have some event driven system that is already there, you can implement Watermill into that because Watermill is not doing anything specific basically. Miłosz [01:00:27]: Yeah. Robert [01:00:32]: One more thing that is also interesting about if you think about frameworks and I think it’s visible in the different languages, maybe you heard sometimes the let’s say title like Spring developer, Symfony developer or whatever. And it’s actually kind of makes sense because there are people that are let’s say experts in all this framework works and how to use that. Miłosz [01:01:01]: I remember like ten years ago, there was a big part of software houses were using Ruby on Rails, for example. I bet they hired experts. So it made sense. Or maybe still makes sense sometime. But… Robert [01:01:21]: I think it’s one big downside because you know, those framework knowledge how it tends to become out of date. So you know you can spend days weeks or learning something about framework, but it can be outdated. And if you switch to go to other programming language or to other company, a lot of effort that you spend to learn stuff, it will be just wasted. Because this is not, let’s say, universal knowledge that you can put in other projects. So I think it’s also worth mentioning that I would not really recommend to specialize in some frameworks, because you can use your time more wisely. Miłosz [01:02:08]: And you can learn some patterns that are unique to the framework. And for example, if it supports messaging, you can you could learn how to send a message and receive it, but you don’t learn how it works underneath. So maybe it’s a better idea to learn even driven architecture, because you know the theory behind it and how it works in general, because it’s better transfers to whatever you will do later. And maybe in general, some more timeless skills like how to split modules in your application, how to make it decoupled, how to write business logic so it’s easy to read and modify. Robert [01:02:55]: Yeah. And there are a lot of discussions now like will AI replace us or whatever. So I would say that if somebody will be replaced. So probably as framework developer, I would be afraid the most because this actually something in what AI is pretty good. So about having the knowledge about all the quirks about frameworks, how to use some strange use cases and how to connect everything. So it’s an example. So I don’t like to write front end, but AI is actually pretty great in that. So if I’m for example, using react and there are some magical things that I have no idea that they existed, I knows that and can use it pretty nicely. And I can imagine that there are people that have all this knowledge how it works. But well, I would be a bit afraid that I may have less, let’s say, job opportunities. And again, it’s also the risk that it will be out of date and from other side. LLMs are not that great in things like design, modularization and you know, splitting bigger problems into smaller. It’s still something that you can do better than it. Miłosz [01:04:10]: And you need to have the knowledge to verify if hat the AI spit out is valid, which is also important. If you do it blindly, it might not end well. So, you know, there are some skills that are here to stay probably whatever you do, like those timeless architecture patterns and so on, which probably make more sense to learn than just one framework and be done with it. Robert [01:04:39]: And I think it’s also something what we promote, and we also see that it works for us, because I think one of the good examples is the repository An article about repository that we have on our blog. So I think we have written it five years ago, and I don’t remember how many times it was on Reddit on /r/golang, but I think it was again there for one week ago with more than 100 upvotes. And I don’t know, probably, I don’t know, 20,000 visits again. So, you know, we written an article about some knowledge that is universal and it was valid five years ago, and I’m pretty sure it will be valid in next five years. Miłosz [01:05:21]: So yeah, even if you use AI heavily to, you know, to update your code, I guarantee if you use repository pattern, it will be much easier to work for the AI than to just have raw queries. Robert [01:05:34]: You know what to suggest to it. Definitely. And so yeah, this this is the approach that we recommend and we see how it works for us. Because you know we’ve been learning from PHP frameworks 15 years ago. And, you know, it’s not helping us that much now. I mean, it was nice because we learned that, okay, things can be built in this way. Some things are possible. But we also from other side was focusing on the things that are universal. And we can just use for longer. Robert [01:06:13]: All right. So time to summarize what we said today. So the thing that I already mentioned. So I think it’s not like it’s always bad to use frameworks. And especially if you are a beginner, it’s pretty useful because it can make you more productive. And it can help you to focus on things that matters. And later you can try to reproduce that what you seen in frameworks and create your own simple libraries that are doing that. Miłosz [01:06:47]: Yeah, but don’t become a framework developer. Just focus on more timeless skills and universal knowledge because the framework specific stuff will probably get outdated soon. And if you learn the universal concepts, it’s easier to join a team that needs something similar, but maybe not exactly. In your in your domain. It’s easier to pick up new skills then. Robert [01:07:15]: Also watch out because I think there is a lot of promotion, let’s say on social media about frameworks. Maybe some is paid, maybe some not. But it’s often focuses on the bright side that okay, it’s making you productive. It’s simplifying stuff. But I think it often misses the long term implications that you see. You will see when you use this for 2 or 3 years, and team will become bigger, and you may being the trouble. And it’s even harder after a time, because you’re spending more and more time on workarounds. And when you spend weeks or months on doing workarounds, it’s pretty hard to just remove them because of skunk. False sunk cost fallacy. Miłosz [01:08:00]: But it’s useful to have some experience with frameworks. So at least you know how they look like, how they work. Especially with stuff like code organization or modules or observability. So you can maybe recreate it with modular libraries later. And you know, seeing a project suffer from framework use may also be useful because you see where the issues are. And, you know, you can often see it in in products, even if you don’t work there. I think, you know, products where it seems like initially they iterate so fast and deliver to market, and then you log in one year later and it looks the same just because there were some some nice initial speed but then lost over time. And yeah, if you join a project like this as a developer it’s a nice chance to learn some anti-patterns as well, which will help you. Robert [01:09:08]: And again, if you join such company and in the first week, say, okay, we need to remove this framework and let’s try to do that. Let’s listen to the previous episode because it’s also not if product is in the stage that okay, it’s all coupled and you cannot maintain that. It also requires some proper approach on refactoring that. And yeah, remember that Go is also pretty specific in terms of frameworks, while in other languages there are some well-established frameworks in Go it’s not. And I’m pretty sure that the situation will not change. So in Go in general, we don’t recommend using frameworks as long as maybe there is some proof of concept or maybe some your pet project. So it may be nice to see okay how some stuff can look like, but for bigger project with more people in general were not recommended in long term. Miłosz [01:10:08]: I can recommend module libraries approach like internal libraries for your company. That usually works works well. You have some consistency. You can do stuff one way, but then easier to throw some things out if you need. You don’t need to stick to it. Robert [01:10:29]: All right, so is it time for Q&A? So let’s see what’s happening in the chat. Where is chat? Hello. So Go is a great choice to build a LLM powered applications, but it is visible to use Golang for developing LLM models. Viable. Yes. So I think I can agree with that. Go is pretty nice language for building applications that are using LLMs because. So if I would for example compare it to Python. So for LLMs it’s often nice to do some gathering of the data for example asynchronously. And good luck with Python. Miłosz [01:11:29]: I actually work on this currently for our platform. And yeah, Go concurrency model works is great for this. Yeah, but for developing models, I don’t think I can say anything. We don’t have experience with this. Robert [01:11:47]: Obviously, but I think, you know, most probably 99.99% people are not developing models. And it’s fine. Like you are not building your own CPU so you can use something that other people did already. I think you can build almost anything in Go, mate. Yeah. Miłosz [01:12:13]: You can, but as as the next comment says, as Konstantin says yeah. I think Python is the tool for now for most deep learning. So yeah, we don’t have any experience with deep learning in Go. So we can’t really answer this. Robert [01:12:32]: But yeah probably Python will be the choice. But from some recent news it seems that you can build TypeScript compiler in Go. Miłosz [01:12:40]: Oh yeah. Robert [01:12:41]: So yeah, it seems that Rust is not the only option. Actually, it was pretty fun to see how many people were pissed because they didn’t use Rust for that. I actually wondered if any of them used write something in Rust. Or maybe it’s some cult. But let’s leave it. Yeah. Daniel mentioned that I just signed your platform guys three days ago. I want to grasp skills in Go. Yeah, I hope you like that. Miłosz [01:13:11]: Welcome on board. Robert [01:13:12]: And yeah, we should also have some actually AI related stuff in our platform. So we should have AI mentor soon. So if you are starting to learn, Go. I think it will be also pretty helpful to with some more complex exercises that we have there. So stay tuned. Okay. The next message. When I think of frameworks in Go, the first one that comes to mind is GoFr, which is super useful for smart and concise microservices with all powers like observability, gRPC, HTTP and queues. I didn’t hear about this one. I’m not sure if you did. Miłosz [01:14:01]: No, not really. Maybe we just stopped looking at new frameworks in Go? Yeah. I think like what we mentioned before applies. Basically it’s I’m sure it works well. Just be careful if you have a bigger project that will be there for a few years. Just keep in mind maybe you will want to migrate out of it. Robert [01:14:27]: And updating five microservices is easy, but 50? It’s just much, much, much harder. Miłosz [01:14:35]: I’m sure it works for many people, for many projects just, you know, keep in mind those cons of it and consider what would you do if you wanted to move out? Robert [01:14:49]: So thought experiment. Can I remove that easily. Miłosz [01:14:52]: If it’s just, you know, gRPC, HTTP and queues then it sounds like just inputs and outputs of your application. Maybe that’s if you keep it separate from the business logic. Maybe that’s fine. Robert [01:15:04]: Yeah. So in other words it’s not MVC framework. So it’s not coupling everything. So it should be easier. Miłosz [01:15:14]: Okay. The next one how to how do you set up an API project. You expect to be big. Do you follow clean architecture? Do you use a package for dependency injection? Yeah. If we expect the project to be big, then clean arch or sure is what we will use and we will cover this in the next episode in two weeks. Robert [01:15:37]: But watch out because I think everybody expects that project will be big. Reality is often different. Miłosz [01:15:44]: So but this is also something we talked about a bit in the previous episode about when to write high quality code because this is similar similar question. Right. I expect this, this product to be to be big, and you have to be careful not to go too much too far with the foundations. So, you know, be wary of thinking like, I need to set up all this foundations now. So later I will just add features and it will be super quick. Most of the time it’s not like this. You will need to refactor the project anyway, whatever you do, because it can change drastically. So it’s better to start with some modular libraries, maybe have some reasonable setup in place, but don’t go too crazy. Robert [01:16:47]: Yeah, and remember 90% of startups fails. So if it’s startup, you know it will be good to not fail because of you. Spend too much time on making your product future proof. So. Miłosz [01:17:03]: Yeah, dependency injection. We used the wire from Google before it was sort of okay, but also had some issues. I think later we switched to just doing manually. Robert [01:17:17]: Yeah, yeah. So wire was nice because it’s code generation based. So I think it’s the best what we used so far, but also it’s kind of become very complex at the end. So we found out that it’s just better to inject by hand because you see what’s happening there. And at the point when you feel that, okay, it’s super hard to inject everything. It may be not a problem of DI, it may be a problem of your code, let’s say. I mean, maybe your dependency graph is too complicated, and maybe this is the problem because I think it’s a good example. DI is a good example that it can surface other problems. And some people may think that, oh, okay, it’s a problem with the DI framework, but in most cases it’s not a problem with DI framework. It’s a problem with your code and dependency graph that you just have too many dependencies, too many interfaces. So the most often we think that, you know, there were interfaces for things that have one implementation and it was injected and it was multiple layers of injection. And, you know, just having it one level in the AI and simplifying it, it just made things simpler. Miłosz [01:18:35]: So yeah, I think we need to talk about it in the next episode as well. Combined with clean architecture because it’s related. But it’s also nice concept of because if you start on a project now, you expect to be big. I think dependency injection is something you shouldn’t be thinking about. because that’s this kind of overengineering maybe like just, you know, start with some something simple. Now start with manual dependency injection and see, I don’t know, three months how it works. If you know if it’s painful then consider some framework for dependency injection. Maybe the framework library for dependency injection. But before it happens, maybe you are overthinking it. Maybe just start. Robert [01:19:26]: Start simple and remember that a lot of boilerplate doesn’t mean that it’s something wrong with that. I mean, it’s just boilerplate. Yeah, but as long as it’s manageable, it’s fine. So this is pretty similar to what we mentioned a bit earlier about structs. So if you’re passing values between structs and doing stupid mapping it’s great. I mean it’s not a problem when you have dumb code and boilerplate. It’s actually good because you can understand. And you know, the alternative will be reflection and we will have no idea how it works. So having basic, you know, you can even have 1000 lines of code file that is doing injection. It’s fine. Yeah, you can totally manage that. And it’s easier to manage than you think. And for example, if you for example, you start to have dependency, like some things are starting to be dependent on something, you just often need to move big blocks of code from bottom to top. And you actually can see what dependence, what state. And if something doesn’t work, compiler will tell you it’s bad. Miłosz [01:20:31]: And yeah, exactly. That’s what my experience with, you know, recent my recent projects with main.go Being 2000 lines long with just project setup. It seems really dumb, but it works. Basically if you extend some constructor, you go to main.go and you press the shortcut in your IDE that says next error. You go there, fill the missing pieces and you’re good to go. Robert [01:21:02]: Yeah. And alternative with this dependency injection frameworks is often some magic that you don’t understand. Highly recommended to try to do it by hand. Miłosz [01:21:13]: So sometimes boring is nice. Robert [01:21:19]: Okay, the next question or comment? We don’t know yet. Actually, we’re reading that without checking what’s there. It’s risky. We’ve been pushing my team toward Cleanish architecture. We have modular monolith modules, use domain driven design patterns, but we break the rules where abstractions are too much overhead. That I think makes sense. So. And it’s also totally fine that we also are doing it from time to time that we were thinking that it will be more complex, but it’s not. And it’s important to again be mindful and see that, okay, it’s not serving anything. Let’s maybe remove domain layer and put everything to application, remove encapsulation. And I know it’s hard to do sometimes when you spend time on that. It’s hard like removing unit tests I know, but it’s sometimes something that you’re doing and it’s it’s fine And it’s nothing wrong in being trying to do, trying to or actually to overcomplicate stuff. Everybody does that. We are doing that. You can try to do your best and you will be doing that. It’s fine. No worries. Miłosz [01:22:31]: Yeah, we kind of mentioned this in the previous episode about breaking rules, so it’s fine to do it, but it’s best if you know why you do it and that you actually are breaking rules, because if you do it by accident, then it’s much harder to decide when to change the approach or when to fix it. But if you do it on purpose, then it probably is fine because you probably know what you are doing. It’s much better than being dogmatic about something and just, you know, announcing that, okay, today we do everything domain-driven design way and any other approach is is wrong. And that’s asking for trouble because there’s no one approach that works for everything. So yeah, that’s sounds like a good, good approach. Wilson. Okay. Robert [01:23:28]: There is another comment about LLMS. I think there is a difference between deep learning and LLMS or and also LLM-powered applications, the latter opening up to more languages. And since it is just just a bunch of API calls, Go is well suited. Yeah, I totally agree. Miłosz [01:23:50]: Yeah, we currently work with with this like adding LLM features to our platform and using it in Go is really trivial. That’s pretty cool. Robert [01:24:01]: Yeah. And again I think I like to compare it to like CPUs or something like that. It’s pretty similar situation that you are using something done in some totally different technology. That’s totally fine. You don’t need to fully understand. It’s useful to understand how it works. But again, it’s something under the API and it’s encapsulated from you. And you can also kind of exchange that. What are your favorite frameworks and why. Oh that’s interesting. So yeah probably as we said earlier. So we don’t have really framework that we like in go but in other languages. So I remember PHP times and Symfony was quite nice for that time. And I like how modular it was. What else? Miłosz [01:24:58]: Yeah, I’m having a hard time coming up with something because I usually felt overwhelmed. Robert [01:25:07]: I won’t mention anything front end. Miłosz [01:25:11]: Yeah. I think I would just say my favorite approach is having those smaller libraries used together. Not in a frameworky way. So I wouldn’t say I have a favorite framework. Robert [01:25:26]: Yeah. But yeah, it totally was useful to use from frameworks because you have some reference that you can, you know, how it could look like. So it’s helpful. Okay. I’m not sure. For the file structure it seems go has philosophy for some directories like cmd, pkg. So I wonder what is the correct way to structure an API project for the most cases? That’s the hard one. I think we probably should do episode just about structure structure in your projects because it’s pretty okay. Miłosz [01:26:05]: It’s okay. This is a follow up to the API project that will grow big. So I think the answer here is very similar to before. Don’t overthink it in in the beginning because you don’t know how the project will grow. Right. Robert [01:26:24]: Yeah, yeah. But probably it’s something that we will cover in the next episode in two weeks. But I think some sort of simple, clean architecture can work. So just having, let’s say three packages even like application interfaces and adapters and you know, it doesn’t differ that much from just having everything in one package, but it’s already some start and it doesn’t cost that much to do that. But again, probably something to go deeper in the next episode. Miłosz [01:26:56]: Yeah. So for sure some separation makes sense if you use the clean architecture, but you don’t have to worry about stuff like should you use it in cmd or pkg because there is no official Go guidelines for doing it. Robert [01:27:12]: There is one package that looks like official, but it’s not official. And I think the Go developers already mentioned that that it’s a bit misleading. I don’t remember the name of that, but it’s something like I don’t remember the you know what I mean? Miłosz [01:27:29]: Yeah, I know this standard Go layout or something like that. Robert [01:27:34]: Yeah, some standard Go something. So but yeah, it’s not official. It’s worth mentioning. Miłosz [01:27:38]: But my point is it’s not that important. You know, if you, if you use src and then you want to migrate to pkg, it’s just you know, you do git move and you will do it for you and you will keep the history of the files. So that’s not the important part. It’s you can look up our wild workouts example. We have some similar structure to clean architecture. But, you know, you don’t have to worry that much about it because you will need to adjust it anyway in the future. So I would avoid this idea of, you know, let’s let’s prepare this, let’s prepare this perfect structure now and in the future we will just add features later. That usually will end bad because you don’t know how the project will change over time. So instead, start with something simple. And you know, if I would take it to the extreme, you could start with a single file and see how long you can maintain it, and then just start splitting it into files. Of course it’s extreme, so don’t do it. Robert [01:28:42]: But yeah, it depends. How big is your team? If your team is one person. Miłosz [01:28:45]: Yeah, maybe it’s fine. Maybe you can start with a single package and then then grow. I think it’s much easier because you don’t make assumptions about how the code will look like in two months, because you don’t know, and it will be a surprise probably anyway. Robert [01:28:59]: But yeah, if, for example, your team is ten people, Some lightweight, clean architecture definitely can help. So. Miłosz [01:29:05]: Yeah. Yeah for sure. So. But as we mentioned last time, clean architecture can be really simple, right? Yeah, but maybe more on this in two weeks. Yep. Robert [01:29:18]: All right. Okay. So it was comment for this one. So how it’s connected to being idiomatic. So yeah, it’s it’s good to be idiomatic. Miłosz [01:29:30]: But yeah this is a controversial topic. You know what’s idiomatic. What’s not idiomatic. Need to be careful with that. So you don’t spend too much time worrying about being idiomatic in Go, which can mean different things. I think we have the fifth episode about unpopular opinions about Go. We have. So maybe maybe more later. So yeah, this is what we have already covered before about the mapping and what is our opinion about GoFr? We mentioned that we didn’t use it, so nothing to add here, I think. Yeah. Daniel says we are doing a great job. You’re welcome. Thanks for joining us. Grzegorz asks if we know Substack. It seems to me that your newsletter should be so accessible. Maybe you will gain new observers. Yeah, we read Substack. At least I do sometime. With our newsletter. We just prefer to be independent, I guess. So we don’t plan migrating. Robert [01:30:55]: We already did three migrations of our newsletter. Miłosz [01:31:00]: But the nice part is it was invisible to our readers. And with Substack, you never know. It seems like a, you know, cool platform right now. But it’s also a startup. So, you know, I think a similar thing happened with Medium ten years ago or so when they became very popular, and then they started putting paywalls. And basically, you have no control over how the how your newsletter looks like. Robert [01:31:31]: And I’m pretty sure that it doesn’t have the all flexibility of newsletter platforms like. So we are doing a lot of things with our newsletter, like the podcast notifications and automations. So there is a lot of stuff under the hood, and I’m quite sure that it may not support that. Miłosz [01:31:51]: So yeah, so we don’t plan on creating but thank you for the suggestion. You can share our newsletter with your colleagues to help us get new observers. And then Wilson says go fx is great. Why would you ever want to migrate away from it? Okay. I guess you misread GoFr. We didn’t use go fx, right? Robert [01:32:26]: Yeah. Yeah, but from what I see, it’s So this one, as far as I remember, it was reflect based. So this was probably the Miłosz [01:32:36]: This is a reactive framework or this is a dependency injection? Robert [01:32:39]: Dependency injection. Miłosz [01:32:40]: Okay. Robert [01:32:40]: So I think I think it was reflect based. So it was. Miłosz [01:32:46]: Okay. Yeah. Robert [01:32:48]: So it was the thing. Miłosz [01:32:49]: So yeah I mean it’s a library, not a framework probably. Right. So you can if you wanted to migrate out of it. It’s probably easier than just a framework. Robert [01:32:59]: Yeah, yeah. But yeah, in terms of dependency injection again, we it’s not that hard to do it by hand. And it’s super explicit and it’s much easier to work with. So yeah we recommend this approach even again don’t be afraid of big files because it’s really simple to handle. Miłosz [01:33:16]: So yeah. What are your thoughts on urfave CLI or is there an example to the DIY go CLI program? We like it, right? This is the one we use for the tdl cli. Robert [01:33:34]: I think I’m just checking because I don’t remember. Miłosz [01:33:38]: Once, but I think we do. And I think it’s quite cool. Robert [01:33:46]: Yeah, yeah. So I think that this is our favorite one. So we have our CLI for our training platform and we are using exactly this one. So other alternatives I think there are not that idiomatic if you are thinking about that, but they have also some weird quirks. And this one has a very nice API and just works super nicely. Miłosz [01:34:10]: So we don’t have an example I think of building a CLI tool. Robert [01:34:16]: But you can check our CLI tool for platform. Miłosz [01:34:19]: Oh yeah, you can, you can do it and. Robert [01:34:21]: I can put it probably to this chat into the screen. Miłosz [01:34:26]: And our going one evening training has some CLI exercises but it’s very basic so I’m not sure if that’s useful. But thank you for the suggestion. We may maybe we should write some article on this, although I think the documentation should be good enough to start this. Really? Not that much. Robert [01:34:52]: I’ve put the link in the screen if you would like to see some CLI example, so you can check this one. It’s a framework worth mentioning. I mean it’s if you ask the question how easy it will be replaced with something else. Quite easy. Miłosz [01:35:07]: So I’m just maybe we speak about CLI frameworks. There’s also the bubbletea the one which I used for the pq library in Watermill. Maybe we can also link it later. I mean, it’s a bit different because it’s for building TUIs, but if you consider building a CLI app, that might be useful. Robert [01:35:36]: All right. So I think there are no more questions on the chat. So last couple of seconds to leave something and and and. Yeah, it’ll be it’ll be time to finish for today. So as we mentioned a bit earlier. So we have our newsletter. So this is the best way to be notified about new episodes. So if you are subscribing to us on YouTube or any podcasting application actually a couple people asked us when will be available on podcasting applications. So we’d like to record one more episode and we’ll upload all three episodes there. And if you’re listening on Apple Podcasts on Spotify, it means that we already did that. But we highly recommend to subscribe to our newsletter so you will not miss any new episode because, you know, YouTube algorithm or Spotify algorithm. We don’t. We cannot impact if it will deliver you the notification newsletter. You will always know that there is a new episode, and the next episode will be about pretty hot topic about clean architecture, and I think we’ll have a lot of nice insights. So highly recommended to be with us. And also, it will be great to hear your comments on that, because it would be great to hear some external feedback, let’s say, for that. And yeah, if you have any comments questions, please leave it on under the YouTube video or on Apple Podcasts or Spotify. We’ll read everything. And if you can include anything to a new episode, please leave it. Yeah. Don’t remember. Don’t forget to give us a thumb up in the YouTube or five star review in Apple Podcasts and Spotify, so it can help us to reach more people and give us some motivation to record new episodes. And yeah, something more? Miłosz [01:37:38]: That’s it. Thank you, everyone, for joining us. Robert [01:37:40]: Thank you. And see you in two weeks. Bye! Miłosz [01:37:42]: Bye bye

Quick Takeaways High-quality code is mainly about keeping good iteration speed over time - can you add features without breaking what works? Not all code needs to be high quality - focus your efforts on the code that’s most important, changes often, or creates the most value. The right time to refactor is after you know your product has value, but before technical debt gets too big - make small improvements bit by bit. Architecture decisions need more thought than other code quality factors since they’re hardest to change later - ask “how hard would this be to remove?” Balance between MVP and quality depends on your situation - for experiments, cut corners on purpose; for critical systems or enterprise products, focus on quality from the start. Introduction In the first episode of No Silver Bullet live podcast, we talk about the balance between writing high-quality code and taking shortcuts. Based on nearly 20 years of working together on various projects, we discuss when it makes sense to move fast rather than aim for perfect code, and how to avoid technical debt that can kill your project. We focus on making mindful engineering decisions instead of blindly following rules like “always do X” or “never do Y”. Different situations need different approaches to code quality. Notes “No Silver Bullet”: The classic 1986 paper by Fred Brooks that our podcast name references, discussing how there’s no single development that will solve all software engineering challenges. Our Learning Platform: The project we discussed that started as an MVP and was later refactored Architecture Patterns: Clean / Hexagonal Architecture (we’ll cover in more depth in a future episode) Domain-Driven Design (DDD) Pareto Principle: The 80/20 rule we mentioned - often 20% of the code creates 80% of the value Future Episodes: Why you shouldn’t use frameworks in Go Is Clean Architecture Overengineering? Development Practices: Mob Programming - team programming approach for handling complex refactoring Time-boxing - setting specific time limits for refactoring efforts Quotes Quality is not about some kind of elegance of code because it’s just an artificial thing. It’s not really helpful for your team if it’s pretty. Miłosz Often you have in the code places that you don’t touch often or it’s not earning a lot of money… Ask what are the places that we’re changing the most often? What are the places that are creating most of the value? Robert I remember one project when we started applying Domain-Driven Design… Everyone loved it in the team… But what I also remember is that it had no paying users and we had to shut it down. Miłosz If you did your dirty POC, don’t miss the time when you should clean it up, because it’s easy to go into a spot where it’s no longer possible to do that. Robert After working with [legacy systems] for long enough, you might think, ‘I’ve had enough of this, I won’t allow my next project to rot like this one.’ It sounds like a good idea, but it can also be a trap. Miłosz Sometimes it can be even the opposite. Having everything super consistent can actually be worse than having inconsistent things, because keeping this consistency requires effort… Sometimes it requires you to use an approach that is not optimal just because ‘we’re doing it consistently.’ Robert A useful mental model here is to care about the useful product first, not the technical design, which of course is important, but I think it’s easy to overvalue it. Miłosz Try to find some places where you can do refactoring in one week… Often you don’t need to rewrite an entire service. You can just do some refactoring in the code and iterate on that. Robert Timestamps 00:00:03 - Introduction 00:01:22 - What is code quality? 00:03:55 - Why quality matters 00:07:00 - Traits of a healthy codebase 00:09:46 - What high-quality is not 00:13:32 - High quality outcomes 00:14:45 - When to skip high quality 00:19:27 - High-quality project story 00:22:30 - Impact on company success 00:25:54 - MVP issues 00:29:40 - How to approach refactoring 00:35:36 - Balancing code quality 00:36:03 - Learning platform example 00:41:30 - When an MVP makes sense 00:43:57 - Where not to cut corners 00:56:14 - CI/CD is crucial 01:00:17 - Company culture 01:06:33 - Good enough & edge cases 01:11:18 - Quality tips 01:24:47 - Q&A Session Transcript Miłosz [00:03]: Hello everyone! Today we talk about writing low-quality code. So most often you want to build high quality software, which makes sense, but sometimes you do not. Sometimes good enough is better than perfect. But when does it make sense? And how to avoid technical debt killing your project? I am Miłosz. Robert [00:27]: And I’m Robert. And this is No Silver Bullet Life podcast where we discuss mindful backend engineering. We spent together with Miłosz almost 20 years working together in multiple teams and multiple projects. And during that time we learned that following advice like “always do X” or “never do Y” is not the best idea. It can limit your growth or lead your projects to failure. In this show, we’ll share multiple perspectives that will help you to make smart choices that we believe may help you to be promoted to senior or principal engineer earlier. Miłosz [01:04]: We want to make it interactive. So if you have any questions, drop them in the chat. We will answer them at the end. We plan a Q&A session. Or if you have any comments about what we are currently talking about, we might pick them up early during the conversation. Robert [01:22]: Exactly, but I think it would make sense to start with the question what the code quality is and why it matters. Because this is not something that we are often thinking of. I mean, many people are trying to implement a high quality code, but without really thinking why they are doing that. And what does it mean that it’s high quality and what it gives to that person? So what do you think Miłosz about that? Miłosz [01:46]: I guess it has many, many layers, probably a long topic in itself. But at a very basic level, I think you can think of the software quality as, you know, the few bugs it has, and this software does its job well. But I think for today that’s not that interesting topic because everyone understands that probably. For me what’s more interesting is the codebase quality, let’s say. And the best indicator for me of it is the iteration speed. So how fast can you add features to your product or just develop your product and while not breaking what it already does well. And what’s probably important is you do it over time. So it’s not for a short project, but over months or years. As the team grows and as the product changes, you are able to keep working with a good pace. I guess in other words, you are not afraid to change code. You can always come back to it and add something new or modify it. So yeah, I would summarize it as iteration speed combined with developer experience. And I guess they often go together well, which doesn’t mean the business is successful because of them yet. But often if you have a team that likes working with a software, it usually indicates that the project is doing well, at least in terms of how fast it is developed. What do you think? Robert [03:47]: Yes, I think you are right. And well, what can I add more? Miłosz [03:52]: It’s a good question why it matters at all. But I guess code quality is something, you know, a topic that exists as long as software itself. And, you know, we have many books and papers from decades ago. People were already thinking about this topic. And for me, I come back to it every time I talk with some friends who recently changed jobs. And they usually tell me something like, you know, it’s okay, but I didn’t realize the codebase can be that bad. Robert [04:31]: But I think it’s quite funny in disguise because you can have signals like that. It’s hard to work with that. But for other side, you can see in the age of AI, let’s say, many people that are you see that they can iterate with code pretty quickly. Just use AI to build stuff. And well, quality is not best in this case, but in some cases they are taking money, making a lot of money. So it can probably indicate that it’s probably not that simple. And there is probably some hidden complexity there. Miłosz [04:59]: And I wonder how it will change. I guess no one knows at the moment how AI will change this. When I work with, you know, like code generating tools, I mean, AI generating tools right now for code, it’s often, you know, there are like two extremes. I’m either super impressed by how well it does or it produces some garbage. And if I didn’t know what it’s doing, I would be lost probably in the generated code. Robert [05:33]: I think we can argue if AI can improve and maybe handle this code better. Let’s see. So obviously large language models have many limitations. Probably many of them can be overcome. But yeah, it’s probably let’s see. It’s probably I would not probably say anything, but it will be like that because I know that it’s easy to say that, yeah, it will be solved. AI will be better, but I think it’s not that simple. And all this old knowledge that you mentioned that from old papers, old books, I think it’s quite funny that sometimes those papers, books have 40, 50 years old even. And, you know, they are still pretty relevant. A lot of people forgot about that, about them. But I think they have a very good point. By the way, fun fact, you can probably check from where our live podcast name is coming from. Miłosz [06:26]: I guess we can summarize it as we still have quality issues as an industry and probably will be even more important topic now. So yeah, I wonder what are some, you know, traits of a healthy codebase. Do you have something that you can, some ways in which you can describe a healthy codebase? Robert [07:00]: Yes. I think all of that will be connected to what you said earlier, that it should have a goal of supporting faster iteration. So the first thing that is coming to my mind are tests, obviously. So if you have tests, you are able to, you have tests and they are pretty fast, you are able to iterate on things pretty fast, because one thing is that you are not afraid that you will break anything that somebody or you did earlier, and if they are fast, you’re iterating quickly. So this is important thing. The second thing that is also quite important is modularity. And it’s especially important when your team is bigger than just you or just five people, maybe just ten people. Because in this case, when you have much bigger team, having good modularity helps you to change things independently. So I think it’s pretty important to consider this in the high code quality. And the third thing is that how easy it is to understand your code? Because you can have a lot of tests. You can have multiple people working on this codebase, but if it’s taking ages to understand how something works and nobody really knows how something works, it’s also very hard to work, to implement something, because you don’t know what you should touch. And it’s taking really, really long time to changing logic there. Miłosz [08:27]: Yeah, that makes sense. And probably all this is automated, so you don’t waste time running in circles during development. So it also kind of sounds like developer experience to me. Robert [08:44]: Yeah, definitely. Miłosz [08:45]: It’s rather developer experience-related than product-related. Robert [08:51]: From other side. I have some impression that often developer experience is really about infrastructure. So you know, DevOps and all this stuff. And obviously it’s important. It’s supporting all those things, but I think it’s not the most important thing. I mean, you can have great CI so it will help you to run your tests. But if your tests are bad, the best CI in the world will not help you. If your modularization is bad and you know having the best Kubernetes cluster in the world will not help you because even it can, I think, make things worse. Because changing one thing will require changes in ten microservices for example. So this is opposite of. Miłosz [09:30]: Yeah, like like we’ve seen in real life. Robert [09:33]: Yep, yep. Miłosz [09:34]: The platform can be great but it doesn’t matter. It doesn’t help if you don’t have those more basic pieces in order. For me, it’s also easier to define what isn’t high-quality code. And I come back to times where we started programming a long time ago, and we didn’t know many patterns back then, but somehow I still knew there is a code I like working with and code I don’t like working with. And later I realized it basically boils down to I like working with the code I wrote recently, because I have all this context in my mind, it’s easier to think about it. And I think it’s also a common trap that when you enter a new codebase, you might assume it’s low quality, because maybe it follows some other patterns that you know and you have no mental model of it yet, you don’t know where to find what. And that’s why, you know, it’s like, it often happens that a new joiner in the team recommends overhauling everything because it’s weird or confusing. So that’s and it’s also about our own code for sure. Like, I’m sure you remember, like thinking what an idiot wrote this. And then of course it’s… Robert [11:12]: Git blame: Miłosz Miłosz [11:14]: Yeah, of course it was you or me. But this is pretty much the same thing for me. So you forget the codebase over time and you don’t remember it, but it doesn’t matter it’s low quality. It’s just hard to work with for now because you don’t have this context yet. And similarly, quality is not about some kind of elegance of code because it’s just it’s an artificial thing. It’s not really helpful for your team if it’s, let’s say, pretty. Robert [11:54]: And I think it’s also hard to measure impact of that. I mean, if you think that your code is elegant, so how you see how it affects any of positive stuff that you mentioned earlier. So yeah, definitely, this is a trap. And it also reminds me that some people like some kind of consistency in the code. So it’s actually kind of quite close to that, that people like to have everything consistent in the code. And I would say that it can be even opposite. So having everything super consistent can actually be worse than having inconsistent things, because keeping this consistency, it requires effort to do that. And sometimes it requires you to use some approach that is not optimal because it’s okay, we’re doing it consistently, it doesn’t make sense, but we’ll do it because somebody said that, okay, we’re doing it consistently. At the end, you just have garbage. Miłosz [12:49]: Since you mentioned that DevOps and platforms, it’s maybe a good example of this because if you have this dream of having a uniform platform that all teams use in the team in the company, then all of them need to follow some similar standards. Robert [13:07]: Oh, it’s the worst if somebody from outside of your team is coming to you and no, no, you shouldn’t do it like that. You need to do it in that because and it’s even worse if this is the person that really like consistency. So yeah. Yeah, definitely. Miłosz [13:19]: So consistency is useful sometimes. Sometimes, not really. As usual, there’s always these extremes are bad. Yeah. And I guess we can summarize this as high quality codebase means quicker delivery of code overall like from very high-level perspective. This is the outcome you want. You want to deliver quickly with stable pace. So for me, I think… Robert [13:57]: Probably it’s a useful question to probably ask yourself like okay, will it help me to. So if you are thinking about code quality, will it help me to deliver stuff faster? Or maybe not. Maybe it is just elegance and it will not give us much. Miłosz [14:15]: Maybe it’s easier if you think of it as maintainability. If you want to keep working on this project over time and do it with a steady pace, that sounds like something where you want high quality. Robert [14:31]: The thing that we mentioned already a couple of times is the fact that we are often talking mostly about how to write high-quality code, but I think it would be interesting question to ask if we always should aim to highest quality code, because it might may make sense. May sound like it makes sense. I mean, you would like to deliver fast, so we’re keeping the code quality high, but literally Miłosz [15:06]: Yeah. Does it make sense to always aim for high quality? Robert [15:09]: Yeah, yeah. Miłosz [15:10]: Well, as you know, there is no silver bullet, so probably not. Sometimes low quality can be a strategy or a tool you choose consciously. But also, as always, it depends on the context and some factors probably. Do you know some? Robert [15:37]: I think that there are a couple of things that could help us with thinking if we should aim for high quality or not. So the first situation that I can imagine is that if we don’t know if this code will be useful for us, so we’re doing some experiment. We’re doing some product that we don’t know if anybody will use. And we know that our long term maintainability may not exist because we may discover that, okay, actually it’s useless. Nobody will use that. So spending hours on making this code maintainable, it will be just waste of time because often maintainability requires some effort at the beginning. Often also we are implementing things that we’re not really touching after implementing that. So if you think about your projects so very often, it’s like that you have some modules or some parts of the applications that you are modifying often and reading often. But there are also some parts that somebody written couple years ago even, nobody have idea how it works, but nobody is really touching that because there is no need to touch that. And it’s totally fine. I mean, it may sound bad, like we should clean up everything or whatever, but it’s the reality that we’ll never have time to do that. And it’s also worth mentioning that we are also trying always to do applications as robust as we can, as stable as we can, but we forget that often those applications are not really mission critical applications. And if it will be down for day, half day or hour, nobody will care or even if somebody will be a bit upset, it’s fine. Miłosz [17:21]: It’s not a nice thing to think about if you develop the application. Robert [17:25]: I’m not saying that all applications should be like that, but some applications definitely doesn’t require that big that big stability, et cetera, et cetera. So it’s also worth mentioning that it’s not always that critical to just have stable applications. Again, I know that it may be not always logical, but I think it’s often visible if something will really be broken because, okay, if you ask your stakeholders if okay, it should be stable. Yeah. Yeah, of course you have to make it stable. Later it’s down for a day. Nobody really cares. Miłosz [18:04]: But as someone developing this product, you probably don’t want to think about it like that. You don’t want to work on something that doesn’t matter. But sometimes it is the case. There are more critical projects. I have a theory of how the extreme take on high quality starts because we often work with some legacy systems. And, you know, after working with one for long enough, you might think, oh, yeah, I have had enough of this, I won’t allow my next project to rot like this one. So from now on, we will start doing everything high quality and we won’t accept any technical debt or some half measures. And it sounds like a good idea, but it can also be a trap if you swing too far to the other extreme. Robert [19:17]: Do you remember some projects that you worked in and it maybe didn’t need to have that high quality code? Miłosz [19:27]: I remember the one when we started applying Domain-Driven Design and similar patterns in Go the first time, and I’m often reminded of it when I talk with former colleagues who worked with us on it, and they usually, over a beer, mention something like, do you remember this project? It was so great. It had all the the best patterns and it was so cool to work with. Everyone loved it in the team and I remember, it is true. But what I also remember is that it had no paying users and we had to shut it down. Robert [20:15]: At least it was beautiful. Miłosz [20:19]: Yeah. It might be dangerous to daydream like this because most projects don’t look like that. Are not perfect examples of patterns or whatever you want to see in them. But this project was actually a very important moment for my career because I realized how little all those discussions mattered in the end. Like, we try to do the best job we could, but after, you know, after you see the project dead, then you realize, you know, maybe this one hour meeting on naming conventions or something like that wasn’t that crucial in the end, you know, of course, you never know. That’s the hard part. But I think it’s like a good perspective coming from this previous approach of “I hate technical debt, I will do everything to do things right” to more balanced approach of “Yeah, maybe we don’t always need to spend so much time on, you know, the high quality stuff.” Robert [21:41]: If I’m thinking about that, it’s also one interesting thing. So survivorship bias. So you mentioned that okay this project failed. But I can imagine if this project wouldn’t fail we can think at the end that yeah okay. So it’s all thanks to our elegant code etc. and I think you should all watch out for that. That okay, if people are saying like as we said at the beginning, always do X because it will make your project successful. And it worked in our big projects that everybody knows. But you know, you don’t know if this was the thing that made this project successful. And it’s probably pretty unlikely. I mean, good technology is useful, but I think it’s not the most important thing at the end to reach the product market fit. Miłosz [22:30]: Maybe it’s just not related that much. You know, there are companies that are very successful, but with probably terrible code quality. And on the other hand, there are failed companies with with great code. Robert [22:46]: And I think it’s also the opposite. So there are even more successful companies with with great code quality, but I think we will go touch on that a bit later. Miłosz [22:55]: Maybe a useful mental model here is to care about the useful product first, not the technical design, which of course is important, but I think it’s easy to overvalue it. Robert [23:16]: And also we don’t want to sound like okay, never write good quality code. We’ll go into that. No worries. Where to, when to it makes sense to go with high quality from the beginning and we’re not too sure or take shortcuts, so wait for a while for that. But I think we may also sound now like, okay, always write bad code. It’s not also like that. Robert [23:43]: What I think is also pretty important is the fact that it’s more maybe related to what your boss expects from you. So for example, if your boss is saying that okay, it needs to be great code scalability code. So your and you are creating for example, proof of concept. So your job is to ask boss, hey, are you sure that it’s the right time for that? And if he will say that, yeah, yeah, we need to handle thousands of transactions per second and make this code great. And you are not able to convince the boss for going the shortcuts. Well, probably it’s better for you to say what he ordered. I mean, at the end, he’s kind of your customer, and you are building what your boss is ordering from you, so not to be in the situation, like, okay, in one year, for example, your traffic is super big and your boss is asking, well, but you promised us that it will be scalable and it’s not. And you are saying, oh, you know, we needed to do something and it’s also not the best situation. So also watch out. Watch out for that. Miłosz [24:57]: For sure there is some responsibility on you to explain. Maybe this is not the best idea. Especially if it will take a long time to implement, which usually it will. It’s if you aim for like huge scale or something before you really need it. But at some point, yeah, you’re not the owner. Robert [25:21]: And remember, if you ask it for the first time, you’ll always hear the question. No, no, it needs to be scalable and high quality. You need to go deeper and say, okay, we can do that. But for example, instead of doing this feature for this time, it will take this time. And, you know, be pretty specific. And you know, your boss is thinking in terms of money and time is money. So, you know, if you use this argument, it can help. But again, it’s not always possible. Miłosz [25:54]: Maybe we could go the other way now and choose some opposite project where actually high quality would be nice to have. Robert [26:03]: I remember one project like that and this is a good example that it’s hard to find a good time to do this refactoring or improvement, because sometimes you are starting this project as a proof of concept. You’re doing it in a dirty way, but at some point you are at the point where fixing that will take really, really a lot of time. And I remember one project like that. And yeah, in this case, we should start the, let’s say, refactoring much earlier because it will be easier. And now at the end, again, we were in situation that total rewrite would be the only way to fix that. And unfortunately reality is that it’s often hard to do total rewrite. And even if you are able to do that, it’s often happening that it’s failing, unfortunately. Miłosz [27:16]: You mentioned about the project that started as an MVP, right? Robert [27:21]: Yeah, yeah. Miłosz [27:26]: So it started as some sloppy code, maybe just to validate some idea and then grew over time into something that’s hard to refactor. Robert [27:37]: Yes, yes, exactly. And again, it validated the MVP in some way and well, but I think the point when we should start to do refactoring was already gone. Miłosz [27:50]: I think that’s the hardest question. Like when is the moment when doesn’t matter to it doesn’t make sense to keep going the current way. I mean, where is the moment in in the project’s lifetime when you work on the MVP and then you decide, okay, we need to do something about it. We need to change the approach. And when does it make sense to keep going the sloppy way. Robert [28:28]: It’s an interesting question. I think it’s probably hard to answer. So it’s probably somewhere related to the question. If you start to see that it’s useful and anybody will use that and it will make some money and… Miłosz [28:45]: Maybe if you ask yourself this question often, then it’s a good sign that probably something could use some refactoring here. Robert [28:55]: But I think you know what the trap here is that if you are working in a project and it’s bad, it’s easy to get used to that. I mean, so you get used to how bad quality of this project is and you’re starting to think like, yeah, okay, maybe it needs to be like that. What’s useful is probably when some new people are joining your team and every person is saying how bad this codebase is. Obviously, you need to also take in mind that often people are just complaining that code is bad, and when it’s not. Miłosz [29:28]: Maybe they don’t know it yet. Robert [29:30]: Yeah. But again… Miłosz [29:32]: On the other hand, new joiners often have this fresh perspective that’s also useful. Robert [29:37]: Definitely. But I think I can have. So unfortunately, it’s hard to give one good answer when to do that. But I think I can give you some hints on how to approach that, because I think that this is also important part though. How to approach such refactor. The first thing that I would consider is timeboxing this, because it’s easy to go with never ending refactoring project or total rewrite that very often are just failing because it’s too big, it’s hard to I mean, stakeholders usually doesn’t like projects. Like, you are going to your cave for one year and trying to fix something. After one year, you’re going like, oh, unfortunately, we need to have another two years, so it shouldn’t be like that. So it’s important to be able to cut it to some smaller pieces. Miłosz [30:37]: And maybe start with some first iteration of changes in a small scope, you know, smaller module or something like that. So you see if it makes sense or not. Robert [30:50]: And probably it’s good to start with places that are often modified. So you can see the benefit of this. As we said about code quality, it’s all about making your development faster. So if you are refactoring things that nobody touched for five years who cares? Miłosz [31:07]: I think that’s super controversial. I mean, when you think about refactoring a big project, I think the first intuition is to pick the least used thing, you know, where nothing happens because it’s low risk, so you won’t break anything. We can do whatever. But yeah, first of all, probably it’s simpler because it’s not used much. And second, there is little benefit in refactoring it in the first place. Maybe it doesn’t need to be refactored ever, because who cares if it’s not used that much? So that’s tricky part. Robert [31:48]: The second thing that’s also pretty important is trying to find place that is not that big and you can change it quickly. So we are not talking about year because it’s super hard to, you know, do one year refactoring of the thing and later deploy that. It’s super risky or you will probably lose a lot of trust. Try to find some places where you can do. I know one day is maybe very optimistic, but let’s say one week. Start with the question what you can refactor in one week, you can deploy after one week and try to do it in small parts. So often you don’t need to rewrite an entire service. You can just do some refactoring in the code and just iterate on that. The good thing is that if you will do it on smaller iterations, you can see if you will break something where it’s broken. Because if you are refactoring something for one year, it’s super hard to later figure out what’s wrong here, because usually you are refactoring some logic that is hard to understand and it’s a mess to try to fix it later. Miłosz [32:59]: And most probably also key here is to make it working end to end, right? So you don’t work on some unused piece of code for a long time and then plug it in. But do the changes in a kind of vertical slice so it touches on the entire codebase, and you can actually deploy it and see if it works or not. Robert [33:22]: I know that it may sound like a stretch goal, especially for harder projects that okay, how you can do refactoring and deploy it after one day. Okay. So maybe if not one day, maybe one week, but okay, if you are not able to do it in one week parts, it’s maybe worth asking questions what you can do to deliver it in a week iterations. Maybe we need to add CI for this service / project. Maybe you need to add some tests. No, I think it’s useful to just start asking questions. Okay. Why I cannot cut this refactoring in smaller pieces. And if you spend some time to try to figure it out, you’ll find out that, okay, it’s actually possible. And you will see benefits really, very, very soon on that. Miłosz [34:08]: There’s one more trap I see here that when you work with code base that you consider legacy, whatever this means in this context, but basically some older project older, which means maybe it’s one year old even, and it’s easy to assume it just has to be this way because everyone keeps complaining about this project. That there is no time to refactor. And just at some point you just give up. And then comes the big rewrite, which is probably not going to help, or it’s going to take massive amount of work and time to finish. So I guess maybe if you see the the project will be there for a while, it’s not going away. It’s probably a good time to start this refactoring earlier than later. Robert [35:09]: And try to do this refactoring in small iterations. So again, not one year refactoring, aim for one day iterations of changes that you are doing there. If not, start with week. But I wouldn’t go with one month because one month is already very, very long time. So if you are not able to do one week iterations on refactoring, let’s ask questions why and what you can do to be able to do that. Robert [35:38]: Does it mean that it’s not possible to balance code quality? And we need to always start with ugly POC and later refactor it with time, with pain, sweat, and uncomfortable questions from our stakeholders. Miłosz [36:03]: It probably boils down to this question of when. When to do it. That’s what I have in mind. So to take for example, the our academy platform, which also started as an MVP, because we just had an idea to create something new and we had no idea if anyone would use it. Robert [36:28]: As a context. So if somebody doesn’t know. So I think three years ago, we had written down from scratch a platform for learning by building projects. So something that nobody is, I think, doing. And we didn’t know if anybody will like that. So it was kind of risky project, especially that at that time we were working in a daily jobs. So we were doing it over nights, over weekends, and we just didn’t have time to do it better. Now, okay, we know that people like that. Buy at that point we didn’t know. But the good thing is that I think after those almost 20 years working with different projects, we found this good time when we started to clean it up. And it wasn’t that painful. Miłosz [37:20]: It was exactly this dilemma because it was a rough MVP. And, you know, I remember the first version was basically mostly TODO statements in the code. Robert [37:35]: There’s still a lot of TODOs there. Miłosz [37:37]: If we had a proper design session on architecture and design, everything, I mean we had some. But to do it properly, I’m not sure if we would not just give up and just, you know, didn’t have the energy to do it. And I think you just scraped the initial POC in a weekend or something. Robert [37:56]: I also remember that it’s also quite funny how it’s changed with the time. So at the beginning, I think our idea was that you push the code in GitHub and it will run some kind of CI to validate your exercises. And during that time we totally changed how it works. And if we’ll go with this GitHub part and do it high quality, we’ll lose a lot of time and will not be the correct state because we’ll just spend time on things that didn’t matter. Miłosz [38:26]: I remember we had completely different idea about how the CLI would work, but then you just spent some evenings trying different UX approaches and we would never know it just before. But anyway, my point is that like last month or two, we were exactly at this spot where we have more time to refactor it. But you know, it’s a tough decision because refactoring is also not not giving you any product improvement. So it doesn’t feel like you improve the project that much at least from the business perspective. But I think it was a the perfect time to actually sit down and fix some of the issues so we don’t continue with adding more hacks to it, because if we did, then one year later it probably would be too late to come out of it. So we fixed the CI improved it a bit. We migrated to Postgres from Firestore and refactor the codebase a bit. And again, it was mostly developer experience, but I think it was a good investment at the time. Robert [39:53]: And it also didn’t take ages at the end. I mean, you were doing ugly development for three years and yeah, it took some time, of course, but it wasn’t like a one year project to fix that. And we’re able to do rather small iterations. Miłosz [40:12]: And it was also tricky to pick what to fix, because I think it is also dangerous to take a naive approach of “let’s look at all TODOs in the code and fix them,” because there are still a lot in the code base, but most of them are probably not relevant if we didn’t have to touch them for the last three years. So it’s better to look for something that actually solves some problems, even if it’s developer experience, which is quite important for an MVP. Robert [40:55]: Especially in bigger teams. Because, you know, if you don’t know us, we are two people team. So not that big team. But so we can still cut some shortcuts that wouldn’t work in a bigger teams. But still there are some things that like CI, that we’re not cutting corners because our time is pretty limited. We are just two and we are not doing only development. We’re also doing a lot of things around. So, you know, having CI that runs ten minutes versus five, it’s a really big difference for us. Miłosz [41:29]: So if your MVP gets good traction and you know you want to continue developing it, I think it’s better to do it earlier than later. At least some some initial improvements. But this is all about MVP’s and I’m wondering if you think this is the best approach for every project. Robert [41:57]: You may guess that there is no silver bullet. Ha ha ha ha. Yeah, but so this approach worked pretty nicely for our project. That doesn’t require super big scale. We don’t have any external customers, if it will be down, okay, some people will be upset about it but it will not be that millions of dollars of our customers are lost, but sometimes we were working on such projects. So let’s imagine that you are working in a project that will be later sold to some enterprise customers, and you know that you need to take into consideration bigger scale from the day one. And you know, if you cut some shortcuts and you will be lucky to have some customer with bigger scale, with bigger requirements on the reliability. And, you know, it will be not stable. Your boss may start to ask your questions like, okay, you promised us that it will be scalable. It will be handling 5000 requests per second, and it’s barely handling 100 requests per second. And you know, it will be kind of your fault if you promised something else. And your boss may be a bit upset because it can lead to losing this customer. So it’s not like you can always cut shortcuts. You need to be pretty aware of what kind of project you are working on. And it’s no silver bullet. What can I say? Miłosz [43:41]: It’s probably a big decision up front. If you want to start with a rough POC or design something more like proper high quality architecture from the ground up. Robert [43:57]: Do you think that there are some places where you should not cut corners ever? So in other words, if there are any places in your projects that you never, ever should. Okay, never say never, ever. But in most cases, you shouldn’t cut corners. Do you have something like this in mind? Miłosz [44:19]: I think if you want to break rules, it’s best if know them beforehand. So you know the patterns and the better you know them, the better you know when it makes sense to break them. And that’s how you know when cutting corners works best. And because I think our platform is a good example of where cutting corners on purpose is much different than doing them by accident, which might be sometimes something similar. But at least you you know how to fix them. And you can leave some some context for yourself in the future so you understand why this works in a simpler way right now. Why this is a hack, but it’s not an afterthought, but something you do on purpose. Robert [45:22]: In other words it would be good to understand if you break some rules why those rules exists. Miłosz [45:33]: And especially now coming back to the AI discussion, which can spit out huge loads of code for you, which is very useful sometimes, especially if you work on front end and you don’t want to do it. But still, if you don’t understand what’s going on behind the scenes it probably won’t take you far. At least that was my experience so far. Robert [46:03]: There’s also another example that I think cutting corners doesn’t make sense, and this is architecture. So for architecture there’s one bad thing that is usually hard to change. So we often seen that we’re working in some projects that were already successful and in terms of traffic customers. But there were some decisions that somebody made at the beginning of the project, and it was already impossible to change them. So we were able to refactor some services, what were inside of them, but still they were working environment where some other decision like using some library, using some framework. Miłosz [46:52]: Frameworks are the worst for it. Robert [46:54]: Yeah. And you know, if somebody were just be a bit more, more mindful at the beginning of this project and think about this architectural outcomes of this decision, it would be much, much easier to develop something. And sometimes we were participating in some rescue projects that were trying to fix those architecture problems, or maybe removing some frameworks that were used there, but it took like man-months to remove something. So let’s imagine couple teams needed to work for a couple of weeks to remove one library. And it was like just crazy amount of time lost just because somebody used some library at the beginning of the project. Miłosz [47:37]: And it was tightly coupled to all the microservices in the company. Robert [47:44]: What helps in this scenario is modularization. So if you are able to modularize things so you can develop them independently. And this is for example a case in our learning platform. So we have some modules that are super ugly. And I have no idea why it works but it works. But we’re not touching that because it works and it works properly. And if it will be the place when we’ll be changing often, obviously we’ll refactor that, but it works. So, you know, it’s maybe it would be good to make it nicer. But let’s be real, we’ll never have time for that. Miłosz [48:21]: That’s a tricky part, right? Because the architecture should be probably high quality. You want to make it the proper way, but it’s also easy to overengineer it. Right? Especially if you start with an MVP that doesn’t need all this consideration before you can just be stuck with some system that you thought will handle all possible use cases in the future, but turns out it’s not really needed. You will be fine with a simpler approach. So that’s something to be careful about. And on the other hand, no design is usually bad design, so you probably want some initial idea at least about how the system will work. Robert [49:19]: I think one of the best advices here is to build stuff. I mean, because it’s really hard to give one advice like how to not over design things and how to see the things that will get into fire later. And what I think worked, at least for us best was just building, building, building and building. And after time you will break a couple of things. We’ll see. You will see how in some cases it worked badly after time or aged badly, for example, I think it’s, for example, useful to work in one project for a longer time to see how it will really what will get into fire from your initial assumptions and it’s help. And yeah, this is what we always advise people to build, to build and do it mindfully. Because if you also do it not mindfully and just build stuff without thinking about that much, it will not help you much because you will be just in the same position for longer time and will not develop yourself. Miłosz [50:26]: Yeah, it makes sense. Sometimes it also depends on the team size, right? And how experienced people are in the team. So, for example, if it’s just the two of us working on the MVP and you know, I see you commit 50 TODOs in one commit, I’m mostly okay with it because I know you will tidy it up later. But in contrast, if you have a team that’s just formed a couple of months ago and people come from different backgrounds, don’t have that much experience with the current architecture and stuff like that, it makes more sense to have more, you know rules in place. Robert [51:21]: To summarize a bit. So I think there’s one tip that can help you to maybe decide if something is important architecture decision. And this is the question that you can ask yourself. And this question is how? So if you are adding something to the code to think how hard it would be to remove that. So I mentioned the history of removing one library for a long time for the project. And if somebody would ask himself a question, okay, how hard it would be to remove this library. The answer will be “super hard” because it’s just coupled to everything. And you can ask this question about almost everything. So if you are using some framework or you’re, I don’t know, using some external vendor, some database, whatever, you can always ask your question how hard it would be to remove that. And if you don’t have that much experience already with building architecture, building products, it will help you to, you know, start thinking about things. And sometimes, unfortunately, the answer will be it will be hard to remove that. And okay, the follow up question will be do I really need to use that? Robert [52:38]: Often, okay, if you think for that for a while, maybe not. Maybe some frameworks or external vendors, maybe you don’t need to use that. But sometimes, unfortunately, you need to use things that you need to couple with. Like if you are using Postgres for example, removing this later will be harder. I will not say that it’s impossible because it’s possible, and it’s probably unlikely that we’ll do that, but it’s a decision that you are taking that okay, I’m using Postgres and I will need to stay with that. So if you see anything with you like need to stay for longer, really be sure that this is the thing that will not hurt you. Like Postgres for example, I would not say that using Postgres is let’s say risky, but for example, using some vendor that, you know, just had founding one year ago and they released their proof of concept that and your CTO would like to use that. And you see that it will be just hard to remove from your code. Maybe it’s not the best idea to use it. Maybe, I don’t know, maybe Miłosz you have some good examples of such things. Miłosz [53:53]: You definitely need to commit. And it’s a good thing to commit to something because it simplifies many things. Just need to be careful when you pick what to commit to. So if it’s a stable technology that’s been around for a long time, like Postgres or basically any SQL database probably, that’s probably very low risk. But as you said, if it’s something experimental sometimes it also makes sense because it’s maybe you know, something, it’s a perfect match to your product. But the issue is if you pick too many of them and then the risk is super high that something will go wrong. And now we have to migrate. And yeah, migrations are fun. Robert [54:45]: But again, remember some things may be experimental and not stable, but it may be easy to replace. So probably one good example are large language models like Claude or ChatGPT. So the API of it is very small. You are sending a list of strings, and you’re receiving a list of strings in a very simplified way. But more or less it’s that and replacing one with another. It’s not a big deal at the end. So you know. Miłosz [55:17]: That’s kind of how our migration from Firestore to Postgres went recently, which but our code wasn’t exactly ready for it from, from the start even though we promote it on our blog to to use clean architecture and use interfaces for it. We didn’t care that much initially for the POC, but it was very easy to reintroduce. So that’s the part about conscious corner cutting. Robert [55:51]: But it’s also probably worth mentioning that in our platform, we are not really storing that much data in Postgres or we didn’t store that much data in Firestore. Miłosz [55:59]: It was quite simple. Robert [56:00]: Yeah, but there are applications where you have a lot of data and it will be, i would not say impossible because it’s always kind of possible, but it will take a lot of time. Yeah. So there’s also one thing that I think you generally should not cut corners on. And this is your pipeline. CI/CD pipeline. And we already touched on that a bit earlier. So why CI is important. So again CI when it’s unstable, slow it’s just wasting a ton of your time. Again, it’s just me and Miłosz in our company and we are building our stuff. And you know, we don’t have that much time to do that. And if our CI is unstable or slow, we have even less time. So making it fast and stable just allowed us to go faster. And again, at the beginning it often happens that it’s unstable, but in general, from my experience is that if you try to not keep CI stable soon, it’s really, really hard to improve it later because it’s ongoing work to keep it stable and fast. And doing it later will just require a lot of time. Miłosz [57:24]: And you keep using it every day, Every person in your team uses it. So all the tiny issues add up and you just keep wasting more and more time over time. It sounds like we talk about developer experience a lot today, but I think it’s also quite important because the morale of your team is a big factor in pushing the project forward. Like if everyone’s frustrated with a slow CI or, you know, flaky tests or a project that’s hard to set up locally, it’s a hard environment to iterate quickly on anything. Robert [58:15]: And remember, if you need to convince your boss to have time to work on CI, remember your boss is thinking in terms of money. Get the average salary in your team say, okay, our CI pipeline is taking 20 minutes. I can take a tea, eat my breakfast during that time. It costs this amount of money. Our team has this amount of people improving performance of CI by 50, 70, 80% will save us this money. Miłosz [58:47]: It’s one of the best, the easiest metrics you can measure. It’s very easy to track how many time you spend on waiting on CI, and how much it improves over time. Robert [58:59]: And if it’s, you know together with flaky tests, it’s even worse. I mean, I’ve seen CI in my life. That pipeline took 20 minutes, but, okay, I know it’s not that bad. I heard stories about pipelines that are taking days, but let’s don’t go there. But, you know, if it’s flaky and you’re waiting 20 minutes three times, and you also there is some delay when you are retrying. It’s just a very big waste of time. Yeah, and it’s also a lot of frustration. That’s it. And to make it worse, it’s also a lot of context switching because okay, you can drink one tea, eat one breakfast, but after the second retry what you will do. Miłosz [59:44]: That’s also why we spent some time to refactor our CI in the last iteration because we had these issues. It’s much better. Robert [59:54]: And remember, it’s also sometimes fine if you see that some tests are unstable, disable that. I mean, if they are flaky and you don’t see value in them, disable them. Remove them. It’s also fine. Miłosz [01:00:11]: Maybe not fine, but better than retrying over and over. Robert [01:00:16]: Definitely. And I think it’s also important to mention one thing because, okay, I know that it’s already a bit complicated, that there are different kind of projects when you can cut corners, should not cut corners, you shouldn’t cut corners. But unfortunately, I need to add another level of complexity here. And this level of complexity is company culture. So, you know, I know that in some companies you can have time to refactor things after creating POC, but unfortunately it’s not the case in all companies and in some companies out there. If you will develop something, you will never, ever have time to refactor it. So it just makes sense to do things right from the first day, spend more time to do it properly because you know that, okay, if I will not do it right now, I will never have time to do it properly and I will just have mess to work with forever. So it’s also important to keep this dimension in. Miłosz [01:01:18]: And it also depends on who decides about this choice, because you probably don’t want to ask your manager or product person or whoever, “should this project be high quality software?”, because no one will tell you, “No, it shouldn’t be.” So on one hand, it’s an engineering decision, but on the other it also needs to be a dialogue sometimes with the management. If there’s some tight deadline you need to reach or something like this, but it’s easier to talk about, you know, not the technical quality. Like an example of bad idea is, okay, we will skip tests for this to make it faster. Instead, I would focus on cutting the scope of the feature of the product. And sometimes the Pareto principle applies here. So maybe you can find some, you know, like maybe you can cut 20% of the scope that will take 80% of the time to fix, to work on or to deliver. And if you have this dialogue with the product person or product team, then it’s much easier to cut the scope in a way that won’t affect your codebase quality. So you can keep your developer experience. You can keep having tests and a nice environment to work in, but you can cut some of the scope, at least in the current iteration or the first iteration. And the Pareto principle is maybe a nice idea here to ask about the project in general. Like maybe some parts of it can follow more strict high quality ideas and maybe some parts can be simpler. Robert [01:03:32]: I’m not sure if you actually explained how Pareto rule works. Miłosz [01:03:37]: I didn’t. So if you don’t know it, the idea is that often things split in this 20/80 percentages. So let’s say 20% of the code takes 80% of time to write because it’s more complex. And the remaining 80% takes 20% because it’s much easier. Robert [01:04:03]: Or 20% of the code earns 80% of money. This is also very often the case. Miłosz [01:04:08]: It turns out this applies to many things in life. So I wonder about the same. Apply that on the project level where we can focus the 80% of focus, let’s say. Robert [01:04:22]: We already touched on people that like consistency in the code, and we already touched a bit earlier that it’s usually not best idea to do. And it’s also with code quality. So some people would like to approach code like okay, it should be consistent quality and standards over entire code base. But unfortunately in practice it’s a waste of time because like I mentioned multiple times, often you have in the code places that you are not touching often or it’s not earning a lot of money. Or maybe it doesn’t need to be stable because nobody will really care about. So I think some important ideas how to find this 20% that is the most important in our code base is asking a question, “What are the places that we’re changing the most often? What are the places that are creating most of the value of the company? Or what are the places that where the stability of our application matters the most?” And in those places you can use some sophisticated methods like Domain-Driven Design or do some better layering, because this code usually that makes money is often changed. It’s usually the most complicated code in our company, and it’s just worth to keep it high quality because it will pay back pretty nicely because we are touching it often. It’s complex, so we’ll be able to work with that faster. And again, it’s important to mention that not all code in your application needs to be bad. Not all code in your application should be good. It’s well, for some magical reason, usually some sort of part or all. So sometimes it’s enough to have 20% of application that is high quality. Sometimes it’s closer to 80. It doesn’t need to be always 80. I mean, it can be 60, let’s say, or 90. But you know, the idea. Miłosz [01:06:34]: And maybe the opposite of high quality is not low quality sometimes, but just “good enough” quality, which might be a tricky balance. But basically the idea is that you start with the happy path. And at least for the first iteration of a feature, you don’t need to implement everything. You don’t need to cover all edge cases that can happen, because there’s very often a very long list of them. But you can focus on this 80% of important, often used features and implement them good enough. And of course, there’s always this other factor here that if you don’t cover the edge cases, then you have to support them manually very often. And it often makes sense because I remember we had projects where we skipped some manual support for the most weird edge cases, because even our stakeholders didn’t know how to handle them. It was so complicated. So why spend weeks of developer’s time automating it if it happens very, very rarely? Robert [01:07:50]: In other words, if you’re automating something, think if, for example, you are not spending a week on automating thing that it’s taking, you know, five hours a year. Miłosz [01:08:01]: Eexactly. I had a project like this before. And when I calculated it was a dumb process that we had to do every month, but when I calculated it, it took two hours over the year for one person. So it feels dumb that you have to do it manually, but then how can you do it? How can you automate it so it makes it worth to save two hours a year? Robert [01:08:33]: I think I have a good example even from today. So the downside of developing our training platform after hours is that we didn’t have time for developing very basic features, like we didn’t have feature for changing your email address. I know it sounds silly, but again we did most of this time over weekends and after work and we were just cutting all functionalities that were not possible. And okay, you can think what can be hard with changing email address. And yeah, I had the same idea this morning. So I thought, okay, it’s super simple, let’s even try to use Claude Code for generating this. Four hours later. I found that there is a bit more complexity for that. So we don’t have actually that many requests to change email address over the year. But I already spent those four hours and it’s there with some shortcuts done anyway. But it’s often like that you have some dumb thing that you are doing rarely, but you think that, okay, automating that, it will just take one hour and one week later, you see that, okay, it was a bit harder. So I just saved my minus four days and eight hours. Miłosz [01:10:01]: But on the other hand sometimes the small issues compound. And you may have, you know, you might spend a bit of time on one issue a year. But if there are 20 issues like that, you may find yourself doing support for full time. So again, balance. You need to pick what needs to be fixed now and postpone what doesn’t need to be fixed right now. Robert [01:10:38]: I think it depends a lot on the company culture. So the thing that I mentioned a bit earlier. So in startups you can probably live for a while with spreadsheets, CSVs, and, you know, your company is based on spreadsheets and CSVs and it’s fine for the beginning. After you are scaling, it’s probably good to maybe keep track of that, of how much time you are spending on such stupid things and balance that. Miłosz [01:11:18]: I guess it’s good to set some expectations also on yourself so you don’t feel bad about having some low quality code somewhere in the codebase. And I remember feeling bad, especially when hiring new people. You know, you have some parts of the project where you don’t want them to look, but it’s completely normal, right? Most projects have parts like that, especially if you have some legacy software that has been there for a long time. So I think you also need to accept that some parts of it will be there. And be careful about this mindset, as you know, things need to be nice and tidy everywhere, which I think is common among developers. I also feel it sometimes that, you know, I would like things to be nice and tidy and all the corner cases covered and all the TODOs fixed, but very often it’s not really useful to to do it. Robert [01:12:35]: And one more thing. If you are joining some new project and you see some old code that you think is bad, try to maybe not blame previous coders because you don’t know actually in what conditions they needed to work. Miłosz [01:12:50]: Sometimes it’s hard. Robert [01:12:51]: Yeah, yeah. How long they needed to spend on some code. So for example, I have an example that I worked in one project and later I was talking to one person that was complaining a bit why it’s that bad. And later I added a bit of context that we written this code in probably two days and didn’t touch it mostly later because we didn’t have time, but it worked good enough. So it’s like it was not our bad will to write something badly. It was just we’re doing some hackathon or whatever and we just implemented it quickly. It worked good enough and that’s it. And yeah, I think it’s also making your life a bit easier, if you will just try to assume that your previous colleagues that are no longer with you did your best. Especially that I also seen it multiple times that after some person is leaving, some people are tending to blame this person that oh, okay, this is bad because of this person. And you know, truth is usually a bit more sophisticated. And often those people that are blaming this person that left, they’re also not without any blame. So, you know, watch out for that. Miłosz [01:14:09]: Now that you mention hackathons, it’s also an interesting topic to think about. Like how do you run a hackathon and then not end up with a pile of garbage, which we did a few times. It’s interesting. But that’s an extreme example. Most of the time you probably don’t create a new product in three days. Robert [01:14:36]: But again, I think in general the idea is good because sometimes it’s allowing you to really cut corners, because you need to cut corners because you have some deadline to do that. And the outcome I think sometimes can be really interesting because you can go much, much further than you think that normally you wouldn’t be able to do that because sometimes it can even take months because of some stupid conventions. And maybe at the end you will find out that, okay, maybe actually we don’t need to do that because it’s no value in that. So yeah, I can really encourage you to run some hackathons if you are able to do that and, you know, try to not do hackathons that don’t care about functionality, to not use any bad words. And yeah, I think it can be an interesting exercise, but it’s also, again, important to clean it up in places that we mentioned a bit earlier. Miłosz [01:15:37]: But sometimes, the high quality, low quality difference is not that big. I mean, high quality sometimes doesn’t require so much effort, even for a hackathon, right, for example, like using the repository pattern or even clean architecture maybe. Robert [01:15:59]: In some lightweight form. Miłosz [01:16:02]: Yeah. It’s not like a huge investment upfront. And once you do it, it’s easier to manage the project. So even for hackathons, you could use some of the more advanced patterns, let’s say. And it won’t slow you down. Robert [01:16:25]: And the thing that we mentioned a couple of times. So try to be mindful about code quality and remember that it’s something that you need to learn with building over time. So after you build a lot of projects you see better where you can cut corners, where you shouldn’t cut corners. So during this episode, we did give you some hints. But again, I think practice, practice and practice, it’s very important. And it’s also fine to be wrong here. So even if with Miłosz we have pretty lot of experience, sometimes we’re also wrong in deciding if we should go with higher or lower quality. And it’s fine. I mean, as long as it’s not some decision that is not reversible, it’s fine. So I think there are a couple of questions that it’s worth asking. So if you are thinking about code quality. So the first thing is that something that we mentioned at the beginning. So if this code is more an experiment, or we know that, okay, we already have customers there or it will be potentially sold to some enterprise customers that will need to just make it working properly from the day one. So it’s good to start with that. Miłosz [01:17:46]: It will also help you decide if you want to use an MVP or not for this project. And a similar question is if this project is mission critical, if something terrible will happen, if it goes down for a couple of hours, let’s say. As I said before, we don’t like to think our code is not important, but maybe at least parts of it are not. Robert [01:18:18]: It’s also good to think, okay, what will happen with that after we develop that. Maybe this is feature complete part and will not touch it in the next year. I know that, okay, the chance that we will touch any code in five years is probably not that low, but maybe in one year what’s the chance that we will be touching that. If we will not be touching this code in one year, we assume we never know, but we can have some assumption. Maybe we can cut some corners there because if we will not be touching that what it will give us if we make it high quality. Miłosz [01:18:53]: And if you touch it, then it’s good to have some context of the approach that we used. So for example when refactoring our Academy platform it was useful to have this context in the comments or the todos that scattered it. And they don’t look nice, but it was super helpful for me to know okay, that’s why we took this approach. But it also doesn’t mean that you have to get rid of all the TODOs. Sometimes it just, you know, it just exists there, and it’s fine. We don’t need to to worry about it. Because if you don’t change it over time, what’s the issue? Robert [01:19:41]: Like prehistorical paintings in caves? Miłosz [01:19:46]: Yeah, kind of. Robert [01:19:48]: I love those comments in the code like, “it shouldn’t be like that. We should fix it ASAP.” Git blame: five years, ago and everything is fine. Miłosz [01:19:59]: It mostly shows that it wasn’t that important maybe. And I remember having some time in the sprint when we are out of work and it’s time to look through the backlog. And I remember looking at the technical debt issues that we added over the years before, months before, and we always complained there was no time to do them. And finally there is time. And I looked at them and I realized I don’t want to work on it because it doesn’t seem important. So then you can just delete them and don’t worry. Robert [01:20:43]: And if you did your dirty POC, don’t miss the time when you should clean it up, because it’s easy to go into spot when it’s no longer possible to do that. And I think it’s very nicely visible in some products that it’s visible that they were able to develop it quickly at the beginning. But you can see maybe three years later that they were not able to add any single feature to this product because it’s so hard to touch. Everybody is so afraid to touch and change anything there, and it’s just slow death for the product. And it’s already too late because it has too many functionalities that it’s hard to replace and it’s maybe just not justifiable job. You may not justify to rewrite it from scratch. Miłosz [01:21:42]: It’s probably worth to keep in mind also that code quality is not the end goal. It will help your projects succeed over time, especially in the long run. But be careful not to make it the metric, to not care too much about the quality itself, which is hard to define in the first place. Robert [01:22:06]: I know that there are many people that are thinking like, I don’t care if this project will be successful. I just care about money. So if you are this person, right, it’s fine. But if you are trying to find a well-paid job, it’s much easier if you have some success stories. So you just didn’t work in the projects that were mediocre or failures because, well, the market is pretty hard now, and I was interviewing many people in my life, and it’s super easy to see if people really worked in their projects that were successful, and they had some positive impact on how those projects were delivered and how they succeeded. And, you know, if you are just working in the projects that “I don’t care if it’s successful”, really, it will be much harder for you. So if it’s you, really, I highly recommend to try to care, even from a selfish perspective. Miłosz [01:23:06]: Probably also watch out for some extreme tips like “never do X”, “always do Y”, because most of the time it’s not so simple. There are more subtle corner cases and context you need to understand. Robert [01:23:28]: I think there are multiple dimensions of parameters let’s say that are affecting that. So if you are listening to us from the beginning, so you see that just code quality can have that many dimensions and you see that out there there are a lot of people that are saying that, oh, you don’t need to care about code quality, or code quality is super important for how fast you’re delivering. So you can see that it’s not that easy. And I think it’s just showing that a lot of people that didn’t probably work in many projects or didn’t work mindfully in these projects may not give you the best advice. Miłosz [01:24:07]: It’s hard to boil it down to some very simple rules. I think it’s quite lazy if someone tells you something like “just just keep it simple”. Yes, of course, I want to keep it simple, but you can’t just take a 12-person team and tell them, just keep it simple guys, it will be fine. So there’s many, many different ways you can approach any project. Robert [01:24:38]: Check good sources, some good live podcasts, for example. Miłosz [01:24:46]: Okay, I think we are nearing the end. Robert [01:24:48]: Yes, I think we should be able to go into our Q&A now. So sorry for not looking for the chat, but our multitasking abilities are limited. So let’s see what questions we have on the chat. So we’re mentioning a bit of our learning platform. So it’s a proof that we have a lot of fans of how it works. And yeah, how the strategy of cutting corners at the beginning and fixing it later, it works. So yeah, we have “Go Event-Driven was really good. Highly recommended for everyone to check out.” That’s true. Miłosz [01:25:32]: Thank you. Robert [01:25:35]: All right. So time for the first question. Do you want to take it? Miłosz [01:25:39]: Okay. So Rohit asks, “do you think that careful design decisions maybe at low level would avoid refactoring later on? Sometimes I have made decisions that to priority addition of changes over code quality. Is that normal?” Maybe that’s the question whether you want to start with a POC and do something dirty and then figure it out. Or do you want to create some proper architecture from the ground up. Robert [01:26:21]: I think it’s probably important to set some expectations here. So I think it’s good to spend some effort on design because if you always have some design so there is no that you have no design. If you will not make design you will have bad design. But it’s not that you will not have the design, but if you try to do your best, even spend ages on the proper design you probably will also not do it totally right. And it’s fine because you have some assumptions at the beginning how it will work, but it will change over time when you’re implementing stuff, and at some point you will end up with need to refactor something and it’s totally fine. So like we mentioned example of our platform for trainings. So our initial idea was totally different. And at the end we ended up with something different and design wasn’t perfect, but we were able to adjust it with the time. Miłosz [01:27:27]: I think the part I would be careful about here is the avoid refactoring later on. So this is not something you want to aim for. You don’t want to make something perfect now, so you avoid refactoring later because it will anyway happen. So rather better to adjust as you go with some small improvements rather than try to come up with a perfect solution that will cover all future use cases. That usually ends badly. Robert [01:28:01]: Remember about the question that I suggested at some point. So I think design is the most important for things that are hard to change and to know if something is hard to change. Ask a question. Is it hard to remove? If it’s hard to remove, probably it’s good to think about the good design or thinking if you need to redo it in this way. I hope we answered your question. If not, please leave a follow up on the chat. Yeah. Second question is a bit simpler. So, “hey guys, will you be releasing on Apple Podcasts later?”. Yes, we’ll release on Apple Podcasts, but we would like to have three episodes done, so it will take more or less one month will be releasing it to another three platforms, so it will stay just on the YouTube for since we will release two more episodes and later it will be on Apple Podcasts, Spotify or whatever application for listening podcasts that you are using. It will be there. If you are in our newsletter. We’ll also send a summary tomorrow. Tomorrow? Yeah, I guess tomorrow. We’ll send summary, transcript, and all this interesting stuff. Another question. “Is hexagonal architecture worth in the real world?” What do you think, Miłosz? Miłosz [01:29:25]: I want to say yes, but. Robert [01:29:28]: There is no silver bullet! Miłosz [01:29:31]: But I can tell you that, we used it a lot in many different projects and teams. Hexagonal, clean architecture or whatever you want to call it. And one thing I noticed is like this. This is the pattern that many former colleagues tell us about that they introduced in their new team. And the new team members also love it. So I think it shows that’s something developers like working with. And we do too. But of course, if it’s a simple POC or some scrappy MVP, you can probably skip it. But then also it is not a big effort upfront. At least, maybe that’s just my perspective because I’m used to it so much. Robert [01:30:28]: I think it requires more effort if you are doing it for the first time. So I will definitely not do it for the first time. Especially that I think clean architecture is sometimes kind of the trap. So people are trying are overcomplicating that. Miłosz [01:30:44]: And we will be talking about it in four weeks, so I think we will cover it more in depth, but I think it’s one of the most useful patterns we use overall. And also not that complicated. And it’s not even about the exact approach, but just having some layers, splitting the important code from the less important code. Robert [01:31:10]: TL;DR yes, it’s worth. Miłosz [01:31:15]: Definitely in the real world it is useful. Robert [01:31:19]: It’s just important to not over-complicate that, unfortunately a lot of people are doing and okay, also what other people are doing is applying clean architecture without understanding that. So it’s leading to a complicated code. Miłosz [01:31:35]: That’s the issue with many patterns. It’s very hard to understand what someone means by clean architecture. do you remember the rant on Reddit about how clean architecture is the worst thing to happen in Go? And you know, I then read the thread. And basically the author said that you shouldn’t use more than three layers. Robert [01:32:05]: Instant question. So how many layers you are using there? TEN? Miłosz [01:32:11]: That’s the general issue with naming. I think it’s similar with Domain-Driven Design. Robert [01:32:16]: Maybe they are combining clean architecture with hexagonal and onion and using all layers from that. Miłosz [01:32:24]: It’s very hard to understand what someone means by one of the acronyms used in tech. Robert [01:32:32]: Long story short, we’ll touch a bit more on that in the next episode. Please remember to subscribe on our landing page of the podcast so you can. You’ll not miss that. All right. Next question. “That’s actually how I’m developing first version, I write the POC, then I do the refactoring in cycles and until I’m happy with the story the coding is telling.” Yep, it’s a good approach. Miłosz [01:33:04]: That’s iterating Robert [01:33:09]: Yep. Okay, the next one. “Is it better to refactor or code review every certain interval? Because in startup environment almost no one wants to add review as their task.” I think that this is interesting question. Robert [01:33:29]: So what’s my experience in a similar situation is that if you have problem with people doing your code review, it’s probably something wrong with team responsibilities. If in the team. I mean, if people are kind of working on things individually, not like a team, it’s a problem because it’s not the team’s responsibility to deliver something, but it’s individual responsibility to deliver something. And from my experience, this is the root cause because, well, if you are responsible for your feature, why you should review other people’s work, you kind of don’t care. And if you are doing more team work, so you’re planning the work as a team. You’re implementing entire feature as a team. You know, it’s interest of all of you to deliver it faster. But I’m not sure. Miłosz [01:34:25]: But, you know, I think the question is more about the interval. So the way I understand this, should you do like two sprints of product work and then one sprint of refactoring or review? And I don’t have good experiences with this approach of this, you know, like dedicating entire sprint to code quality. I don’t remember it ending well, at least in the projects I was in. So I don’t know. What do you think, Robert? I think it’s better to iterate. So just do small improvements as you go. Robert [01:35:14]: Yes. There’s also one thing in this question that I think it’s also pretty relevant. Miłosz [01:35:18]: Oh, unless you do what we did. So we actually had to dedicate this one iteration pretty much to changing, to refactoring the code base so you can start working quicker. So sometimes this “go faster” initiative also can work. I would just be careful with stuff like okay, maybe it’s about if you know what you want to do beforehand. So if you know we have this issue with the CI, let’s focus one week on improving the CI. Then it sounds pretty good to me. But if you want to do something like okay, we worked six weeks on the product. Now let’s take two weeks to fix the TODOs. That sounds like more tricky part because you don’t know what you what you want to solve. So I would try to, you know, first understand what you want to fix. Robert [01:36:21]: And there is one more thing that, so please let us know if we’ve answered this question properly. We’re after ten hours work day and it’s almost 8 p.m. here in Poland. There’s also one more thing that may be worth mentioning here. So for some hardcore refactorings. So what we also recommend for that is mob programming. So you are gathering with the team. One person is coding and you’re kind of helping this person. I will not cover how mob programming works, but I think that not a lot of people know about that. Worth checking, especially for harder refactorings because it’s easier to kind of synchronize with the work that you are doing with the entire team. I know that for some people it’s like, oh, we’re wasting entire team’s time for this. And it would be easier if one person will work on that. But for other side, somebody needs to review that. And the good thing about mob programming is that you have real time review. So everybody is looking on that and you are kind of doing a review together with them. Miłosz [01:37:27]: The bad part is that it promotes looking for consistency. So you have to be very careful about cutting offtopic discussions or nit picking of how something works. Robert [01:37:43]: But I think it’s good for things that let’s say you will be working on for a week and it wouldn’t be possible to do code review of that, because sometimes it happens that you have that big refactoring that there’s no way to do code review of that. If you are doing mob programming, there is kind of no need of doing code review. All right. So I hope we answered. If not, please let us know. All right. So the next one Miłosz maybe you would like to pick up this question? Miłosz [01:38:14]: “It is always good practice to keep your code at a minimal level without looking for perfection, but at least enough clean to be comfortable to give it to someone else without being nervous about it.” Yep, sounds like a good baseline level. Like, don’t don’t stress too much about perfection. But you know, take effort. Yeah I think that’s actually a good question to ask yourself. Am I comfortable giving it to someone else and having them work on it? Would I be ashamed of myself. I used to, you know, half-joke about this, about our platform, that if we hired someone to work on it, our reputation would be over because they would see all the TODOs in the code. Yeah. So that’s pretty good. Good idea. Robert [01:39:15]: All right. And so time for so far, the last question. If anybody has more questions, you have still some time. So “how do you measure that you’re overthinking and overengineering.” So I’m not sure what you think about that, but I would say that what I personally like is I think that most of us doesn’t like, but it’s sometimes also useful. So having some timebox or deadline. So if you’re timeboxing your work for some time, or giving yourself some deadline, or maybe some days doing it externally. So it’s kind of forcing you to not overthink things because, okay, you have some time to do that and you need to focus on thinking on things that really matter. And I think I noticed it for the first time when I had some external deadlines that I noticed that, okay, I needed to refocus on things that did matter and other things I didn’t have time to run it. Again, I know that we don’t like deadlines, but this is the small, useful part of them. Definitely. I’m not sure what’s your take on this? Miłosz [01:40:29]: I think it’s hard to measure yourself because we don’t like to think that we are wrong about something. But what I like is cutting scope. And I think it’s something we did this week, actually. And when we we have a new project going on and we had an initial idea how to do it and with like a brainstorming session and then we thought, okay, let’s now remove everything that is not absolutely required to make it work. And let’s just finish this v0.1, deploy it and see if someone uses it. And very often you later realize, oh, actually, this is good enough. Maybe we didn’t need all this complexity. We initially planned for. Or as Robert mentioned before, with our platform, we had much more complex setup in mind, first with CI connected to it and all of it, and then we stripped it from all this and focused on the critical path. And that usually helps because you can see that maybe you don’t need all this. Robert [01:41:55]: Also having some kind of MVP, maybe connected with some deadline that you’re putting yourself on that doesn’t need to be that strict. But again it helps. And I think what also helps is to have some end-to-end part working. So for example now we are working to adding some AI on our platform. But also I think we also have some nice idea how to have it working end to end with some limited functionality. And we’re not overthinking that that much because okay, we’re focusing on the simplest part and we can start to overthink it later after we’ll see that it was working. Because the problem with overthinking is that you can overthink things, that you later find out that it’s not useful, and you need to overthink another thing. Miłosz [01:42:45]: Sometimes it also helps to have someone else who can tell you, you know, maybe you just want to use this AI API or this database. Maybe you should do it in your side project, not in this one. You know, like someone who can tell you if you if you go too far, it should be easier for someone else. But then you need high levels of trust to accept this. Robert [01:43:20]: Okay. So I think it would be it for today so we can finish up. So if somebody doesn’t know us yet. So I highly recommend to check our blog. I don’t know if you can see QR code. It’s not that big. Let me try. We’re using this platform for the first time, so let’s do it like this. Okay. It’s not so big. Miłosz [01:43:54]: True MVP. Robert [01:43:55]: This is QR code with our blog, so highly recommend to check this out so there is more content there that is mostly about Go. So if you know Go it may be interesting for you. If you don’t know Go, it may be also interesting for you. If you liked this live podcast episode, fun fact is that this is the first episode that we recorded ever, so I hope that you liked this. So if you liked this, I highly recommend you to subscribe to us on YouTube so it can reach more people. And, you know, you can be also sure that you are not missing any new episode because we are already planning a couple more. Oh, I see that actually a lot of people just joined us because it was 70 people and now it’s 94. Hello for everybody who just joined, we’re finishing, but you can scroll to the beginning to check that. So yeah, if you would like to also to support us, please let all your friends know about that so it will help us to release new episodes. What else? So we already have a couple more episodes planned. So the next one will be why you shouldn’t use frameworks in Go. So if you are writing Go, it may be very relevant for you. If you’re not writing in Go, it may be also relevant because I think it may be pretty universal advice. The next episode will be Is Clean Architecture Overengineering. So we covered a bit about that earlier, but we’ll go a bit deeper in this episode. Miłosz [01:45:50]: That’s a broad topic. Robert [01:45:52]: Yeah. After that, we’ll cover how we believe you can learn faster. So we also covered a bit of that today, but we’ll cover definitely much more. Also if you’d like to not miss an episode we totally recommend to subscribe to our newsletter on our blog, because YouTube may not send you notification or Twitter may not send you a notification. In our newsletter, you will be sure that you are receiving the notifications so you will not miss the episode. And yeah, if you have any comments questions and you are listening it a bit later, leave us a comment on the YouTube. We’ll read that. And if you have anything that we should include in the future, please write down and we’ll try to include it in the next episode. Please also give us a thumbs up if this is already uploaded to Spotify, Apple Podcasts, or whatever, please also leave us a five star review so it can reach more people. Something to add Miłosz? Miłosz [01:46:57]: Thank you, Robert, for meeting me today. Robert [01:46:59]: Yeah. Thank you. Miłosz [01:47:00]: See you in two weeks. Robert [01:47:01]: Yeah. See you in two weeks. Thank you. Miłosz [01:47:03]: Thank you everyone. Bye bye. Robert [01:47:05]: Bye bye.

We registered Three Dots Labs as a company in 2014, hoping to quit our jobs soon to work on our products. The “soon” turned into ten years of launching side projects and working evenings and weekends. We finally decided to take the leap last year, and we plan to share how it’s going. If you’re in a similar spot, you may find it helpful. By the end of 2024, we spent some time planning the next year and our approach to work. We reviewed our backlog and discussed what we wanted to do next. We’ll experiment with 11-week iterations, with a week off in between. It’s not like the usual sprint — it’s more a way to stay focused during this time. We’ve just wrapped up the first iteration, and you’ll find the summary of it below. Our main goal was to get ready for the next iterations. There were many things we wanted to wrap up before starting new projects. We also gave ourselves time to chill a bit after working two full-time jobs for a long time. The main theme was “go faster”, so it’s mostly housekeeping stuff. We decided it’s important to do it now because there are just two of us, and we can’t afford to waste time during development. Our ultimate goal is to create a place where we love to work, and frustrating things like slow CI or a messy codebase are real motivation killers. What we do now falls roughly into three categories: Academy (our learning platform). Watermill (our open-source library). Content (blog posts, podcast, etc.). Academy We had an idea for hands-on programming training a few years ago, but we weren’t sure if people would enjoy learning like this. We created an MVP to see if it’s technically possible. Go in One Evening was the first training we created. The project started as a scraped-together prototype that we iterated on to improve the UX. When it gained traction, we built more features and created another, much bigger training. We did all this while still working full-time, so we struggled to find time and never bothered to refactor the project. We ended up with quite a low-quality codebase that made the most money out of all our products. (I used to half-joke that if the codebase ever leaked, our reputation would be over.) If you ever worked on a professional project, you know it’s an extremely common scenario. There are projects with great technical quality that no one uses, and they die. And there are terrible big balls of mud that make millions. But having technical debt is fine if you take time to clean up some of the mess. For us, it’s an interesting place to be in. When you work as a developer, it’s easy to complain about not having enough time to do things right, while the business needs to move fast and make money. Now, we feel both pressures at the same time. We need to move the product forward but don’t want to get stuck working with a messy codebase. The frustration quickly adds up. We finally have time for things like this, so we focused on them during the last few months. Moving to a monorepo and updating the CI Our CI got slow and fragile over time, and as you may know, it’s one of the most frustrating things to have around when working on a project. It’s also your main feedback loop when committing code. You waste precious time if you have to wait or fight to make the CI pass. While cleaning it up, we also decided to move to a monorepo, which makes sense when only the two of us are working on it. We ended up with one repository that keeps all of our active projects (including this blog) and a set of GitHub Actions that run only when the relevant project changes. It went smoothly, apart from the regular experience of testing the CI (commit, push, wait, read the error, repeat). It also improved the developer experience right away. Various CI workflows in one repository. The pipeline for Academy. As you can see, the test times could still be improved. That’s mainly because of the cache and dependencies. The best part is that we can now deploy just what has changed. For example, changing something on the website won’t deploy the backend, and vice versa. Migrating out of Firestore We started Academy with Firestore as the database. It’s easy to start a project with it, as there’s no need to set up any infrastructure. It was also about 10x cheaper for our use case than hosted SQL. But working with the document model is far from ideal for typical web applications. You can’t just run a quick SQL query to get the data you need among all collections. Migrations are painful, as you must write code to transform the data. And the emulator used in the local environment often has its own issues. So, we decided to migrate to Postgres. Do you know how useful it is to hide your storage code behind an interface in case you want to change the database? We promote it a lot, but we didn’t care that much for the MVP. Some people claim you never change the database, so it’s not an argument for using interfaces. But we had to deal with this exact use case. First, I changed the code to use the clean architecture pattern. Thanks to this, I could test both implementations with the same tests, and changing between databases was a single config change. It’s super helpful to be able to quickly roll back a risky change like this. It also helped to clean up the code from a few hacky patterns we used in the MVP. Still, the change wasn’t as easy as just replacing the implementations. Migrations are always more complex than they seem. I spent about two weeks finishing everything. The refactoring alone took a few days. Then, I had to set up the infrastructure, write the migration script, run tests, and debug issues. We also keep two regions of the platform (EU and US), so it was another factor to consider. It’s a good example of a tough refactoring decision since this change did not improve the product in any way. In this case, we agreed it’s worth it for the long-term benefits. Small issues add up over time, especially if it’s something as critical for the project as the database. Jumping around the exercises in the CLI We added a minor but often requested feature to our tdl CLI tool. You can now jump to another exercise you completed before. Watermill Watermill is our open-source Go library for building event-driven applications. We released Watermill 1.4 last autumn but didn’t wrap up everything we wanted to. A few PRs needed our attention, like #25 in watermill-kafka. Often, a PR looks like a trivial change, but then it turns out to be a rabbit hole. In this case, we had to adjust the tests to make the changes work, which was a solid day of Robert’s work. Another example of tough dilemmas is the issue #402. We often don’t foresee all use cases when adding a new feature. Then, we must decide whether to keep the current behavior, even if it’s incorrect, or change it and potentially break someone’s code. Together with the fix, Robert updated the CQRS examples to show the best way to use the component. We deprecated the gogo/proto marshaler and introduced a new one, cqrs.ProtoMarshaler. It was long overdue, as gogo/proto has been deprecated for a long time. Finally, watermill-sql v4 is still a release candidate. New issues are coming in, and I’ve been struggling to stabilize the tests with the recent changes. Maintaining an open-source project often feels like a full-time job, and we can’t spend all our time like this. At the same time, we want Watermill to stay independent of external sponsors so no one tries to monetize it, which would probably push it towards a vendor lock-in approach. On the other hand, it has a nice synergy with our Go Event-Driven training, so we have a good reason and motivation to keep improving it. We’ll try to find a reasonable balance here. As always, big shoutout to all contributors who help us push the project forward. Content On our blog, it was a relatively quiet end of the year. I released The Over-Engineering Pendulum, a less technical post about my experience with over-engineering in startups. We want to share more on similar topics in the future. Last year, we ran a couple of Watermill release parties and live Q&A sessions for Go Event-Driven trainees. We liked the format, and we wanted to experiment with it more. During this iteration, we started preparing to launch our live podcast, No Silver Bullet. You can read more on it in Robert’s post. We have a long list of topics for the blog and the podcast. After long sessions of sorting them, we ended up with close to 150 items. Changing the Newsletter Platform Our newsletter is the primary way to communicate with our readers and also how we sell our trainings. We’ve been using ConvertKit to send emails, and it worked quite well, but the automation workflows were a huge pain to work with. We had to do some nasty workarounds to make sure the ebooks were delivered reliably to newsletter subscribers. The product wasn’t improving, so we moved out to avoid spending more time on it. This wasn’t the first migration we made. A few years ago, we moved off MailChimp for similar reasons. This time, we moved to MailerLite, which seems to be a better fit for us. It took about two weeks on Robert’s end, but hopefully, it will save us headaches in the future. Plans for the next iteration Looking forward, I hope we’re done with migrations for now. 😅 We have a few goals for 2025, but we try to not plan too far ahead. Often, we get a new random idea and want to pursue it right away (that’s how the Academy started). We want to experiment with the live podcast episodes and see where they lead. Another experiment is adding new features to the Academy backed by LLMs. We have some ideas that should turn out more interesting than the usual “we now use AI chatbot as customer support” stories. Our main goal is to help people move through the training more smoothly. We definitely want to release another training in the coming months. We don’t fully know the scope yet — we’ll ask our newsletter subscribers for feedback soon. Another goal is finishing Go with the Domain, and releasing a revised edition. It’s probably what we get asked for the most, but we’re not sure about any dates yet. Thanks for reading! I hope to see you live during one of the No Silver Bullet episodes and on the Iteration #2 summary.

Leaving a principal software engineer role while having a newborn kid, a mortgage to pay, and a house being built may not sound like the best idea. Still, I took a significant pay cut so I could make a living by educating people about software. Some people literally told me that I’m crazy. Well, Miłosz (the co-founder of this blog) and I decided to build a company we love to work at without VC funding. It wasn’t an easy decision. Finances, of course, are always a concern. Also, we may not have as much time to work after hours as we used to. Let’s also be brutally honest; we won’t be alive forever. Still, we have families we need to provide for, and we want to build a company we can be proud of. Let’s start with how it all began. Our Journey I met Miłosz in 2008, when we were in high school. At that time, we started writing software together — after school and after work (sometimes, we even worked at the same company). For a decade, we have been working two full-time jobs. Our after-hours projects were more or less successful. For example, in high school, one of our online game servers became the biggest one in Poland. We earned enough money so that Miłosz could buy a car, and we even threw high school parties. We were making good money for the time! During that time, we saw how different software development strategies work in different scenarios. We learned how to cut corners and write code in the simplest way possible. For two-team projects, following “the simple way” was usually fine, but for bigger projects, we needed to learn more advanced techniques. Starting this Blog A couple years later, when we were leading and mentoring teams, we saw that some techniques were hard to grasp for our less experienced teammates. Sharing our knowledge to a wider audience sounded like a good idea. So, 7 years ago, we started this blog. Since then, we have created 39 detailed articles. A good example is our article about Live website updates with Go, SSE, and htmx. There are a couple other articles describing SSE in Go, but most of them are quite surface-level. We like to go extra deep to show real-life scenarios. It’s great to see that there is a demand for in-depth content about complex techniques. In fact, every year, our blog is visited by around 270k people! Watermill - Monetizing Open Source While Staying Open Source Around the time we started our blog, we also worked on a project that explored how event-driven architecture would make the application faster and easier to develop. Unfortunately, most of our team didn’t have any experience with this kind of architecture. At that time, we asked ourselves a question: “How can we make building event-driven Go services as simple as HTTP API?” This is how Watermill was born. It is now the most popular Go library to build event-driven applications (at least according to GitHub stars, since we don’t have a better measure). You may be asking yourself: “Don’t you want to monetise it? Is it time to change the license and start charging people?”. That’s not our plan. We treat Watermill like a good marketing tool, and that’s it. We have also heard that Watermill is useful for people, and that makes us happy! We are self-funded, so we can afford such extravagance. Truly, we don’t believe that potential income would be worth losing our community’s trust. The Process of Finding Better Ways to Share Knowledge About 4 years ago (3 years after starting our blog), we released our book Go With The Domain. We already knew the content we were creating was useful. This was the first in-depth book about Domain-Driven Design in Go. We later heard from people that their teams used it as a guide on how to develop their applications. But it was still missing one critical piece. You can read tons of books and watch hundreds of videos and take all sorts of courses, but if you don’t put into practice what you have learned, you won’t accomplish anything.* There is no shortcut here; you need to mindfully use things you want to learn and evaluate the tradeoffs and hidden complexity. The best way to learn is by working on real-life projects. Not TODO apps or calculators; such applications are too simple. You need to learn how to overcome problems you’ll likely face while building real-life applications. Learning in the Age of AI These days, everyone (including me and Miłosz) uses AI during the development. Learning practical skills is even more important now than ever before. AI is widening the gap between junior and principal engineers. Junior engineers now struggle to find a good job. Ironically, it’s hard to find skilled principal engineers who can build applications that AI can’t handle (and likely won’t be able to handle anytime soon). Principal engineers can do their job faster, while junior engineers may spend hours mindlessly copy-pasting AI-generated code without actually learning anything. The demand for engineers who can do simple applications is decreasing, as AI can handle these jobs more efficiently. If you are afraid of AI taking your job, you should find a way to make yourself indispensable. Building a Custom Learning Platform Learning by practice requires a lot of time and patience. You need to find a good project theme and proper materials. It may also be frustrating when you are stuck and don’t have anybody to ask for guidance. To speed up the learning process, we built a learning platform that supports users through building complex, real-life projects. It’s not yet another platform where you watch videos and code in the browser. It’s a way to put what you’ve learned to practical use. Building a learning platform from scratch requires more time than running an open-source project or a blog. We didn’t have a choice; we had to work on it during the weekends and on holidays. We have been very productive! People were surprised when we told them we were going to work full-time at Three Dots Labs. They asked, “Aren’t you already working at Three Dots Labs full-time?” Friday night. 9:37 p.m. Time to party? Nope! Coding! We created a CLI tool, which clones and executes code. Our CLI will clone exercises for you, then execute them. It will save you a lot of time. This is why the feedback loop is so fast: A Platform is Nothing Without Content Our first training on the platform is called “Go In One Evening.” Many people said, “No way! It’s not possible to learn Go that fast!” We love to build things that people don’t believe can be real. We were leading teams of people who had never written a single line of Go in their lives. We saw for ourselves how quickly they became productive - so we know it’s possible. Data we have gathered confirms this. Statistics on how long it took to finish Go In One Evening:50% of trainees finished training in less than 7h, and 75% finished training in less than 12h The agenda includes creating a simple social media platform clone and building the ls CLI tool. It’s designed for software engineers who know how to code already but want to learn Go. We don’t waste your time with how variables and functions work; you already know that stuff. Instead, we focus on things that will be essential in your work. “Go In One Evening” is not our only training. We also have our flagship training, “Go Event-Driven,” where we teach how to build production-grade, event-driven systems. The core idea is the same: we teach by building real-life projects. What people say about our trainings I won’t spend much time describing how it works; I would encourage you to try it! Financials Last year, 746 people joined our trainings, translating to $129,741 in revenue. While that may sound like very good money, it’s less than we earned at our jobs. Our income is not stable yet, so we don’t take it for granted. If you have a steady office job, you may not appreciate that you will always be paid for your work. It doesn’t matter if you have a bad month. In this business, however, you can work on a project for a year, earn $0, and waste your time and money. Hackernews is full of such stories: Hacker News is full of “failure stories” What Are Our Next Plans? This is where we are right now. What are our next steps? We want to continue doing all initiatives, like our blog and open-source projects. But we have a couple new things on our plate, including: starting our own podcast, doing more online trainings, and sharing what we are working on (building in public). That may not sound very original; hundreds of people are doing the same things. But there are a couple of things we want to do that are totally different! Let me explain. Why Do We Need Another Live Podcast? If you go to the gym, you may hear the term bro science. Some guys will tell you things like, “Take this pill, and you can exercise less and still get bigger. Don’t be a moron who wastes time.” For less experienced people, that may sound appealing, especially when you hear it from some big guys. The same thing happens in the software world. There are a lot of bros who give software advice that is supposed to be a silver bullet and make your life easier. I don’t accuse anyone of lying; they are probably sharing what worked for them. However, people should stop advising as if it’s the silver bullet for all problems and acting as though their solution is the only way to do things. Something that works for a solo developer will not necessarily work on a global financial platform. Social media algorithms make this problem worse because they favor polarization and extreme opinions. You may ask, “Why should I care?”. Companies are good at rejecting people who don’t know how to solve company problems. If you don’t want to find yourself out of a job market, you need to learn how to adapt. I’ve been interviewing people for years, and I could easily tell whether someone is truly interested in solving real-world problems or just follows CV-driven development. If you are ambitious, you probably prefer to work with passionate people as well. Joining such teams is not easy – they usually have a high bar to join them. That’s why we’re creating a space where you can learn battle-tested programming practices that actually scale - from solo side hustles to complex enterprise systems. No bro science, just real-world experience. Polarization in Software Polarization is quite prevalent in the software community now. There are two very distinct camps out there: Engineers: For them, technical excellence is important; they care about maintainability Shippers: For them, technology is secondary; they care mostly about shipping as many features that customers can use as quickly as possible; they are known to take shortcuts There is an ongoing war between those two camps. You often hear very different arguments from both camps, like: “It won’t scale,”, “Just ship it,” “That’s not how it should be done,” “You don’t need unit tests,” “It won’t be maintainable,”_ and “Quality doesn’t matter; we can always rewrite it.” Both camps have valid points. The problem starts when people are unable to see the other side. Often, Engineers focus too much on technical excellence, and they may not be delivering anything usable for far too long. On the other hand, Shippers, after releasing a successful MVP, may wade into technical solutions that won’t work for bigger teams or products. Can’t We Do . . . Both? We should be open to diverse opinions. Wouldn’t it make more sense to merge these two approaches? It’s not easy to learn how to balance the pros and cons of shipping and engineering. This requires time. We need to learn how to build software, and we also need to understand how this software affects development. Miłosz covered this topic nicely in the The Over-Engineering Pendulum post. Even though Miłosz and I have been building software together for almost two decades, we still struggle with this balance. NEW! 🎉 No Silver Bullet Live Podcast: A Software Deep Dive from Multiple Perspectives No Silver Bullet ideally illustrates the message and unique selling point of our live podcast: showing multiple perspectives on which tools and techniques work in various scenarios. Over the last 17 years, we have written software for various projects. From legacy companies to startups. From simple applications to global settlement platforms. Most of the time, we acted as hands-on engineers, but we used to be managers and founders as well. We want to share our unique, diverse experience with you. We want to teach you how to apply patterns and use appropriate tools for different projects. Like always, we aim to push the bar even higher. We want you to be part of our show so you can comment or ask questions about the topics we cover. It’s a win-win; we can go deeper into topics you care about and improve our content. In No Silver Bullet, we’ll cover more general software engineering topics. But Go-related topics are also in the queue! Our setup is ready; we will go live for the first time at the beginning of March! We have a list of the first 5 episodes on our live podcast page. We want to go deep for every topic, so we will stream an episode every two weeks. Don’t miss our live podcast episodes! Join our newsletter with over 20k subscribers! Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! NEW! 🎉 Next Trainings and Training Mentoring Trainings are the best way for Three Dots Labs to make money. We also know that these trainings give people the most value for their money. Our trainings are a great way to leverage your skills and, hopefully, find a better-paying job. You may be wondering: “When will the next training be available?” “When will training be available?” This is a question we hear on a daily basis! Before we launch the next training, we want to finish our mentoring functionality, which will greatly help you learn these skills even faster. When you are learning something new, it’s important to be able to ask for help. This can also help you save a lot of time with problems that may be solved by someone more experienced quickly. With the current state of LLMs and our knowledge of your most common questions and obstacles, we can give you hints like we would sit next to you! I’m already sure that it will be one of the coolest features that we have. We aim to announce the next training sometime in March. Why We Are Focused on Go for Now Go is one of the best-paid languages, according to the JetBrains State of Developer Ecosystem Report 2024. You don’t need a PhD to learn it. Compared to other newer languages (like Rust), finding a job as a Go engineer is much easier. This year, we will remain focused on Go trainings. If you don’t know Go yet, how about learning a new programming language with the “Go In One Evening” training? NEW! 🎉 Building in Public We want to share our journey as we create a sustainable business that we love — and that you can benefit from. If you are interested in following us, we will share what worked and what didn’t work every 12 weeks. We’ll share information about our finances, and we’ll discuss what mistakes we made along the way. Miłosz is already working on a summary of the first iteration, which will be out soon! Aim for Sustainable Growth We aim to build a dream company that we genuinely love to work at, a company we don’t want to sell. We are realistic; we always keep a list of things we don’t like to do but need to get done. It’s all a matter of finding the right balance. Often, the “fun things” are not justifiable from a business perspective. Since we don’t have any external deadlines, though, we can work on things that matter and make sure the products we are putting out are the best they can possibly be. We can spend this one extra mile to change good things into outstanding. Like with our trainings platform — we spent a lot of time optimising code execution and deploying it in multiple locations to make it fast. In effect, you may feel like it’s executed locally. Would it change a lot if it were slower? Probably not, but it’s cool! Will We Succeed? Our unusual approach may seem naive to many people. But based on our experience, we believe we will succeed. We always love to question the status quo and blaze new trails. After all, we created Watermill and wrote Go With The Domain. Now, we are working on online trainings. You can expect to see new articles on the blog, and we will continue working on Watermill. Will this strategy work? What if our assumptions are wrong? Will we have to return to full-time jobs to pay our mortgages? The best way to follow our journey is by joining our newsletter. We prefer this over social media, as we aim for a long-term relationship with you. Social media posts show up, then disappear. An email stays in your inbox forever. We are also independent of its algorithms and owners. If you are already one of our 17k newsletter subscribers, please tell your colleagues about our live podcast! Will you cheer us on? Do you think we will succeed? Let us know!

I used to picture my dream job as this: I work for an early-stage startup that recently raised a round. The business idea is promising, there’s much to build, everything seems possible, and we have time to prove the product-market fit. I join as the first engineer, lay the foundations, and pick the tech stack. Can you imagine a better gig? Finally, you get to do things your way. No more arguing and no old code dragging you down. This time, the project will have a high-quality codebase and no technical debt. Starting from scratch feels great — I’m excited just thinking about it. In the end, I joined early-stage startups after someone else had laid the groundwork. The upside was that I could see the mistakes from a safe distance instead of making them myself. I ended up rethinking and refactoring existing codebases more than bootstrapping them. After a few rounds of this and hearing startup stories from friends, I was shocked at how similar startups are. (I love the irony — they try to become unicorns, after all.) One common issue stood out: Product teams often struggle due to the mess caused by over-engineering in the early days. Once the team grows, it suddenly takes ages to deliver what was trivial before. Eventually, someone asks: “Why does it take so long?” The engineers are surprised and frustrated. It can’t be the codebase — it follows the best industry standards. It’s extremely difficult for founders to notice this before it’s too late. Often, it’s a surprise even to the technical leaders. How Over-Engineering Starts You’ve probably seen over-engineering before. It’s a two-person team using ten microservices and obsessing over tiny performance tweaks where they don’t matter. Or adopting the popular but unstable technology and going big on infrastructure when a single VM would be good enough. Or reinventing solved problems and implementing them from scratch. It’s fun to build complex systems and imagine that technology gives you the edge. But it’s rarely true — most software is boring. Only small parts of what you create are unique. Still, there are reasons to think big. The founders have a grand vision for the technology, and they make it clear: We need to build for scale. We’re creating an operating system to disrupt our industry. We have to write the core software now to simply add features later. Perhaps this leads to overthinking what has already been solved for years. If you’re building the next unicorn, why use one database running on one server? It doesn’t make you unique among other wannabes. If you aim for the stars, better design something that supports this imaginary future. But you also need to deliver and validate quickly at this stage. You must keep the momentum as you build the product over time, so you fear getting lost in a messy codebase. You feel pressure to keep the standards high. It’s tempting to over-engineer in the hope of avoiding this mess, but it only makes it worse. It takes the time that you could spend on actually helpful essentials. Who cares if you implemented a service mesh if your CI pipeline isn’t green? Do you need that custom framework if the local environment is awful to work with? How will horizontal scaling save you from unmaintainable code? High Quality Your challenge is finding the balance between over-engineering and building a messy prototype. It’s somewhere between moving fast and doing things right. We tend to see speed and high quality as opposites. You need to be fast to stay alive. But what does quality even mean? Engineers have strong opinions and vague ideas about quality. Some understand it as “I don’t like this code. It should use another design pattern.” They are religious about formatting. They moan, “Give us time to refactor.” It’s a naive take, far from product engineering. It uses quality as an excuse to boost the ego. Code shouldn’t be the end goal. Another take is thinking of quality as flexibility. Building systems that are easy to extend and configure feels rewarding. It’s cool if you want to solve code puzzles for fun. But if you want to deliver working software, you need some constraints. Focus on what’s in front of you. Use a proven, boring tech stack. Stay away from creative or dynamic solutions. If you’re getting excited about building something universal to solve issues in the future, check if you could simplify it into something you can use now. You likely won’t need to scale anytime soon. But if you invest in the basics, they will serve you well once you need to do it. The Essentials In the early days, I like to focus on the developer experience. Instead of creating a framework or platform that supports all possible use cases, I prefer to prepare a great environment for engineers to work in. There’s much to build when starting from scratch, and new work quickly piles up. Good foundations limit time wasted on non-product things. They help you iterate quickly, which has real value compared to refactoring for code prettiness. Above all else, frustration is the worst productivity killer. It can radically change an engineer’s mindset from “it doesn’t feel like work” to “it’s the last thing I want to do.” My first essential pick is an automated build system (a CI/CD pipeline) that runs tests and static analysis, builds the project and deploys it to production — ideally, a modern one that is easy to configure. The same steps run on feature branches, so the main branch is never broken. It doesn’t have to be complex. Make it run git pull if that’s what you need. It’s easy to set up, and the benefits will be there every day for all engineers. You can’t afford to have deploys blocked by broken changes or spend time discussing trivial issues a linter could catch. Next, prepare a great local development environment. It should be easy to set up, work everywhere, and run fast. You should be able to run the CI tools locally with the same versions and results. Use live code reloading so it’s pleasant to work with. Finally, create guidelines on how to do common things. Think of the building blocks from which you build the application. Assembling elements is easier than creating a universal framework. It lets you keep some standards without trying to predict all future use cases. It’s more about healthy constraints than doing things right. You don’t want to discuss every week how to write an HTTP handler, call an external API, use database transactions, or run a function periodically. Decide on something and cut the discussions early so you can focus on writing the code that matters. In any case, watch out for decisions that are hard to reverse. Opinionated frameworks make starting easy, but getting out of them can take long months. Aim for the Middle I like the pendulum metaphor: once we go to one extreme and get bitten by the results, we tend to avoid it at all costs and swing the other way. It applies to many things in life, including technology hype cycles. For example, consider delivering software. A service is a way of deploying code, not an isolation unit. So, use modules for isolation. But be ready to spin up a separate service if needed. Sticking with a massive monolith forever is as extreme as starting with a distributed system. But somehow, we tend to swing from one extreme to the other. Similarly, once we work with a messy codebase, we start paying extra attention to quality. Then, it’s easy to go too far in the other direction and over-engineer your next project. I still find it challenging to find this balance. It’s a topic that leads to never-ending discussions with no clear answers. There’s one thing I’m sure helps: If you have seen the extremes, at least you know to aim somewhere in the middle. Before It’s Too Late When I hear of teams struggling with messy systems, it strikes me how a simple decision in the past could have prevented it. Once you’re there, migrating out is incredibly difficult. In contrast, building from scratch feels much easier. I wish we had metrics to track this, but I’m skeptical if that’s possible. For example, achieving high test coverage is easy, but it doesn’t mean your tests do anything meaningful. One thing I’ve seen work well is regularly shipping small features. The key skill is splitting the work into chunks that make sense by themselves. The naive approach is splitting by artificial boundaries, such as one feature per microservice or separate backend and frontend parts. Instead, aim for changes that are end-to-end complete but small enough to ship quickly. Watch out for making it a target metric, though. It’ll encourage shortcuts in the code, pulling you towards the messy codebase.

It’s Autumn over here, and it usually means another release of Watermill! 🍂 It’s hard to believe it’s already been five years since the v1.0 release. In case you’re new to Watermill, here’s TL;DR. Watermill is a Go library for building message-driven or event-driven applications the easy way. Think of it like an HTTP router but for messages. It’s a library, not a framework, so you don’t need to change your architecture to use it. Over the last year, Watermill grew to 7.5k GitHub stars and 55 contributors, just on the main repository. It supports 12 Pub/Subs including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS, SQL, and RabbitMQ. To learn more, see the documentation and examples. What’s new? Let’s see what happened in Watermill over the last year. For the complete list of changes, see the GitHub releases page. Hall of Fame 🏆 Big thank you to everyone who contributed to all Watermill repositories since the last release! 💪 @AlexCuse @arthurspa @balenio @dferstay @diegofrata @dkotik @finaltrip @igtulm @ivaaaan @kahowell @KirylJazzSax @llulioscesar @looklose @marczahnfc @Mariscal6 @minghsu0107 @mountcount @prOOrc @Provet @qulaz @rjfonseca @russell @SargtLa @sevaorlov @shaneing @Smixi @stong1994 @tobiasjaster @thejoeejoee @thpk @unjello @vitorarins @wolfapple Note Side Note on Design Watermill 1.0 was released in 2019, and the public API has been stable. I especially like the modular design, which allows you to combine low-level components like building blocks to create more complex features. You can see this in rather simple logic of components like FanIn, FanOut, or the Forwarder. You can also notice this approach in the new features described below. We added a few simple components that work well separately, but you can combine them for more interesting effects. Delayed PostgreSQL Requeuer One common issue developers new to messaging have is dealing with errors. People often share an output like the one below, asking what’s happening. Infinite loop of errors. What you see above is a single message failing to process and being retried indefinitely. It blocks other messages, so it seems like the application is stuck in an infinite loop. This is how most Pub/Subs work, and it’s crucial if you care about message ordering. But sometimes, you don’t want a broken message blocking the entire topic. We used to work with Google Cloud Pub/Sub a lot, and one helpful feature it has is retrying messages. Once a message fails, you can make it reappear in the queue after a delay. Other messages are not blocked, and the failed message can be retried later. It’s a helpful feature I missed in other Pub/Subs. There are ways to recreate this, but it’s not as straightforward. For example, you can use delayed messages with RabbitMQ, but you need a plugin, so it’s not compatible with all providers. I thought it would be cool to have this feature in Watermill working out of the box, and here we are! Watermill now supports a universal requeuer component. It works with all Pub/Subs, but for now, it also requires PostgreSQL to be around (this can be extended in the future with more implementations). The PostgreSQL is used as an intermediate poison queue where the failed messages go until they are retried. It also supports delaying the messages, so you can avoid this infinite loop of errors. There are a few layers of how this works, and I’ll dive deeper into them below. Let’s first see the high-level API you need to get started. You can see the full example in the Watermill repository. First, create the SQL requeuer. You need to pass a publisher to the regular Pub/Sub so it knows where to route messages. delayedRequeuer, err := sql.NewPostgreSQLDelayedRequeuer(sql.DelayedRequeuerConfig{ DB: db, Publisher: redisPublisher, Logger: logger, }) if err != nil { panic(err) } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/delayed-requeue/main.go Full source In your router, add middleware provided by the requeuer. router := message.NewDefaultRouter(logger) router.AddMiddleware(delayedRequeuer.Middleware()...) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/delayed-requeue/main.go Full source Finally, run the requeuer: go func() { err = delayedRequeuer.Run(context.Background()) if err != nil { panic(err) } }() Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/delayed-requeue/main.go Full source That’s it if you’re happy with the default settings. The messages will be moved to the SQL poison queue if an error happens. After a delay, they will be moved back to the original topic. Note The PostgreSQLDelayedRequeuer is available in watermill-sql v1.4.0-rc.1: it’s still a release candidate. Read more below in the section on watermill-sql. Poison Queue CLI When working with poison queues, the main challenge is visibility into what’s happening. You need to know what messages are failing and why. Watermill 1.4 ships with a pq CLI tool to help you check this. go install github.com/ThreeDotsLabs/watermill/tools/pq@latest export DATABASE_URL=postgres://user:password@localhost:5432/db pq -backend postgres -topic requeue You can preview the poison queue, move messages back to the original topic, or delete them. Delayed Messages Delayed messages can be useful for more than requeuing. Many scenarios involve delays: “Ask the customer for feedback in two days”, “Send an invoice reminder in a week”, etc. It’s not a complex logic to implement, but if you already use events, you can leverage it and have delays working out of the box. Watermill’s new delay package allows you to add delay metadata to messages. The delay metadata does nothing by itself. You need to use a Pub/Sub implementation that supports it to make it work. For now, it’s supported by the PostgreSQL Pub/Sub with NewDelayedPostgreSQLPublisher and NewDelayedPostgreSQLSubscriber. There are two APIs you can use. If you work with raw messages, use delay.Message: msg := message.NewMessage(watermill.NewUUID(), []byte("hello")) delay.Message(msg, delay.For(time.Second * 10)) If you use the CQRS component, use delay.WithContext instead (since you can’t access the message directly). You can also use delay.Until instead of delay.For to specify time.Time instead of time.Duration. ctx = delay.WithContext(ctx, delay.Until(invoiceDueDate)) err := commandBus.Send(ctx, cmd) if err != nil { return err } See the full example for more details. PostgreSQL Queue The Delayed PostgreSQL Pub/Sub is based on the new queue schema adapter. It’s a simpler schema that doesn’t support consumer groups. In return, it allows you to specify a custom WHERE clause to filter messages by a custom condition, like metadata or payload fields. You can also enable delete on ack, so the table doesn’t grow with time: the messages are deleted after they are acknowledged. sub, err := sql.NewSubscriber( db, sql.SubscriberConfig{ SchemaAdapter: sql.PostgreSQLQueueSchema{ GenerateWhereClause: func(params sql.GenerateWhereClauseParams) (string, []any) { return "metadata->>'urgent' = true", []any{} }, }, OffsetsAdapter: sql.PostgreSQLQueueOffsetsAdapter{ DeleteOnAck: true, }, }, logger, ) Requeuer The Requeuer component is a wrapper on the Router, similar to the Forwarder, but simpler. It listens to messages on one topic and routes them to another, dynamically generated one. You can combine it with the Poison middleware that moves failed messages to a separate topic. You can use the Requeuer to move them back to the original topic based on the Poison metadata. This is exactly what the PostgreSQLDelayedRequeuer does. It combines the Requeuer, Poison middleware, DelayOnError middleware, and PostgreSQL queue adapter and hides all this behind a simple API. AWS SNS/SQS Pub/Sub v1.0 Using SNS or SQS makes a lot of sense when your services are hosted on AWS. Adding support for Pub/Subs managed by AWS was one of the most requested message brokers. In the current release, we have added support for both SNS and SQS. Thanks to that, you can choose the tool that fits your needs better. While adding support for SQS is not challenging, implementing a proper Pub/Sub on top of SNS is not trivial. SNS differs from other Pub/Subs: it’s not just a matter of subscribing to a topic and receiving messages. You need to create a queue, subscribe to the topic, create a valid access policy, and then receive messages from the queue. It also doesn’t help that AWS error messages are sometimes a bit cryptic. The good news is that we already made this work for you, and you can use it out of the box. newSubscriber := func(name subName) (message.Subscriber, error) { subscriberConfig := sns.SubscriberConfig{ AWSConfig: aws.Config{ Credentials: aws.AnonymousCredentials{}, }, OptFns: snsOpts, TopicResolver: topicResolver, GenerateSqsQueueName: func(ctx context.Context, snsTopic sns.TopicArn) (string, error) { topic, err := sns.ExtractTopicNameFromTopicArn(snsTopic) if err != nil { return "", err } return fmt.Sprintf("%v-%v", topic, subName), nil }, } sqsSubscriberConfig := sqs.SubscriberConfig{ AWSConfig: aws.Config{ Credentials: aws.AnonymousCredentials{}, }, OptFns: sqsOpts, } return sns.NewSubscriber(subscriberConfig, sqsSubscriberConfig, logger) } As always, you can find a fully working example (including a local emulator so that you can run it in Docker locally): SQS example SNS example Big thanks to @Mariscal6 for creating a big part of the implementation! watermill-amqp: v3 released We released a major version of the AMQP Pub/Sub. The breaking change was introduced in the PR #27 originally contributed by @tobiasjaster . The BuildTopology method of TopologyBuilder was missing the routing key. We decided to introduce a params struct to make it more flexible in the future. Upgrading You don’t need to do anything if you use the default TopologyBuilder. If you implement a custom TopologyBuilder, update the BuildTopology method. It now takes params amqp.BuildTopologyParams instead of queueName string, exchangeName string. - BuildTopology(channel *amqp.Channel, queueName string, exchangeName string, config Config, logger watermill.LoggerAdapter) error + BuildTopology(channel *amqp.Channel, params BuildTopologyParams, config Config, logger watermill.LoggerAdapter) error Inside the method, replace queueName with params.QueueName and exchangeName with params.ExchangeName. watermill-sql v3 and v4 We introduced the v3 major version of watermill-sql a while ago. This made the PostgreSQL implementation more bulletproof. Now, the v4 is around the corner. Similar to watermill-amqp, we decided to make SchemaAdapter and OffsetsAdapter interfaces more flexible. Instead of raw arguments, they now take a params struct and always return an error. It’s a release candidate because we want to include one more breaking change so we don’t bump the major version again soon. There’s an ongoing development in PR #29: streamline the database interfaces and add pgx adapter contributed by @julesjcraske . The aim is to allow watermill-sql to be used with pgx and ORMs not compatible with database/sql, a common request from the community. Stay tuned! Mapping Watermill’s log levels to slog Sometimes, the default INFO logs of Watermill may be too verbose for your application. To help you with that, we added a new feature to map Watermill’s log levels to slog levels. You can now map Watermill’s log levels to slog levels with NewSlogLoggerWithLevelMapping: logger := NewSlogLoggerWithLevelMapping( nil, map[slog.Level]slog.Level{ slog.LevelInfo: slog.LevelDebug, }, ) New docs website We refreshed Watermill’s documentation. It should be now easier to find your way around it. See the new version at watermill.io. Refreshed logo Watermill’s Gopher got a new look! 🎨 New Examples In case you missed it, there’s a new example featuring Server-Sent Events and htmx. You may also like the examples on distributed transactions where the outbox pattern is used with Watermill. There are also the mentioned above examples on delayed messages and delayed requeue Releases Dashboard You can now track the releases of all Watermill repositories in one place: releases.threedots.tech Terraform We set up a Terraform repository to manage all Pub/Sub repositories. If you want to take a look, it’s here: watermill-terraform. Celebrating the Launch Just like the last two times, we’re hosting a live event to celebrate the release of Watermill v1.4! We will share more details about the changes, show live demos, and answer your questions during Q&A. We start on October 30 at: 🇪🇺 6 PM CEST 🇺🇸 9 AM PT / 12 PM EST 🇮🇳 9:30 PM IST Join us on YouTube (Add the event to Google Calendar) We will also have news about our Go Event-Driven training. We have a tradition of running a quiz during the live events. The prize is a unique crotchet Gopher. 😎 Meet the Gophers at our live event! See you there!

Recently, I struggled for a couple of hours to understand why the API tests of one project were slow. In theory, we designed tests to run in a fully parallel way – the duration of tests should be close to the longest-running test. Unfortunately, the reality was different. Tests took 7x longer than the slowest test without using 100% available resources. In this article, I will show you a few techniques to help you understand and optimize your tests execution. Optimizing tests that use CPU efficiently is simple (in most cases just add more resources). We’ll focus on a scenario of optimizing single-threaded, CPU-heavy integration, component, API, and E2E tests. It’s hard to fix a problem that you can’t see It’s difficult to understand how tests run from the output generated by go test. You can see how long each test took, but you don’t know how long a test waited to run. You also can’t see how many tests ran in parallel. It becomes even harder when your project has thousands of tests. Surprisingly, I didn’t find any tool that helps visualize Go test execution. As a hobbyist frontend engineer, I decided to build my own over a weekend. vgt the missing tool for Visualizing Go Tests vgt can parse JSON Go test output to create a visualization. The quickest way to use it is by calling: go test -json ./... | go run github.com/roblaszczak/vgt@latest or by installing and running go install -u github.com/roblaszczak/vgt go test -json ./... | vgt In a perfect world, this is how ideal test execution of tests that are not CPU-bound should look: All non-CPU bound tests are executed in parallel. Output from vgt. Each bar represents the test execution time of a single test or subtest. Unfortunately, the tests I was recently debugging looked more like this: These tests can be optimized. Output from vgt. While the CPU was not fully used, tests ran one by one. It is also a good sign: we have a big room for improvement. Parallelizing Go tests By default, in Go all tests within a single package are run sequentially. This is not a problem when tests use all CPU cores efficiently and tests are split into multiple packages. But our CPU wastes cycles when a database query, API call, or sleep blocks tests – especially if we have a lot of tests in a single package. Tests without t.Parallel() flag. Output from vgt. To fix this problem, *testing.T provides the t.Parallel() method, which allows tests and sub-tests to run in parallel. Warning t.Parallel() is not a silver bullet and should be used with caution. Use t.Parallel() only for tests that have blocking operations like database queries, API calls, sleeps. It can also make sense for CPU-heavy tests using only a single-core. For fast unit tests the overhead of using t.Parallel() will be higher than running them sequentially. In other words, using t.Parallel() for lightweight unit tests will likely make them slower. Parallelism limit Even if you use t.Parallel(), it doesn’t mean that all tests will run in parallel. To simulate this scenario, I wrote an example test that will simulate 100 tests doing API calls. func TestApi_parallel_subtests(t *testing.T) { t.Parallel() for i := 0; i < 100; i++ { t.Run(fmt.Sprintf("subtest_%d", i), func(t *testing.T) { t.Parallel() simulateSlowCall(1 * time.Second) }) } } func simulateSlowCall(sleepTime time.Duration) { time.Sleep(sleepTime + (time.Duration(rand.Intn(1000)) * time.Millisecond)) } As long as the target server is not overloaded and tests are appropriately designed, we should be able to run all tests in parallel. In this case, running all tests should take at most 2 seconds. But it took over 16 seconds instead. Despite using t.Parallel(), the execution graph shows many gray bars representing pauses. Tests marked as PAUSED are limited due to the parallelism limit. These tests can be optimized. Output from vgt. First, let’s understand how tests with t.Parallel() run. If you’re curious, you can check the source code of the testing package. What’s important, test parallelism is set to runtime.GOMAXPROCS(0) by default, which returns the number of cores reported by the OS. parallel = flag.Int("test.parallel", runtime.GOMAXPROCS(0), "run at most `n` tests in parallel") Full source: cs.opensource.google/go/go/+/refs/tags/go1.23.1:src/testing/testing.go;l=442 Full source On my Macbook, runtime.GOMAXPROCS(0) returns 10 (as I have a 10-core CPU). In other words, it limits tests run in parallel to 10. Limiting tests to the number of our cores makes sense when they are CPU-bound. More parallelism will force our OS to do more expensive context switching. But when we are calling a database, API, or any blocking I/O, it makes tests longer without fully using our resources. The situation can be even worse when API tests run in CI against an environment deployed in a separate VM or cloud. Often, CI runners for API tests may have 1-2 CPUs. With 1 vCPU, API tests will run one by one. We can simulate this by setting the env GOMAXPROCS=1. Tests with 1 vCPU – this is how it may look in CI. Output from vgt. Parallelism is effectively set to 1, so we see a lot of gray bars representing waiting time. To fix this problem, we can use the -parallel (or -test.parallel – they have the same effect) flag. Go documentation says: -parallel n Allow parallel execution of test functions that call t.Parallel, and fuzz targets that call t.Parallel when running the seed corpus. The value of this flag is the maximum number of tests to run simultaneously. While fuzzing, the value of this flag is the maximum number of subprocesses that may call the fuzz function simultaneously, regardless of whether T.Parallel is called. By default, -parallel is set to the value of GOMAXPROCS. Setting -parallel to values higher than GOMAXPROCS may cause degraded performance due to CPU contention, especially when fuzzing. Note that -parallel only applies within a single test binary. The 'go test' command may run tests for different packages in parallel as well, according to the setting of the -p flag (see 'go help build'). Tip Don’t change GOMACPROCS to a value higher than the available cores to force more parallelism. It will have more effects than just on tests — it will spawn more Go threads than cores. It will lead to more expensive context switching and may slow down CPU-bound tests. Let’s see how the same test will behave with the extra -parallel 100 flag: go test ./… -json -parallel=100 | vgt We achieved our goal - all tests run in parallel. Our tests were not CPU-bound, so the overall execution time can be as long as the slowest executed test. Tip If you are not changing the test code and want to test their performance, Go may cache them. To avoid caching, run them with the -count=1 flag, for example go test ./... -json -count=1 -parallel=100 | vgt. Tests in multiple packages Using the -parallel flag is not the only thing we can do to speed up tests. It’s not uncommon to store tests in multiple packages. By default, Go limits how many packages can run simultaneously to the number of cores. Let’s look at the example project structure: $ ls ./tests/* ./tests/package_1: api_test.go ./tests/package_2: api_test.go ./tests/package_3: api_test.go Usually, we may have many more packages with tests for more complex projects. For readability, let’s simulate how tests will run on one CPU (for example, in a CI runner with 1 core): GOMAXPROCS=1 go test ./tests/... -parallel 100 -json | vgt Running tests on a runner with 1 core with 3 packages. . Output from vgt. You can see that every package runs separately. We can fix this with the -p flag. It may not be a problem if your tests run on machines with multiple cores and you don’t have many packages with long-running tests. But in our scenario, with CI and one core, we need to specify the -p flag. It will allow up to 16 packages to run in parallel. GOMAXPROCS=1 go test ./tests/... -parallel 128 -p 16 -json | vgt Running tests on a runner with 1 core with 3 packages. Output from vgt. Now, the entire execution time is very close to the longest test duration. Tip It’s hard to give -parallel and -p values that will work for all projects. It depends a lot on your types of tests and how they are structured. The default value will work fine for many lightweight unit tests or CPU-bound tests that efficiently use multiple cores. The best way to find the correct -parallel flag is to experiment with different values. vgt may be helpful in understanding how different values affect test execution. Parallelism with sub-tests and test tables Using test tables for tests in Go is very useful when you need to test many input parameters for a function. On the other hand, they have a couple of dangers: creating test tables may sometimes be more complex than just copying the test body multiple times. Using test tables can also affect the performance of our tests a lot if we forget to add t.Parallel(). This is especially visible for test tables with a lot of slow test cases. Even one use of test table can make our tests considerably slower. func TestApi_with_test_table(t *testing.T) { t.Parallel() testCases := []struct { Name string API string }{ {Name: "1", API: "/api/1"}, {Name: "2", API: "/api/2"}, {Name: "3", API: "/api/3"}, {Name: "4", API: "/api/4"}, {Name: "5", API: "/api/5"}, {Name: "6", API: "/api/6"}, {Name: "7", API: "/api/7"}, {Name: "8", API: "/api/8"}, {Name: "9", API: "/api/9"}, {Name: "10", API: "/api/9"}, {Name: "11", API: "/api/1"}, {Name: "12", API: "/api/2"}, {Name: "13", API: "/api/3"}, {Name: "14", API: "/api/4"}, {Name: "15", API: "/api/5"}, {Name: "16", API: "/api/6"}, {Name: "17", API: "/api/7"}, {Name: "18", API: "/api/8"}, {Name: "19", API: "/api/9"}, } for i := range testCases { t.Run(tc.Name, func(t *testing.T) { t.Parallel() simulateSlowCall(1 * time.Second) }) } } Using a test table without t.Parallel(). Output from vgt. The solution is simple: add t.Parallel() to the test table. But it’s easy to forget about it. We can be careful when using test tables. But being careful doesn’t always work in the real world when you’re in a hurry. We need an automated way to ensure that t.Parallel() is not missed. Linting if t.Parallel() is used In most projects, we use golangci-lint. It allows you to configure multiple linters and set them up for the entire project. We can configure which linter should be enabled based on the file name. An example configuration will ensure all tests in files ending with _api_test.go or _integ_test.go are using t.Parallel(). Unfortunately, as a downside, it requires keeping a convention in naming test files. Tip Alternatively, you can group your tests by type in multiple packages. So, one package can contain API tests, and another can contain integration tests. Note As mentioned earlier, it’s not only pointless, but even slower to use t.Parallel() for all kinds of tests. Avoid requiring t.Parallel() for all types of tests. This is an example .golangci.yml configuration: run: timeout: 5m linters: enable: # ... - paralleltest issues: exclude-rules: # ... - path-except: _api_test\.go|_integ_test\.go linters: - paralleltest With this config, we can run the linter: $ golangci-lint run package_1/some_api_test.go:9:1: Function TestApi_with_test_table missing the call to method parallel (paralleltest) func TestApi_with_test_table(t *testing.T) { ^ You can find a reference for configuration in golangci-lint docs. Tip Not all tests can always run in parallel. In this case, you can disable the linter for this specific test with //nolint:paralleltest. //nolint:paralleltest func SomeTest(t *testing.T) { //nolint:paralleltest t.Run("some_sub_test", func(t *testing.T) { }) } Parallelism quirks: does grouping tests with t.Run() affect performance? I’ve often seen in many projects a convention of grouping tests with t.Run(). Have you ever wondered if it affects performance in any way? To check this hypothesis, I wrote 50 tests like this: func TestApi1(t *testing.T) { t.Parallel() t.Run("1", func(t *testing.T) { t.Run("1", func(t *testing.T) { t.Run("1", func(t *testing.T) { simulateSlowCall(1 * time.Second) }) }) }) t.Run("2", func(t *testing.T) { t.Run("2", func(t *testing.T) { t.Run("2", func(t *testing.T) { simulateSlowCall(1 * time.Second) }) }) }) } func simulateSlowCall(sleepTime time.Duration) { // for more reliable results I'm using constant time here time.Sleep(sleepTime) } To compare, I also wrote 50 tests without using t.Run() for subtests: func TestApi1(t *testing.T) { t.Parallel() simulateSlowCall(1 * time.Second) simulateSlowCall(1 * time.Second) } func simulateSlowCall(sleepTime time.Duration) { // for more reliable results I'm using constant time here time.Sleep(sleepTime) } Does it affect test performance by affecting parallelism in any way? Let’s see what vgt will show us. Tests without using r.Run() for subtests. Output from vgt. Tests using r.Run() for subtests. Output from vgt. Despite the chart looking a bit uglier, execution times are the same for grouped and non-grouped tests. In other words, grouping tests with t.Run() does not affect performance. Summary Having fast and reliable tests for efficient development is crucial. But over the years, I’ve learned that reality is not always so simple. There is always more work to do, and tests are not something that our product’s users or boss directly see. On the other hand, a small investment in tests can save a lot of time in the future. Reliable and fast tests are one of the best investments in your project’s return on investment (ROI). Knowing some tactics to convince your team leader or manager to find time to improve tests is useful. It’s helpful to think in terms that your manager or boss uses. You should consider the benefits they care about. Reducing delivery time - you can even calculate how many minutes per month the entire team wastes by waiting for tests or retrying them. It may be useful to multiply this by the average developer’s salary. It’s also helpful to track bugs shipped to production because tests were so flaky that nobody noticed them. There are more ways to make your tests more useful. I’ve also observed that people often struggle with naming different types of tests – we will give you some hints about that, too. If you are interested in more articles about testing, check out: 4 practical principles of high-quality database integration tests in Go Microservices test architecture Is vgt useful for you? Don’t forget to give it a star on GitHub and share it with your friends or social media!

In the previous post, I looked into running transactions in a layered architecture. Now, let’s consider transactions that need to span more than one service. If you work with microservices, a time may come when you need a transaction running across them. Especially if the way they are split was an afterthought (the unfortunate but likely scenario). Service A calls service B, which calls service C, and if something goes wrong at the end, the system becomes inconsistent. It would be helpful to have a way to roll back the changes in all the services. Now, you’re looking at distributed transactions or the saga pattern. It’s a solved problem. Sometimes, there are good reasons to use them. But more often, it’s overkill. If you go down this path, your architecture quickly becomes much more complex than you’d like. (I learned this the hard way.) If you need things to be consistent, why split them in the first place? The whole idea of using microservices is to keep independent concepts separate. Let’s face it: your microservice boundaries are likely wrong if you consider using a distributed transaction. This is super common and not easy to solve. But it can get much worse if you apply the wrong pattern to the mess you already have. Before you try it, I want to show you an alternative that might be simpler. It’s not a silver bullet, but consider it before committing to distributed transactions. ❌ Anti-pattern: Distributed Transactions Stay away from transactions that span more than one service, except when there's no other way. They are hard to test, debug, and maintain. Your system becomes tightly coupled and more challenging to extend. The Example The snippets below continue the example from the previous post. Here’s a quick refresher: we work with an e-commerce web app where users get virtual points. They can use these points as a discount for their next order. The challenge is keeping the user’s points and the applied discount consistent. Let’s assume we work with two services: users for authentication and orders for placing orders and tracking users’ discounts. Initially, the split seemed like a good idea because they are separate areas. Later, we realized we couldn’t update users and their discounts within one transaction. The orders service exposes an HTTP endpoint for adding a discount. In the users service, the handler for using points as a discount looks like this: func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { err := h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) { err := user.UsePoints(cmd.Points) if err != nil { return false, err } return true, nil }) if err != nil { return fmt.Errorf("could not update user: %w", err) } err = h.ordersService.AddDiscount(ctx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not add discount: %w", err) } return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/01-distributed-monolith/users-svc/app.go Full source We first take the user’s points. After committing the transaction, we call the orders service to add a discount. It works well in a happy path scenario, but the HTTP call works outside the transaction. If anything goes wrong, the points are gone, but we add no discount. The best we can do is log the error and fire an alert so someone on-call can manually add the discount for the annoyed customer. Welcome to the distributed monolith. ❌ Anti-pattern: The Distributed Monolith Be extra careful when designing the boundaries of microservices. What seems like a minor decision in the beginning can have terrible consequences in the future. If in doubt, stick to a single service and decouple your code using modules — the modular monolith approach. One way out of this mess is to accept past mistakes and merge the microservices into one service. Depending on the approach used and the size of the services, it may not be a quick task. And if your product backlog is full and people who designed the microservices are still around, it’s unlikely to happen. If you see the problem as “we miss consistency between services”, running a distributed transaction seems like a good solution. But the real problem is “the boundaries are wrong”. It’s easy to make the project much more complicated because of this misinterpretation. One alternative is to embrace eventual consistency. Eventual Consistency The idea is that the data (the user’s points and applied discount in our example) stays consistent but no longer within a single transaction. In practice, there’s a slight delay — usually milliseconds — during which the system is inconsistent but then catches up. The system may become out of sync for longer if something goes wrong. But the important part is that it eventually becomes consistent again. We don’t accept inconsistency — we’re just okay with waiting for it to happen. Think about sending a bank transfer. The money disappears from your account immediately but isn’t visible on the other account right away. You know it will eventually happen within a few hours or days, so it’s not an issue. Whether it’s acceptable in a given scenario is often a business decision. Since business stakeholders generally don’t consider this, engineers must explain it and suggest solutions. Having a transaction boundary sounds reasonable in the “using points as a discount” scenario. But since we already have to work with a distributed monolith, choosing eventual consistency might be the next best thing instead of implementing a distributed transaction. Note I could have picked another example that would be more fit for eventual consistency. In some cases, data doesn't need to be consistent at all. But this scenario is closer to what you can see in your daily work. Using a distributed transaction here may be tempting because it's not obvious if the data can be eventually consistent. Events Eventual consistency usually means working with events. Going this path, we no longer call the orders service directly. We take the user’s points and publish an event, like PointsUsedForDiscount, to record that it happened. The event is stored on a message queue (AKA a Pub/Sub), a reliable, highly-available infrastructure for messages. The orders service receives this event and reacts by applying a discount for the next order. We can then update the discount value on the website (for example, with Server-Sent Events). Most of the time, it takes milliseconds, and the customers can’t tell a difference. It’s sometimes better to retry the operation internally instead of showing a scary error message to the user. After all, the issue might be temporary, and retrying is often a good enough fix. However, you need to consider it separately for each scenario. It’s often crucial that the user knows an error has occurred. If the orders service goes down, it won’t interrupt the process. The event will be delivered again and again until it’s processed successfully. Once the service comes up, the discount will be applied as expected. It doesn’t fix all the issues. For example, the customer won’t see the discount if the service is down for a few hours. But it gives us a solid way of retrying the operation if it’s a short issue. In many scenarios, retrying for a few hours is also acceptable (generating reports, issuing invoices, etc.). In most cases, we don’t need to manually investigate what data needs to be backfilled, and no one needs to wake up in the middle of the night. The system auto-heals with time. ✅ Tactic: Embrace Eventual Consistency Not all operations need to be strongly consistent, even if it initially seems like it. Keep eventual consistency in mind when designing distributed systems. It's often simpler than going the distributed transactions way. Note Events or messages? While sometimes used as synonyms, there's a difference between events and messages. A message is a transport unit you publish to the Pub/Sub or receive from it. It's similar to an HTTP request with a method, path, headers, and body. An event is a specific message. It represents a fact about something that has already happened in the system. The event is represented by the payload held by the message (encoded as JSON or another format). It's what you would include in the HTTP request's body. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Implementation To start working with messages, you must pick a broker (the infrastructure part), just like when choosing a database. You publish messages to the broker, which asynchronously pushes them to all subscribers. In this post, I’m going to use Redis. Most popular brokers (Pub/Subs) have a Go library (SDK) that lets you interact with it. But they usually offer a low-level API. As much as I like Go’s net/http, I still prefer using Echo or Chi for their high-level API. Similarly, I will use Watermill to work with messages. Watermill is a Go library we maintain. I won’t explain how it works here in detail. See the Getting Started page for an overview. It supports many Pub/Sub backends, like Kafka, AMQP, NATS, or Redis. Note All the ideas below are universal and can be implemented with any other messaging library. Watermill is not a framework, so it's easy to add or remove from a project. It's one of the core design ideas. You can see that I won't include Watermill-specific code in the logic layer. Feel free to use anything else if it fits your needs better. We’ll use the pair of EventBus and EventProcessor components from Watermill’s cqrs package. First, let’s replace the HTTP call with publishing an event. The event bus lets us publish messages with a JSON payload. The setup should be easy to grasp even if you don’t know Watermill yet. client := redis.NewClient(&redis.Options{ Addr: redisAddr, }) logger := watermill.NewStdLogger(false, false) publisher, err := redisstream.NewPublisher( redisstream.PublisherConfig{ Client: client, }, logger, ) if err != nil { return nil, err } eventBus, err := cqrs.NewEventBusWithConfig(publisher, cqrs.EventBusConfig{ GeneratePublishTopic: func(params cqrs.GenerateEventPublishTopicParams) (string, error) { return params.EventName, nil }, Marshaler: cqrs.JSONMarshaler{}, Logger: logger, }) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/02-eventual-consistency/users-svc/events.go Full source We create a Redis publisher and an EventBus. It’s going to use the event’s name as a topic. (A topic is what you subscribe to to receive messages.) It’s going to marshal events into JSON. In the command handler, we replace the synchronous call with publishing an event. type EventPublisher interface { Publish(ctx context.Context, event any) error } type PointsUsedForDiscount struct { UserID int `json:"user_id"` Points int `json:"points"` } func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { err := h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) { err := user.UsePoints(cmd.Points) if err != nil { return false, err } return true, nil }) if err != nil { return fmt.Errorf("could not update user: %w", err) } event := PointsUsedForDiscount{ UserID: cmd.UserID, Points: cmd.Points, } err = h.eventPublisher.Publish(ctx, event) if err != nil { return fmt.Errorf("could not publish event: %w", err) } return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/02-eventual-consistency/users-svc/app.go Full source In the orders service, we replace the HTTP handler with an event handler. Let’s set up an EventProcessor for subscribing, just like we set up the event bus for publishing. client := redis.NewClient(&redis.Options{ Addr: redisAddr, }) logger := watermill.NewStdLogger(false, false) router := message.NewDefaultRouter(logger) eventProcessor, err := cqrs.NewEventProcessorWithConfig(router, cqrs.EventProcessorConfig{ GenerateSubscribeTopic: func(params cqrs.EventProcessorGenerateSubscribeTopicParams) (string, error) { return params.EventName, nil }, SubscriberConstructor: func(params cqrs.EventProcessorSubscriberConstructorParams) (message.Subscriber, error) { return redisstream.NewSubscriber( redisstream.SubscriberConfig{ Client: client, ConsumerGroup: "orders-svc", }, logger, ) }, Marshaler: cqrs.JSONMarshaler{}, Logger: logger, }) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/02-eventual-consistency/orders-svc/events.go Full source The setup is almost identical to the publishing side. The main difference is that we provide a constructor instead of a single subscriber. I won’t dive into the reasons, as they are irrelevant to what we want to do. To receive messages correctly, we must use the same marshaler and topic configuration as in the event bus config. Next, we need an event handler. It simply maps the event payload to the command and executes it, just like the HTTP handler did. type OnPointsUsedForDiscountHandler struct { addDiscountHandler AddDiscountHandler } func (h OnPointsUsedForDiscountHandler) Handle(ctx context.Context, event *PointsUsedForDiscount) error { cmd := AddDiscount{ UserID: event.UserID, Discount: event.Points, } return h.addDiscountHandler.Handle(ctx, cmd) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/02-eventual-consistency/orders-svc/events.go Full source We add it to the event processor. err = eventProcessor.AddHandlers( cqrs.NewEventHandler( "OnPointsUsedForDiscountHandler", onPointsUsedForDiscountHandler.Handle, ), ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/02-eventual-consistency/orders-svc/events.go Full source The behavior is exactly the same as that of the HTTP endpoint. The message is just another transport, and the handler is another entry point to the application. The main difference is that it’s asynchronous, so no client waits for it to complete. That’s it! With a few changes, we replaced the synchronous HTTP call with a message handler. Of course, there’s the hard part of running a production-grade Pub/Sub. I used Redis, but you can pick what you feel comfortable with or what your cloud provider offers. You can even start with an SQL database if you don’t want to set up new infrastructure. Are you experienced engineer who wants to learn Go basics? You don't become an engineer by watching videos. Learn Go hands-on by building real projects. The Outbox Pattern There’s still one scenario when things can go wrong. Since publishing the message works over the network, it can fail, just like the HTTP call to another service. Pub/Subs are usually highly available, but issues still happen (never assume the network is reliable!). We could lose the messages after the transaction is committed. It’s one of the most common mistakes we’ve seen: ignoring this scenario with the hope it will never happen. It’s unlikely, sure, but you don’t want to take this risk. Imagine figuring out what events you lost and manually correcting the system’s state. We need to store changes in the database and publish the event within the same transaction. Is this possible? Yes, we can use the outbox pattern. The idea is to save the event in the same database as the regular data within the same transaction. Then, we asynchronously publish it to the Pub/Sub. This way, the event and the stored data will always be consistent, and we will not rely on the Pub/Sub to be around. The events can be simply stored in a dedicated SQL table. The main challenge is waiting for new events and moving them to the Pub/Sub. Depending on the database, you can use some custom software, although configuring it is often quite complex. You can implement it from scratch, but it’s not trivial. In this example, I’ll use Watermill’s Forwarder component to achieve this. Its behavior is straightforward, thanks to the heavy lifting done by the Pub/Sub implementation. It listens to messages from the given Pub/Sub and publishes them to another Pub/Sub. Watermill supports Postgres as one of the Pub/Sub implementations, so adding it to the existing setup is easy. (Yes, you can simply use an SQL database as your messaging infrastructure.) We face another design dilemma. How can the publisher and repository work together without mixing the logic and the database code? We’ll extend the UpdateFn function so it decides what events should be published. (For more on how this pattern works, see the post on database transactions.) func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { return h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, []any, error) { err := user.UsePoints(cmd.Points) if err != nil { return false, nil, err } event := PointsUsedForDiscount{ UserID: cmd.UserID, Points: cmd.Points, } return true, []any{event}, nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/03-outbox/users-svc/app.go Full source The repository publishes the returned events. updated, events, err := updateFn(user) if err != nil { return err } if !updated { return nil } _, err = tx.ExecContext(ctx, "UPDATE users SET email = $1, points = $2 WHERE id = $3", user.Email(), user.Points(), user.ID()) if err != nil { return err } eventBus, err := NewWatermillEventBus(tx) if err != nil { return err } for _, event := range events { err = eventBus.Publish(ctx, event) if err != nil { return err } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/03-outbox/users-svc/repository.go Full source Note how a new event bus is created using the tx variable. This is how we ensure publishing happens within the same transaction. We must amend how the event bus is created in the users service. We will no longer publish directly to Redis. We need to change the original publisher from Redis to Postgres. publisher, err = watermillSQL.NewPublisher( db, watermillSQL.PublisherConfig{ SchemaAdapter: watermillSQL.DefaultPostgreSQLSchema{}, }, logger, ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/03-outbox/users-svc/events.go Full source Next, we wrap this publisher with the forwarder’s Publisher. It introduces an envelope around the message so the forwarder knows how to handle it. Its config is minimal. publisher = forwarder.NewPublisher( publisher, forwarder.PublisherConfig{ ForwarderTopic: forwarderTopic, }, ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/03-outbox/users-svc/events.go Full source The last part is running the forwarder. While it could be a standalone service, keeping it running as another goroutine within the users service is also fine. We need an SQL subscriber to receive the stored events from the database, and a Redis publisher to publish the messages where the orders service expects them. sqlSubscriber, err := watermillSQL.NewSubscriber( db, watermillSQL.SubscriberConfig{ SchemaAdapter: watermillSQL.DefaultPostgreSQLSchema{}, OffsetsAdapter: watermillSQL.DefaultPostgreSQLOffsetsAdapter{}, InitializeSchema: true, }, logger, ) if err != nil { return nil, err } client := redis.NewClient(&redis.Options{ Addr: redisAddr, }) redisPublisher, err := redisstream.NewPublisher( redisstream.PublisherConfig{ Client: client, }, logger, ) if err != nil { return nil, err } fwd, err := forwarder.NewForwarder( sqlSubscriber, redisPublisher, logger, forwarder.Config{ ForwarderTopic: forwarderTopic, }, ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/03-outbox/users-svc/events.go Full source Note that the forwarder topic needs to match between the forwarder publisher and the forwarder itself. In this case, it’s the same forwarderTopic constant (can be any string). Finally, we run the forwarder. go func() { err := forwarder.Run(context.Background()) if err != nil { panic(err) } }() No changes are necessary on the orders service side. As far as it’s concerned, it receives events from Redis, as it used to. ✅ Tactic: The Outbox Pattern Use the Outbox pattern to publish events within a database transaction. Don’t leave it to chance whether all events will be successfully published. Simplifying or Complicating? There are many moving parts here. Event-driven patterns are not trivial, and you need to understand what you’re doing to some degree. If you never used these concepts before, it can seem complex. I used to have these concerns as well. The system seems simpler with just the synchronous APIs and a database. Why complicate it with adding all this background processing? As counterintuitive as it is, you can simplify your distributed system using these ideas. Because you work with facts that happened (events), you don’t need to consider rollbacks, which terribly complicate your flow. A rollback can interfere with other flows, so you need distributed locks to protect the state. Rollbacks can also fail. What do you do when this happens? As the snippets show, little code is needed to start working with events, and the handlers are also straightforward. This is where Watermill helps by removing the low-level boilerplate. You can focus on setting up the handlers and publishing messages with an API that’s easier to use than HTTP endpoints. Once you grasp the basics, your mental model will start changing, and eventually, it will click. You’ll get the intuition on how to design event-driven systems. Events & Coupling Make no mistake, events are still a form of a contract. The fact you use them doesn’t magically make your services decoupled. Incorrectly designed events are another trap you can fall into. What I like about the pattern is that it limits what the services know about each other. But it’s easy to fail at this. The example above shows that the event is designed quite poorly. The UsePointsAsDiscount name makes the users service aware of what happens in the orders service after it receives it. It’s too late to escape this now, though. The service boundaries are just wrong from the beginning. Well-designed events state a fact that happened in this service’s domain. The service publishing the event should not know or care what happens when another service consumes it. Testing Testing event-driven services is not much different, but here are some tips. Run your Pub/Sub locally in a Docker container and test against it. Cloud-based Pub/Subs usually offer an emulator (but often with no feature parity, so watch out). Testing the event handler code makes little sense. Event handlers should just execute your application logic with the data from the event. Instead, use component tests to test the behavior of the service using the public API. A test scenario can be something like: Once this event is published, an entity read over HTTP should change. Once this HTTP endpoint is called, an event should be published. Once this event is published, another event should be published. If you use testify, the Eventually and EventuallyWithT functions are helpful. (The second one allows using asserts inside). They run the assert code periodically until it’s successful or until it times out. Use these instead of plain sleeps, as they slow down your tests. assert.EventuallyWithT(t, func(t *assert.CollectT) { row := discountDB.QueryRowContext(context.Background(), "SELECT next_order_discount FROM user_discounts WHERE user_id = $1", userID) var discount int err := row.Scan(&discount) require.NoError(t, err) assert.Equal(t, expectedDiscount, discount) }, 2*time.Second, 100*time.Millisecond) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/05-distributed-transactions/tests/all_test.go Full source When reading events, watch out for others that may interfere. With many tests running in parallel, it’s common to have many events of the same type published. You need to filter them based on the ID or some metadata. Using UUIDs for this is best so your tests are completely independent. Finally, run your tests in parallel to speed them up. With lots of time spent on I/O and waiting, this quickly adds up. There are some quirks to running Go tests in parallel, though. Robert is currently working on an article on how to do it right — join our newsletter, and we’ll let you know once it’s ready. Monitoring Observability is a long topic, and there’s much to be said about metrics and tracing for event-driven systems. This is beyond the scope of this post, but I want to mention monitoring the queue of waiting messages. If a message fails to be processed (i.e., the handler returns an error for whatever reason), it will be delivered again. Depending on the Pub/Sub you picked and its configuration, it can block other messages from being delivered. This is your key metric to monitor. Having unprocessed messages for a long time usually means something went wrong, and it’s not a temporary issue. You need manual intervention, or your system will be out of sync. More Examples This scratches the surface of switching to eventual consistency, but it should be enough to get you started. To learn about different use cases, check Watermill examples. There are some basic ones, but also more advanced and interesting examples. As with most programming topics, reading about something is rarely enough to fully get it. For me, building something on my own is what makes me fully understand it. But it’s often hard to come up with an idea for a project to build. We wanted to recreate this way of learning, so we designed a hands-on event-driven training. Consider joining us if you like learning like this.

As I join a new company, I often feel like an impostor. After all the interviews, they really seem to know what they’re doing. I’m humbled and ready to learn from the best. On one such occasion, a few days in, I dealt with a production outage and asked the most senior engineer for help. They came to the rescue and casually flipped a value in the database with a manual update. 🤯 The root cause was that a set of SQL updates were not done within a transaction. Suddenly, I regretted not asking for higher compensation right away. Onboarding is fun. What I learned this way is that even if something seems like a fundamental concept (e.g., SQL transactions), it may often be overlooked. SQL seems like something we all know well, and there are few surprises. (It’s 50 years old!) Perhaps it’s a good time to reconsider, as we’re past the “NoSQL is cool” hype phase and back to “just use Postgres”, or even “SQLite is good enough”. I want to focus here on how to keep transactions in the code rather than on their technical complexity. Once you need to organize a bigger project, perhaps by using layers, splitting the logic from the database code is not always that obvious. It’s easy to end up with a mess leading to obscure bugs. The key idea of layers is to keep the critical parts of code (the logic) out of the implementation details (like the SQL queries). One way to achieve this separation is the Repository pattern. The hard part is handling transactions in this setup. When using transactions, you usually fetch data from the database, run some logic, update the data, and commit the transaction. How do we model this in the code so we don’t end up with a mess across the layers? I often see people asking about this, and I know the pain from my experience. We had to figure this out, and the ideas evolved over time. Here’s a list of various approaches you can consider and everything I’ve learned along the way. SQL Transactions 101 Let’s quickly reiterate why we need transactions at all. Consider an e-commerce web app where users receive virtual points for purchasing products. The points can later be spent on discounts. The user decides how many points they want to spend on a discount. We have to check if the user has enough points, then decrease them and apply a proper discount for the next order. Let’s say a user wants to use 100 points. It could look roughly like this: SELECT points FROM users where id = 19; -- in the code: check if points >= 100 UPDATE users SET points = points - 100 WHERE id = 19; UPDATE user_discounts SET next_order_discount = next_order_discount + 100 WHERE user_id = 19; We should consider what can go wrong here, even if it’s unlikely. Our application can suddenly exit after the first UPDATE statement. Or, a network outage could prevent the second UPDATE from being executed. In this case, we would decrease the user’s points without giving them a discount. Or a different scenario: If the user sends two (or more) requests very quickly, they could be processed concurrently. If the requests execute the SELECT query before any updates, the check for the points balance won’t work correctly, and the user could end up with -100 points and a 200 discount value. ❌ Anti-pattern: Skipping transactions Use transactions for sets of queries that rely on each other. Even if unlikely, things can go wrong mid-request. You’ll be left with inconsistencies in the system that need manual effort to be understood and fixed. If something can theoretically happen, it will likely happen in production at some point. Don’t accept solutions that almost always work. The 1% chance can turn into a nightmare investigation scenario in which you can’t understand what happened in your application and how to fix it. We can easily improve this by introducing a transaction. BEGIN; SELECT points FROM users where id = 19 FOR UPDATE; -- in the code: check if points >= 100 UPDATE users SET points = points - 100 WHERE id = 19; UPDATE user_discounts SET next_order_discount = next_order_discount + 100 WHERE user_id = 19; COMMIT; Now, either all of the updates get saved or none. Besides the BEGIN and COMMIT statements that manage the transaction, note the FOR UPDATE clause added to the SELECT. It locks the row so that further SELECTS have to wait until this transaction is done. This lets us correctly process parallel requests. Warning FOR UPDATE works well for simple uses cases, but it may cause performance issues. It locks the entire row, so if many concurrent updates are competing for it, they will all have to wait. On a bigger scale, this can lead to a bottleneck. An alternative is to use another isolation level, like REPEATABLE READ. This is out of the scope of this post but you can check out this article for details. Whatever you choose, consider running stress tests to see how your code behaves under load. It will help you avoid surprises in production when a traffic spike hits. Layers Splitting code into layers is pretty common, no matter how many you use or what you call the approach (clean, hexagonal, ports & adapters, onion, etc.). It makes sense for many reasons, like separation of concerns, enabling parallel work, and easier testing. We wrote about these concepts in our other articles. For the rest of this post, I assume you keep your SQL code in a place (struct, file, package, whatever) separate from your application logic and inject one into the other. In the example snippets, I keep the “layers” in different files but in the same package for simplicity. http.go — the HTTP handler (we don’t care about this one much). app.go — the application logic. repository.go — the repository (the database storage). In the application logic, I define the UsePointsAsDiscount command and a handler for it (see Robert’s CQRS article for more details on this). You may prefer using something slightly different, like a service struct with methods or use cases. That’s all fine. The key is that this part of the code knows nothing about the database used. The repository is injected inside the command handler using an interface defined close to the handler’s definition. Here’s how the code can look like with no transactions. type UsePointsAsDiscount struct { UserID int Points int } type UsePointsAsDiscountHandler struct { userRepository UserRepository discountRepository DiscountRepository } type UserRepository interface { GetPoints(ctx context.Context, userID int) (int, error) TakePoints(ctx context.Context, userID int, points int) error } type DiscountRepository interface { AddDiscount(ctx context.Context, userID int, discount int) error } func NewUsePointsAsDiscountHandler( userRepository UserRepository, discountRepository DiscountRepository, ) UsePointsAsDiscountHandler { return UsePointsAsDiscountHandler{ userRepository: userRepository, discountRepository: discountRepository, } } func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { if cmd.Points <= 0 { return errors.New("points must be greater than 0") } currentPoints, err := h.userRepository.GetPoints(ctx, cmd.UserID) if err != nil { return fmt.Errorf("could not get points: %w", err) } if currentPoints < cmd.Points { return errors.New("not enough points") } err = h.userRepository.TakePoints(ctx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not take points: %w", err) } err = h.discountRepository.AddDiscount(ctx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not add discount: %w", err) } return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/01-no-tx/app.go Full source There are two repositories: one for the user and one for the discounts. The handler’s logic is rather easy to grasp: we validate the command (points must be positive), check if the user has enough points, and subtract them from their account while adding a discount. The repositories are trivial: type PostgresDiscountRepository struct { db *sql.DB } func NewPostgresDiscountRepository(db *sql.DB) *PostgresDiscountRepository { return &PostgresDiscountRepository{ db: db, } } func (r *PostgresDiscountRepository) AddDiscount(ctx context.Context, userID int, discount int) error { _, err := r.db.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = next_order_discount + $1 WHERE user_id = $2", discount, userID) return err } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/01-no-tx/repository.go Full source Note In your project, you may call the repository a storage or something else. It’s a similar concept, although the Repository pattern is not only about database access. See Robert’s post for more details. We’d like all of this to run within a single transaction, but we have a bunch of logic mixed with two repositories. Let’s see how to approach this. Note All examples are available in the go-web-app-antipatterns repository. There’s a Docker Compose definition that lets you run all of them locally. They use Postgres as the database. Transactions in the logic layer (avoid if you can) The first idea for dealing with transactions across repositories is passing a transaction object around. For running code in a transaction, we can use a helper function like the runInTx below. func runInTx(db *sql.DB, fn func(tx *sql.Tx) error) error { tx, err := db.Begin() if err != nil { return err } err = fn(tx) if err == nil { return tx.Commit() } rollbackErr := tx.Rollback() if rollbackErr != nil { return errors.Join(err, rollbackErr) } return err } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/02-tx-in-logic/app.go Full source It takes a function (you’ll see many anonymous functions in this post) that receives the transaction object. Whatever happens within the fn function doesn’t know or care how the transaction is started, committed, or rolled back. Here’s how it works when used inside the command handler. func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { return runInTx(h.db, func(tx *sql.Tx) error { if cmd.Points <= 0 { return errors.New("points must be greater than 0") } currentPoints, err := h.userRepository.GetPoints(ctx, tx, cmd.UserID) if err != nil { return fmt.Errorf("could not get points: %w", err) } if currentPoints < cmd.Points { return errors.New("not enough points") } err = h.userRepository.TakePoints(ctx, tx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not take points: %w", err) } err = h.discountRepository.AddDiscount(ctx, tx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not add discount: %w", err) } return nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/02-tx-in-logic/app.go Full source The good part is that this approach works. But we can’t ignore the fact that it mixes the application logic with implementation details (the SQL transaction). It may not seem like a big deal: we will likely not change the database, so why be concerned about it? For one, handling the transaction with the commit/rollback sequence complicates the flow. Instead of working with plain application logic, you now have to consider this additional behavior, which has nothing to do with what your application does. As the logic grows, you must carefully consider whether something should work within the transaction or outside of it. A rollback will affect everything you put inside the function. Not to mention the weird tx argument you must pass to the repository methods. Testing becomes awkward because you need to pass the transaction even if the method doesn’t need it at the time (the GetPoints method could also work without transaction in another context). You can’t mock the SQL connection to test the command handler, so it becomes an integration test instead of a simple unit test checking the logic. You could try to work around this with an abstract Transaction object or perhaps pass it through the Context, but it only masks the root issue of mixing the implementation details with the logic. ❌ Anti-pattern: Transactions mixed with logic Avoid mixing transactions with your application logic. It’s challenging to grasp how it works, test the logic, and debug issues. Transactions inside the repository (better, but far from perfect) If the transaction belongs to the database layer, why not keep it there? The hard part is that we deal with two repositories, so they must somehow share the transaction object. Sometimes, it’s really the case (and I’ll show you how to do it in a bit). But first, we should consider if we really need two repositories. It’s common to think of the database tables as your entities. This way, you end up with one repository per table and the application logic orchestrating them. Often, this split of repositories has no practical value. ❌ Anti-pattern: One repository per database table Don’t create a repository for each database table. Instead, think of the data that needs to be transactionally stored together. Data that should be transactionally consistent should also be coherent and kept as one. Domain-Driven Design proposes the idea of an aggregate — a set of data that must always be consistent. If you follow this idea, you don’t keep a repository per SQL table, but a repository per aggregate. (It’s the short version of the whole idea. We have a proper article coming up on aggregates soon.) Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! We think of the user and the discounts as separate concepts (entities, structs, etc.). And it makes sense, as conceptually, they are different things. The user is used for identity and authentication. Placing orders with discounts is just one of many things a user can do on the website. But we also want to keep the user’s points and discounts consistent. We can consider them part of the same aggregate — a set of objects stored together transactionally. In practice, we can consider discounts part of the User aggregate, even if we store them in another SQL table. In that case, we need just one repository. And this lets us move the transaction handling there. The application logic becomes trivial now. func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { if cmd.Points <= 0 { return errors.New("points must be greater than 0") } err := h.userRepository.UsePointsForDiscount(ctx, cmd.UserID, cmd.Points) if err != nil { return fmt.Errorf("could not use points as discount: %w", err) } return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/03-tx-in-repo/app.go Full source And the repository exposes a method specific to this one operation. func (r *PostgresUserRepository) UsePointsForDiscount(ctx context.Context, userID int, points int) error { return runInTx(r.db, func(tx *sql.Tx) error { row := tx.QueryRowContext(ctx, "SELECT points FROM users WHERE id = $1 FOR UPDATE", userID) var currentPoints int err := row.Scan(&currentPoints) if err != nil { return err } if currentPoints < points { return errors.New("not enough points") } _, err = tx.ExecContext(ctx, "UPDATE users SET points = points - $1 WHERE id = $2", points, userID) if err != nil { return err } _, err = tx.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = next_order_discount + $1 WHERE user_id = $2", points, userID) if err != nil { return err } return nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/03-tx-in-repo/repository.go Full source ✅ Tactic: Aggregates Keep the data that needs to be strongly consistent within the same aggregate. Keep a repository for the aggregate. This approach still has some downsides. One is that we had to move the “has enough points” logic to the repository. The method name (UsePointsForDiscount) makes it obvious the repository knows something about the logic, and occasionally it may be fine. But ideally, we’d have the logic in the command handler. The other issue is that with many methods like this, the repository’s interface grows and gets difficult to maintain and test. This is against the layers split we wanted to achieve. While useful, this approach doesn’t scale well. It seems we could use a more generic Update method. The UpdateFn Pattern (our go-to solution) A universal Update method should load the user from the database, allow us to make any changes, and store the result. The challenge, again, is how to make it work with transactions. First, we need a model that combines the user with discounts (our aggregate). I use encapsulation to hide the fields, as we want the model to be always valid in memory. The only way to change the state is by calling exported methods. type User struct { id int email string points int discounts *Discounts } func (u *User) ID() int { return u.id } func (u *User) Email() string { return u.email } func (u *User) Points() int { return u.points } func (u *User) Discounts() *Discounts { return u.discounts } type Discounts struct { nextOrderDiscount int } func (c *Discounts) NextOrderDiscount() int { return c.nextOrderDiscount } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/04-update-func-closure/models.go Full source Adding a discount works by calling UsePointsAsDiscount. func (u *User) UsePointsAsDiscount(points int) error { if points <= 0 { return errors.New("points must be greater than 0") } if u.points < points { return errors.New("not enough points") } u.points -= points u.discounts.nextOrderDiscount += points return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/04-update-func-closure/models.go Full source Easy to grasp, trivial to test. No need to think about database transactions when reading it. In fact, no need to think about database tables, too. That’s the kind of code I want to work with. Now, let’s introduce the update method interface. type UserRepository interface { UpdateByID(ctx context.Context, userID int, updateFn func(user *User) (bool, error)) error } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/04-update-func-closure/app.go Full source The updateFn argument is how we keep the logic separate from the database details. The function receives the User model that can be modified. Any changes made to it are saved to the storage if no error is returned. The updateFn returns an updated boolean value indicating whether the user has been updated. If true is returned, the repository saves the user. We don’t have a use case for returning false now, but it’s often helpful when updating multiple values (like in a generic PATCH update request). The update may be unnecessary and can be skipped then. The command handler looks like this now. func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { return h.userRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) { err := user.UsePointsAsDiscount(cmd.Points) if err != nil { return false, err } return true, nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/04-update-func-closure/app.go Full source Let’s take a look at the repository implementation. func (r *PostgresUserRepository) UpdateByID(ctx context.Context, userID int, updateFn func(user *User) (bool, error)) error { return runInTx(r.db, func(tx *sql.Tx) error { row := tx.QueryRowContext(ctx, "SELECT email, points FROM users WHERE id = $1 FOR UPDATE", userID) var email string var currentPoints int err := row.Scan(&email, &currentPoints) if err != nil { return err } row = tx.QueryRowContext(ctx, "SELECT next_order_discount FROM user_discounts WHERE user_id = $1 FOR UPDATE", userID) var discount int err = row.Scan(&discount) if err != nil { return err } discounts := UnmarshalDiscounts(discount) user := UnmarshalUser(userID, email, currentPoints, discounts) updated, err := updateFn(user) if err != nil { return err } if !updated { return nil } _, err = tx.ExecContext(ctx, "UPDATE users SET email = $1, points = $2 WHERE id = $3", user.Email(), user.Points(), user.ID()) if err != nil { return err } _, err = tx.ExecContext(ctx, "UPDATE user_discounts SET next_order_discount = $1 WHERE user_id = $2", user.Discounts().NextOrderDiscount(), user.ID()) if err != nil { return err } return nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/04-update-func-closure/repository.go Full source The repository has no logic at all. It fetches the complete data from the user and discounts tables, translates them to application models (with the Unmarshal functions), and calls the updateFn function. Then, it stores everything back in the database. All of it happens within a transaction. Tip The repository is quite verbose now. Using an ORM can help you get rid of the boilerplate. But be careful what library you choose, as some ORMs can do more harm than good. Check the libraries that never failed us. With this, we achieved the goal: the logic stays in the command handler, and the transaction remains in the repository. ✅ Tactic: The UpdateFn pattern Use an Update method that loads and stores the aggregate. Keep the logic in the updateFn closure. Note Isn’t this inefficient? A common concern about the Update method is that it needlessly updates all the fields even if they didn’t change (for example, the email in the example above). Unless you’re saving huge datasets or dealing with massive scale, it shouldn’t really matter. You could track which fields changed and update only these, but it will likely only complicate your architecture with no actual performance gains. Be pragmatic and avoid premature optimization. If in doubt, run stress tests. The Transaction Provider (for the edge cases) I mentioned that sometimes, keeping a transaction across two repositories makes sense. While your first reaction should be, “Why isn’t this a single repository?” there may be a valid use case for it. It usually works well for things that are not strictly related to your application’s domain but are more technical or platform-related. For example, consider keeping an audit log of the users’ actions. Saving it within the same transaction makes sense, but keeping it inside the User aggregate seems like overkill. We can work around having an explicit transaction object mixed with logic using the Transaction Provider pattern. It works similarly to the first approach, but the application code is easier to read with the transaction mechanism working in the background. type UsePointsAsDiscountHandler struct { txProvider txProvider } type txProvider interface { Transact(txFunc func(adapters Adapters) error) error } type Adapters struct { UserRepository UserRepository AuditLogRepository AuditLogRepository } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/05-tx-provider/app.go Full source The transaction provider has a Transact method that takes a function as an argument. Within it, we can access adapters — a struct of all the dependencies we need, working within the same transaction. Here’s how it looks in the command handler. func (h UsePointsAsDiscountHandler) Handle(ctx context.Context, cmd UsePointsAsDiscount) error { return h.txProvider.Transact(func(adapters Adapters) error { err := adapters.UserRepository.UpdateByID(ctx, cmd.UserID, func(user *User) (bool, error) { err := user.UsePointsAsDiscount(cmd.Points) if err != nil { return false, err } return true, nil }) if err != nil { return fmt.Errorf("could not use points as discount: %w", err) } log := fmt.Sprintf("used %d points as discount for user %d", cmd.Points, cmd.UserID) err = adapters.AuditLogRepository.StoreAuditLog(ctx, log) if err != nil { return fmt.Errorf("could not store audit log: %w", err) } return nil }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/05-tx-provider/app.go Full source Note that both UserRepository and AuditLogRepository come from adapters, not the handler. Also, we pass no transaction object to the repository methods. The repositories now take the tx in the constructor. It’s handy not to depend on *sql.Tx directly but to use an interface. It also lets you pass an *sql.DB (for example, in tests). type db interface { QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row ExecContext(ctx context.Context, query string, args ...interface{}) (sql.Result, error) } type PostgresUserRepository struct { db db } func NewPostgresUserRepository(db db) *PostgresUserRepository { return &PostgresUserRepository{ db: db, } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/05-tx-provider/repository.go Full source The TransactionProvider implementation looks like this: type TransactionProvider struct { db *sql.DB } func NewTransactionProvider(db *sql.DB) *TransactionProvider { return &TransactionProvider{ db: db, } } func (p *TransactionProvider) Transact(txFunc func(adapters Adapters) error) error { return runInTx(p.db, func(tx *sql.Tx) error { adapters := Adapters{ UserRepository: NewPostgresUserRepository(tx), AuditLogRepository: NewPostgresAuditLogRepository(tx), } return txFunc(adapters) }) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/04-transactions/05-tx-provider/app.go Full source This approach seems clean enough, although you have to be careful. With a set of transaction adapters, it’s easy to go too far and start calling methods from different repositories in a single handler. It shouldn’t become your go-to tool. Stick to the UpdateFn pattern instead. ✅ Tactic: The Transaction Provider Sometimes, it makes sense to share a transaction across repositories. In such a scenario, use the transaction provider pattern. Be careful not to over-use it. Warning The transaction provider is a risky pattern. If you use it together with FOR UPDATE, you need to be extra careful, as all the SELECTs need to include the FOR UPDATE clause to work correctly. This can quickly get out of control when working within a team with rapid changes. Once again, consider using a different isolation level, like REPEATABLE READ. Distributed Transactions This wraps up our way of working with SQL transactions within a single service. But there’s a related topic that often comes up and that we should consider: transactions across services. I cover this in Distributed Transactions in Go: Read Before You Try. Check the Source Code You can see the complete source code on GitHub. There are basic tests that verify that all examples work in the same way. You can run it all locally with Docker Compose.

In case you missed the memo, the Single Page Application hype period is over, and we’re now back to PHP and jQuery, I mean rendering HTML on the server. I’m excited! It brings me back to the early 2000s when we were all web developers, not frontend or backend engineers. But there’s one thing I would miss from the SPA era: live updates. The classic websites often relied on the “refresh” button, which wasn’t that great. While polling for updates periodically is a solution, it’s inefficient. It’s much better to push updates to the client once they happen. This post shows how to push live updates to your website using Go, Server-Sent Events (SSE), and htmx. As the example project, I use a tiny microblogging website where you can react to posts. Below, you can see the embedded example in two “windows”. You can click on reactions and see them update in real-time in the other window. You will also see the reactions and views counters update as other readers interact with the posts. (You can open the example in a new tab here). Server-Sent Events WebSockets seem like the most popular option for pushing live updates from the HTTP server to the browser. Meanwhile, Server-Sent Events (SSE) is a great alternative. It is simple to set up and good enough for many use cases. It uses standard HTTP connections, so you don’t need a custom protocol. All modern browsers support SSE. SSE endpoints work just like standard HTTP endpoints with a slight twist. You set the Content-Type header to text/event-stream and keep writing data in a text format like this: event: notifications data: {"unread_messages": 14} event: message data: <h1>Hello, data: world!</h1> What follows the event: line is an optional event type, which can be any string you want. The multiline data field is the payload sent to the browser. Every “event” is separated by an extra new line. The specification mentions a few more things, but that’s all you need to start. In Go, SSE endpoints are slightly different from net/http endpoints because you don’t just write data to the ResponseWriter and return from the function. Instead, you reply with the status code (200 OK) and keep writing the “events” in the format above. The connection stays alive until the client (the browser) closes it. Here’s a complete example of sending back a “ping” event every 10 seconds. func main() { http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "text/event-stream") w.WriteHeader(http.StatusOK) for { select { case <-r.Context().Done(): return default: } fmt.Fprintf(w, "data: ping\n\n") if f, ok := w.(http.Flusher); ok { f.Flush() } time.Sleep(10 * time.Second) } }) http.ListenAndServe(":8080", nil) } Two things are worth mentioning here. To avoid buffering, we use the http.Flusher to send the response immediately. To exit the infinite loop, we check if the request’s context is done. On the client side, you can handle it like this: <script> const es = new EventSource("http://localhost:8080/ping"); es.onmessage = (event) => { console.log(`Received: ${event.data}`); }; </script> Most online tutorials end here and wish you happy coding. Perhaps it’s good enough if you want to create a demo project in one afternoon. But the example is far from something you would use in production. I don’t want you to read this post, then go back to your project and think, “Uhh, so what do I do now?” If you follow our blog, you know we like to focus on real-world examples. Let’s see how to use SSE in a more complex scenario — the microblog example you saw at the top of this page. It’s not quick and easy, and there are many things to consider, but it should make you comfortable enough to use a similar mechanism in your projects. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! The Microblog Example The core component of the example is a “post”. The key feature is that the reactions and views are updated in real-time as other users interact with the post. In this example, we’ll render HTML on the server side, using no JavaScript code for interactivity except the htmx library. The complete example is on GitHub. You can run it locally using docker-compose. Note JSON API Example For another example featuring a Twitter-like web app, see the server-sent-events example on the Watermill repository. It shows a similar approach using a Single Page Application with Vue.js and a JSON API. Tools used Here’s the stack I use in the example. I won’t dive deep into all the components, but I’ll mention them briefly here. Echo — a lightweight HTTP router I like for error handling that is simpler than in net/http. templ — an HTML templates library based on code generation. It can get weird at times, but overall, I’m happy with it and like it more than html/template. It’s best used together with an IDE plugin. htmx — a library for using AJAX and SSE with no need to write JavaScript. Watermill — an event-driven library we maintain for working with messages. PostgreSQL and Google Cloud Pub/Sub for storage and messaging infrastructure. (You can choose a different Pub/Sub for messaging, even Postgres.) Deciding what and when to push When designing an SSE endpoint, you must decide what the payload should be, when to send an update, and to whom to send it. What The payload is just text, and it’s up to you how to encode it. It can be a regular JSON API response or an HTML you would embed directly on the website. Remember that each line should have the data: prefix, and the payload needs to end with two new lines (\n\n). When You need a way to know when something changes in your application so you can push the updates. For example, if the user receives a message, you show a red bubble in the UI. The SSE endpoints are long-running, so you may have hundreds or thousands of goroutines running in the background that you must notify of the change. In reaction, each should send an event to the client. Since you’re likely running more than one instance of your service, this can’t work in memory. To Whom You often only want to notify some users of something that happened. If I send you a message, I expect a notification to appear on your screen, but not for anyone else. So, you need a way to filter what happens and choose who should get the update (and which SSE endpoints to trigger). In this example, it goes as follows: What: the post “stats” model, including the numbers of views and reactions as HTML. When: when someone sees the post or reacts to it. To Whom: everyone who sees the updated post. Other posts are not updated. Implementing SSE Endpoints While you absolutely can create SSE endpoints from scratch (or with a library), and it’s not that complex, the hard part is triggering the updates in reaction to something that happened. (And doing this over the network since running a single service instance rarely happens in production.) As with running an HTTP server, you don’t want to reinvent the wheel here. We usually approach anything events-related with Watermill. It’s a Go library we maintain that abstracts away the low-level details of Pub/Subs. (Getting close to 7k GitHub stars ⭐️). You can use it with any existing codebase as it’s not a framework but a lightweight library (just like htmx). It supports many Pub/Subs, so it’s easy to start with the infrastructure you already have (even an SQL database). Watermill Primer (Feel free to skip this part if you’re already familiar with Watermill.) The documentation goes in-depth on how Watermill works. Below is a TL;DR version. First, you need a Pub/Sub — a system that lets you work with messages across the network (also known as a “message broker” or a “queue”). Common picks are Kafka or RabbitMQ, but it could just as well be an SQL database. Watermill abstracts away all Pub/Subs into two interfaces: type Publisher interface { Publish(topic string, messages ...*Message) error Close() error } type Subscriber interface { Subscribe(ctx context.Context, topic string) (<-chan *Message, error) Close() error } You can Publish messages and Subscribe to them. There’s always a topic involved — a string that decides who gets the message. Here’s basic Watermill architecture in one picture: The core part of Watermill is the Message. It is what the Request is for the net/http package. The simplest message has just an optional ID and a payload. The payload is a slice of bytes, so you can use any marshaling you want (JSON, Protocol Buffers, plain strings, etc.). msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!")) While all Watermill’s components are based on the Publisher and Subscriber interfaces, using them directly is a relatively low-level API. In this example, we’ll use the CQRS component of Watermill, which is a higher-level API. It’s based on the same ideas but removes some boilerplate, like serialization and deserialization. We’ll use the EventBus to publish events and EventProcessor to subscribe to them. High-level architecture overview We want to publish two events: PostViewed is published when someone sees the post. PostReactionAdded is published when someone reacts to a post. Each will have an event handler that updates the post’s stats in the database. (A similar concept as an HTTP handler.) The handlers should also publish the PostStatsUpdated event. We will use it to trigger the SSE updates. Publishing events First, let’s create a publisher. I use Google Cloud Pub/Sub Publisher, but it can be swapped with any other publisher supported by Watermill. All the configuration needs is a project ID. (Note that Google Cloud Pub/Sub is just one the Pub/Subs Watermill supports. You could easily change this to another supported Pub/Sub. Kind of like an ORM would work with MySQL, PostgreSQL, and SQLite.) logger := watermill.NewStdLogger(false, false) publisher, err := googlecloud.NewPublisher( googlecloud.PublisherConfig{ ProjectID: cfg.PubSubProjectID, }, logger, ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source Publisher works with messages, meaning you must marshal events (structs) into bytes and choose what topics to publish them to. It’s very common to use the same marshaling for all events and topics that follow some convention, like the event name being part of them. We will use the EventBus component to simplify the publishing API. You can think of it as a high-level wrapper on the publisher (as you can see, it’s the first argument). You pass the configuration options once and then can publish events with a single method call. eventBus, err := cqrs.NewEventBusWithConfig( publisher, cqrs.EventBusConfig{ GeneratePublishTopic: func(params cqrs.GenerateEventPublishTopicParams) (string, error) { return params.EventName, nil }, Marshaler: cqrs.JSONMarshaler{}, Logger: logger, }, ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source The configuration takes a Marshaler, so we use the cqrs.JSONMarshaler{} (all messages will be marshaled to JSON). The GeneratePublishTopic function returns the topic’s name based on the available parameters. Instead of passing a topic directly to Publish, we define this function to determine the topic based on the message. The EventBus calls this function every time a message is published. In this case, we’ll use the params.EventName. So, if you consider a struct like this: type PostViewed struct { PostID int `json:"post_id"` } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source It will be published on the PostViewed topic. (The chosen marshaler provides a way to extract the event name). Publishing events using the event bus is trivial. Thanks to the setup of marshaler and GeneratePublishTopic, we pass the event struct to Publish, and the rest happens behind the scenes. In the HTTP handler, we can use something like this: event := PostViewed{ PostID: post.ID, } err = h.eventBus.Publish(ctx, event) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/http.go Full source Subscribing to events I decided to make the HTTP endpoints just publish the events. The event handlers update the post’s stats in the database asynchronously. This way, the client doesn’t need to wait for the changes to be applied, and the view will be eventually updated via SSE. We need two event handlers to update the stats in the database. The first updates the views count, and the second updates the reactions count. The CQRS component used for subscribing to events is the EventProcessor. As with the EventBus, some setup needs to be done initially. But thanks to this, writing the handlers later will be very pleasant. The idea behind it is similar to the EventBus, but it’s the other way around. First, create a Router. It’s a similar concept to the HTTP routers you’re familiar with. The component runs in the background and routes messages to handlers. router, err := message.NewRouter(message.RouterConfig{}, logger) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source Similarly to HTTP routers, Watermill’s router supports middlewares. For example, you can add the Recoverer middleware so panics in handlers don’t blow up your server. router.AddMiddleware(middleware.Recoverer) Now we can create the EventProcessor. eventProcessor, err := cqrs.NewEventProcessorWithConfig( router, cqrs.EventProcessorConfig{ GenerateSubscribeTopic: func(params cqrs.EventProcessorGenerateSubscribeTopicParams) (string, error) { return params.EventName, nil }, SubscriberConstructor: func(params cqrs.EventProcessorSubscriberConstructorParams) (message.Subscriber, error) { return googlecloud.NewSubscriber( googlecloud.SubscriberConfig{ ProjectID: cfg.PubSubProjectID, GenerateSubscriptionName: func(topic string) string { return fmt.Sprintf("%v_%v", topic, params.HandlerName) }, }, logger, ) }, Marshaler: cqrs.JSONMarshaler{}, Logger: logger, }, ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source The first argument is the router. It’s similar to how the EventBus “wrapped” the publisher. Then comes the config. The Marshaler and GenerateSubscribeTopic are the same concepts as in the EventBus. The only difference is they come at the other end of the Pub/Sub. In the EventBus, the marshaler marshals the message, and the function decides to which topic to publish it. Here, the Marshaler unmarshals the message back on the struct, and GenerateSubscribeTopic decides to which topic to subscribe to. SubscriberConstructor is what the name says: it returns a new Subscriber. You may wonder, why not use a single subscriber, as we did with the publisher in the EventBus? Publishing messages is straightforward: you marshal a struct, send the bytes to a topic, and you’re done. Subscribing is where things get more interesting. For example, you run two replicas of the same service. How do you ensure that only one replica receives a message from the Pub/Sub? The strategy depends on the Pub/Sub. In Google Cloud Pub/Sub, you use a single “subscription” bound to a topic and share it among the replicas. That’s why having a subscriber constructor is helpful in this context. It allows us to specify what subscription to use for each event type. In this example, the subscription joins the topic name with the handler name. For example, PostViewed_UpdateViews. As promised, with the setup done, adding message handlers is quite simple. Note that the functions are generic (with inferred types), so you work with strongly typed events! The handler name is used to generate the subscription name, so it needs to be unique within handlers. err = eventProcessor.AddHandlers( cqrs.NewEventHandler( "UpdateViews", func(ctx context.Context, event *PostViewed) error { return repo.UpdatePost(ctx, event.PostID, func(post *Post) { post.Views++ }) }, ), cqrs.NewEventHandler( "UpdateReactions", func(ctx context.Context, event *PostReactionAdded) error { return repo.UpdatePost(ctx, event.PostID, func(post *Post) { post.Reactions[event.ReactionID]++ }) }, ), ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source Note If you’re curious about the UpdatePost method, check Robert’s article on the repository pattern. Tip Be careful when refactoring The names used for creating the topics and subscriptions are essential. If they were accidentally changed, you could lose messages. You should not change the event or handler names once they’re running in production. If you need to, consider creating a new event or handler. For example, PostViewedV2. The last part is running the router, just like you would run an HTTP server. go func() { err := router.Run(context.Background()) if err != nil { panic(err) } }() Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/main.go Full source Note To learn more about message routing, see the Consumer Groups Example. Publishing PostStatsUpdated We’ll use one more event to trigger the SSE updates: PostStatsUpdated. It includes the post’s ID and a record of what has been updated (views or the reaction ID). type PostStatsUpdated struct { PostID int `json:"post_id"` ViewsUpdated bool `json:"views_updated"` ReactionUpdated *string `json:"reaction_updated"` } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source Note Since the release of this post, I realized the naive approach of getting the post from the database on each update doesn’t scale well (hundreds of SELECT queries on each update, depending on how many visitors we have). I’ve updated the example so the PostStatsUpdated event includes all stats. This way, the SSE endpoint doesn’t need to query the database at all, except for the initial call. You can see how big was the impact on the database load. I left the original version in the code snippets for simplicity. You can see the changes on GitHub. After updating the post, both handlers should publish the PostStatsUpdated event. err = eventProcessor.AddHandlers( cqrs.NewEventHandler( "UpdateViews", func(ctx context.Context, event *PostViewed) error { err = repo.UpdatePost(ctx, event.PostID, func(post *Post) { post.Views++ }) if err != nil { return err } statsUpdated := PostStatsUpdated{ PostID: event.PostID, ViewsUpdated: true, } return eventBus.Publish(ctx, statsUpdated) }, ), cqrs.NewEventHandler( "UpdateReactions", func(ctx context.Context, event *PostReactionAdded) error { err := repo.UpdatePost(ctx, event.PostID, func(post *Post) { post.Reactions[event.ReactionID]++ }) if err != nil { return err } statsUpdated := PostStatsUpdated{ PostID: event.PostID, ReactionUpdated: &event.ReactionID, } return eventBus.Publish(ctx, statsUpdated) }, ), ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source SSE Router It’s time to implement the SSE endpoints. Watermill also provides an SSE component that works well with other internals. The main component is called SSE Router, and the idea behind it is pretty simple. When you call its AddHandler method, it subscribes to the given topic in the configured subscriber. The method returns a regular HTTP handler you can use with any HTTP router you want. Whenever a message appears in the chosen topic, it will be propagated in a fan-out fashion to all running SSE endpoints within. First, let’s create the SSERouter (it comes from the watermill-http package — use the v2 version!). The config requires an UpstreamSubscriber: you need to subscribe to a Pub/Sub that contains the events. We use Google Cloud Pub/Sub again. I’ll explain the details of the configuration a bit later. We use a string marshaler as we’re going to return plain HTML. subscriber, err := googlecloud.NewSubscriber(googlecloud.SubscriberConfig{ // ... }, logger) if err != nil { panic(err) } sseRouter, err := http.NewSSERouter(http.SSERouterConfig{ UpstreamSubscriber: subscriber, Marshaler: http.StringSSEMarshaler{}, }, logger) if err != nil { panic(err) } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source Then, you need to implement the StreamAdapter interface: type StreamAdapter interface { InitialStreamResponse(w http.ResponseWriter, r *http.Request) (response interface{}, ok bool) NextStreamResponse(r *http.Request, msg *message.Message) (response interface{}, ok bool) } Full source: github.com/ThreeDotsLabs/watermill-http/pkg/http/sse.go Full source These two methods are very similar. The first one is how you respond to the initial HTTP request. If needed, it lets you return an error and write it to the ResponseWriter. This is important because as soon as you write any data, it’s too late to change the response code or the headers. So, InitialStreamResponse is where you handle things like validation or authentication. If any errors happen, return ok equal false to stop the handler. Otherwise, what you return becomes the first event sent to the client. NextStreamResponse is called for each incoming Message. You can return a response to be sent to the SSE clients that use this endpoint. Or you can skip the message (again, return ok equal false). By default, whatever you return as the response is marshaled to JSON with event type data. You can override this with a custom marshaler, as we did here. You can also return the ServerSentEvent struct, which lets you explicitly specify the Event and Data fields. In our case, InitialStreamResponse simply returns the post’s response. func (s *statsStream) InitialStreamResponse(w http.ResponseWriter, r *http.Request) (response interface{}, ok bool) { postIDStr := r.PathValue("id") postID, err := strconv.Atoi(postIDStr) if err != nil { w.WriteHeader(http.StatusBadRequest) w.Write([]byte("invalid post ID")) return nil, false } resp, err := s.getResponse(r.Context(), postID, nil) if err != nil { w.WriteHeader(http.StatusInternalServerError) w.Write([]byte(err.Error())) return nil, false } return resp, true } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/http.go Full source NextStreamResponse is similar, but it also checks if the post’s ID in the event matches the one in the URL. If not, it skips the message. It means a post has been updated, but not the one this endpoint returns. func (s *statsStream) NextStreamResponse(r *http.Request, msg *message.Message) (response interface{}, ok bool) { postIDStr := r.PathValue("id") postID, err := strconv.Atoi(postIDStr) if err != nil { fmt.Println("invalid post ID") return nil, false } var event PostStatsUpdated err = json.Unmarshal(msg.Payload, &event) if err != nil { fmt.Println("cannot unmarshal: " + err.Error()) return "", false } if event.PostID != postID { return "", false } resp, err := s.getResponse(r.Context(), postID, &event) if err != nil { fmt.Println("could not get response: " + err.Error()) return nil, false } return resp, true } Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/http.go Full source As you can see, there’s no usual error handling here. The best we can do is log the error and return false to skip the message. The handler already replied with 200 OK, so it’s too late to change the status code. Alternatively, we could return a custom response with the error message to be displayed to the user. With the stream adapter ready, we can create a handler on the SSE Router: statsHandler := sseRouter.AddHandler("PostStatsUpdated", &statsStream{repo: repo}) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/http.go Full source The first argument here is the topic to listen to. The returned value is a ready-to-use http.HandlerFunc. Most HTTP libraries and frameworks are compatible with net/http, so you can use it with whatever HTTP router you like. I use echo, so there’s a small conversion of the path value needed: e.GET("/posts/:id/stats", func(c echo.Context) error { postID := c.Param("id") c.Request().SetPathValue("id", postID) statsHandler(c.Response(), c.Request()) return nil }) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/http.go Full source Finally, you need to run the SSE router in a separate goroutine: go func() { err := sseRouter.Run(context.Background()) if err != nil { panic(err) } }() Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/main.go Full source And that’s it! Whenever a message is published on the PostStatsUpdated topic, the SSE Router propagates it to all clients listening to the GET /posts/:id/stats endpoint. The handlers created by the SSE Router already handle all implementation details, so you don’t need to worry about setting the headers. When called without an Accept header or with a value other than text/event-stream, the handlers act as regular GET handlers, returning the response from InitialStreamResponse. Creating an EventStream in JavaScript automatically passes the header for you, but keep this in mind when debugging your endpoints with a tool like curl! # Regular HTTP response curl localhost:8080/posts/1/stats # SSE response curl -H "Accept: text/event-stream" localhost:8080/posts/1/stats Configuring the Subscriber Remember the part that we want each event to be processed only by one service replica at a time? In the case of events used for SSE, you need a counterintuitive approach: all subscribers need to process each event, as the SSE endpoints will be running across all of your service instances. In other words, in this case, each replica should have its own subscription that’s not shared with anyone. For Google Cloud Pub/Sub, an easy way to do it is to generate a unique subscription name on the service’s startup. For example, using a “short UUID” would generate subscription names like PostStatsUpdated_lkcNowPZ99M123xPwqcxp1. Keep in mind this can have some impact on your Pub/Sub. For example, when using Google Cloud Pub/Sub, it’s best to set the expiration policy for such subscriptions for one day, so they’re deleted when no longer used. There’s a hard limit of 10,000 subscriptions; you could quickly hit it this way. subscriber, err := googlecloud.NewSubscriber( googlecloud.SubscriberConfig{ ProjectID: cfg.PubSubProjectID, GenerateSubscriptionName: func(topic string) string { return fmt.Sprintf("%v_%v", topic, watermill.NewShortUUID()) }, SubscriptionConfig: pubsub.SubscriptionConfig{ ExpirationPolicy: time.Hour * 24, }, }, logger, ) Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/events.go Full source If that sounds like a lot to consider, it’s because there is! Running production-grade Event-Driven systems comes with many advantages, but it’s not trivial. That’s why I go into detail here, so you know what to expect in production, not just in toy examples. (If that sounds like something you’d like to learn in-depth, see our Go Event-Driven training.) htmx The last piece of the puzzle is the client-side code. In the example, we use htmx, a library that lets you make AJAX requests with HTML attributes. It also supports SSE with an extension. The templating in the snippets below comes from templ. <div hx-ext="sse" sse-connect={ "/posts/" + post.ID + "/stats" } sse-swap="data"></div> Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/views/pages.templ Full source The sse-swap attribute is the event type to look for from the stream. By default, Watermill’s SSE endpoints use data, so that’s what we use. Every time an event is received from the /posts/:id/stats endpoint, its payload will be injected inside the div. (Remember, our events are HTML.) We also use htmx to send the reaction form asynchronously (a classic POST AJAX request). In this case, we use hx-swap="outerHTML", which replaces the entire form with the response from the server. It’s a button with a ✅ “check” suggesting that the reaction has been added. The SSE will eventually update the stats. (Although there might be a slight delay. If you care about UX, returning a “fake” number increase could make sense here.) <form hx-post={ "/posts/" + postID + "/reactions"} hx-swap="outerHTML"> <input type="hidden" name="reaction_id" value={ reaction.ID } /> <button type="submit" class={"btn", "btn-outline-secondary", "m-1", templ.KV("animated", reaction.JustChanged)}> <span class="emoji">{ reaction.Label }</span> <span class="counter">{ reaction.Count }</span> </button> </form> Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/views/pages.templ Full source Animations If you’re used to working with Single Page Applications, using htmx might initially feel weird. For example, consider animating an element that has just been updated. In a classic SPA, you would get the JSON response from the SSE endpoint, compare the values with what’s in the “model”, and decide whether to animate the element. While htmx allows for “hooks” after the request is done, it’s probably not what you want. Instead, you need to adjust your mental model a bit. The server code is the source of truth in this setup. There is no “client” that decides how to display things. In this example, I use a CSS class to mark the updated element. The class includes an animation that pops up the element for a moment. The server code decides whether to add the class or not (based on the data in the incoming event). (templ.KV is how you add a class conditionally using templ. It will be present if stats.Views.JustChanged is true.) <div class={ "d-flex", "align-items-center", templ.KV("animated", stats.Views.JustChanged)}> <span class="me-1">👁️</span> <small class="text-muted">{ stats.Views.Count + " views" }</small> </div> Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/views/pages.templ Full source Other things to consider Two kinds of SSE endpoints How you return events from SSE endpoints is totally up to you. Here are two ways that make sense in different scenarios. An endpoint that returns the same data model initially and on every update. With each triggered update, you kind of “refresh” the model, perhaps embedded on the website. This is what we use in the example above. An endpoint that returns nothing initially and then keeps sending unique updates as they happen. You can append each new event to some list, for example. It’s how you would implement notifications or a web chat. At-least-once delivery When working with virtually any Pub/Sub, you must be aware of the “at-least-once” delivery guarantee. You may receive the same message twice because of network issues or your server going down at the wrong moment. Don’t try to work around this. Instead, embrace that this can happen and design your handlers to be idempotent. It means that processing the message twice (or more) has the same effect as processing it once. In the example above, we don’t guard against it. If the same message was processed twice, it would add an extra view or reaction in the database. It’s not a big deal in this case, and we can live with it. One way to prevent it would be to store the processed message IDs in the database and check it on each update. Watch out for HTTP/1.1 In modern browsers, there’s a limit of 6 open connections per server over HTTP/1.1, which can be a big issue when using SSE. Your website won’t work well if someone opens it in several tabs. For best results, use SSE with HTTP/2 where this limit doesn’t apply. Most modern web servers support HTTP/2, so make sure you enable it. Local environment tricks Here are two tips unrelated to SSE that might be useful for running your app locally. Mounting /go/pkg and go cache In the docker-compose definition, you can mount the /go/pkg and /go-cache directories to speed up the build process. This way, you don’t have to download the dependencies whenever you rebuild the container. services: server: # ... volumes: - go_pkg:/go/pkg - go_cache:/go-cache volumes: go_pkg: go_cache: Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/docker-compose.yml Full source Reflex for regenerating templ and rebuilding the server Reflex is my go-to tool for live code reloading. (See my post on the dev environment setup.) When working with templ, you can use a configuration like the one below to regenerate the templates and rebuild the server after every change. It’s a great way to speed up your development process, so you don’t need to keep switching between code, the terminal, and the browser. -r '(\.go$|go\.mod$)' -s go run . -r '\.templ$' templ generate Full source: github.com/ThreeDotsLabs/watermill/_examples/real-world-examples/server-sent-events-htmx/docker/reflex.conf Full source Go build something! That should be all the theory you need to build something cool. If you have any questions, let me know in the comments. Once again, the complete source code is on GitHub You can run it locally with docker-compose up. Give this stack a try; I had lots of fun working with it. Good luck!

Here’s a rant I often see in developer communities: I used to love programming because I like building stuff. But my full-time job killed my passion. I spend more time in meetings, fighting over deadlines, and arguing in reviews than working with code. Am I burned out? Is there hope, or do I need a new hobby? Sounds familiar? No wonder we keep looking forward to using a new framework or database — we’re bored. So here’s my tip to cure some of your burnout: Start coding for fun again and reclaim your hobby. And what is more fun than creating your own world with code? Wanting to make my own video game pushed me into programming 20 years ago. I didn’t make a career out of it, even though I created some small games using different tech stacks. I keep returning to it, and for me, it’s the most fun activity related to coding. Especially if done with friends over a weekend for a game jam. This post is my attempt at getting you to write games from scratch. (And if you want to pick up a new language too, check Go in One Evening!) Why Go? I like it, and Ebitengine, the library I will show here, is great to work with. Like many Go libraries, it helps you do what you need and stays out of the way — it’s not a framework. Of course, a bigger game engine simplifies many things but also comes with bloat and a specific approach. A big part of why making games is fun is you get to do things your way and don’t need to care if it’s the right way. And it’s a great exercise to learn how games work. First, a warning. Once you understand how games work, some of the magic is gone when playing them. You will start wondering, “How does it work?” instead of enjoying the game. Thankfully, making games compensates for this. Note Disclaimer: I’m a hobbyist game developer. I do it as it’s tons of fun, but I never worked on a professional game. Expect personal experience here. I might show some code that would be dumb when used in a “real” game (performance-wise, for example). Let me know in the comments if you find any errors! The basics I remember having a hard time understanding how exactly games work. How do you make the objects in the game do what they do? It’s more obvious if you use a game engine, but what if you start from scratch? You probably know that a movie is a very long collection of images, displayed one after the other so fast that our brains can’t tell the difference. Usually, 24 or 60 images (frames) per second (FPS). A video game is the same idea, but the images (frames) don’t exist beforehand, so you must generate (draw) them on the fly. And you can press some buttons to change how the next frame looks. This may seem too simplistic, right? “What? But how do you make a character jump?” You move the image up the screen over time and then back down. And you stop it from moving when it “touches” other images. 2D games are basically made of images drawn on top of each other. I was surprised to learn that games are basically infinite loops. You keep drawing images on the screen. for { DrawFrame() } Besides drawing, you need to update the logic of the game. It includes things like checking buttons pressed, updating the player’s health, and checking collisions. The logic decides what images to draw and where, and the Draw function does it. Each time the logic is updated is known as a tick. for { // One tick UpdateLogic() DrawFrame() } I know this looks hardly exciting, but let this idea sink in: your game does these two things over and over, 60 times per second. Just a single thread with this infinite loop (for now, forget goroutines exist). Even though the idea is simple, keeping the loop running at the same speed on all systems is not trivial. Ebitengine takes care of this for you, running the game at 60 ticks per second. Logic is the part you should be familiar with: it’s regular Go code. Drawing images is where things get more complex, and that’s why you use Ebitengine. It helps with drawing images, playing sounds, and checking inputs. How the logic works is mainly on you. Ebitengine provides a Game interface you need to implement. type Game interface { Update() error Draw(screen *ebiten.Image) Layout(outsideWidth, outsideHeight int) (screenWidth, screenHeight int) } The Update and Draw functions work as in the loop above. Layout is used for scaling the game screen — you don’t need to worry about it now. Update is called every tick and updates the game logic. If an error is returned, the game quits. Draw is called every frame. The screen argument is an image you draw on, like a canvas. Setting up the project First, let’s kick off the boring stuff: create a module and add Ebitengine (formerly called Ebiten, hence the package name). go mod init game go get github.com/hajimehoshi/ebiten/v2 Here’s the minimal code you need to start a game. It’s just an empty implementation of the Game interface. Running it should show a blank black window. package main import "github.com/hajimehoshi/ebiten/v2" type Game struct{} func (g *Game) Update() error { return nil } func (g *Game) Draw(screen *ebiten.Image) { } func (g *Game) Layout(outsideWidth, outsideHeight int) (screenWidth, screenHeight int) { return outsideWidth, outsideHeight } func main() { g := &Game{} err := ebiten.RunGame(g) if err != nil { panic(err) } } Note I’m pretty liberal with error handling in these examples. We’re just prototyping and having fun, so I’m okay with raising a panic. With most errors, you can either crash the game or log it and continue. Loading assets If your graphic skills are similar to mine (the term “programmer art” exists for a reason 🙃), you may prefer using ready assets when prototyping. Unless you want to create them yourself, I encourage you to play with free-to-use assets. Personally, I love the collection from Kenney. Every time I look for inspiration, I browse the 2D assets there and find something that looks like a cool game to make. I’ll use the space shooter pack below to make a trivial Asteroids clone. To load the assets in code, use embed — it’s like it was meant for making games. You know the paths are right on compile time. You also don’t need to care about the distribution, as the binary contains the assets. It works even for mobile apps! You will likely have many assets, so consider embedding the entire directory: import "embed" //go:embed assets/* var assets embed.FS You can now load the images from the assets collection. I’m using the must pattern here and keep sprites as global variables for simplicity. (A sprite is just another name for a 2D graphic.) Note the empty "image/png" import used for decoding. import ( "image" _ "image/png" "github.com/hajimehoshi/ebiten/v2" ) var PlayerSprite = mustLoadImage("assets/player.png") func mustLoadImage(name string) *ebiten.Image { f, err := assets.Open(name) if err != nil { panic(err) } defer f.Close() img, _, err := image.Decode(f) if err != nil { panic(err) } return ebiten.NewImageFromImage(img) } Drawing Images All the drawing happens in the Draw method. func (g *Game) Draw(screen *ebiten.Image) { } The screen argument is the image displayed every frame (after the method returns). Your job is to draw other images (or text) on it. The key method of ebiten.Image is DrawImage, which draws another image onto it. screen.DrawImage(PlayerSprite, nil) The first argument is the image you want to draw. The second is the ebiten.DrawImageOptions struct that decides where and how the image is drawn. Passing nil keeps the default options. Here’s the result: the player’s spaceship is drawn in the screen’s top-left corner. The drawing options struct has a few fields; the one to focus on first is the GeoM matrix. The docs go into detail on the math behind it, and math is super helpful in making games. But if I just wanted to make a game, the last thing I’d like to read about is algebra, so I’ll try to simplify it here. Do you know how when you copy and paste an image in a graphics editor, the pasted part stays selected, and you can keep moving and transforming it? That’s how I see DrawImage. You can think of DrawImageOptions as a “cursor” you can use to position and change the image. Drawing uses X and Y coordinates, with the Y axis going downward. By default, everything is drawn at the (0, 0) point, so the top-left corner of the base image. To move the image, use GeoM.Translate: op := &ebiten.DrawImageOptions{} op.GeoM.Translate(150, 200) screen.DrawImage(PlayerSprite, op) This Translate call moves the image 150 pixels right and 200 pixels down. You can use negative numbers to go left and up, respectively. (If you move too far away, you may not see the image on the screen). Use GeoM.Rotate to rotate the image: op := &ebiten.DrawImageOptions{} op.GeoM.Rotate(45.0 * math.Pi / 180.0) screen.DrawImage(PlayerSprite, op) This rotates the image 45° clockwise. (The unit is radians. If you prefer working with degrees, use the degrees * math.Pi / 180.0 formula to get them.) Can you see how the image is partially off-screen now? That’s because the pivot point (the point “around” which the image is being rotated) is the top-left corner of the image. I’ll show how to work around this in a bit. Using negative values rotates the image counter-clockwise. Finally, there’s GeoM.Scale, which lets you draw smaller or bigger images: op := &ebiten.DrawImageOptions{} op.GeoM.Scale(2, 2) screen.DrawImage(PlayerSprite, op) This Scale call makes the image twice as big horizontally and vertically. Passing negative values flips the image horizontally or vertically. For example, to draw the image upside-down, use: op.GeoM.Scale(1, -1) You now know the basic tools to control images in a game: their position, rotation, and scale. You can combine all of them in a single DrawImage call. There’s a caveat, though: the order matters. If you use Translate first and then apply the rotation or scale, you may not see what you expect because the change will be applied to the new position. To illustrate this, let’s see how to rotate the image around its center (a common use case). You need to move the pivot point to the image’s center. You can find it simply by dividing the image’s width and height by half. (Calculating stuff based on the image size is also super common.) width := PlayerSprite.Bounds().Dx() height := PlayerSprite.Bounds().Dy() halfW := float64(width / 2) halfH := float64(height / 2) You need to move the image by the negative values first. This is so the image’s center aligns with the origin point (0, 0). Then, apply the rotation and move the image “back” by the same amount. op.GeoM.Translate(-halfW, -halfH) op.GeoM.Rotate(45.0 * math.Pi / 180.0) op.GeoM.Translate(halfW, halfH) It might be easier to grasp using this animation of all steps (scaled three times): Changing colors Another useful option is changing the sprite’s color. You can use it to draw an image that looks like the sprite but with a solid color. Of course, this makes the most sense if you use sprites with a transparent background. The syntax is similar, although you need to use the recently added colorm package. op := &colorm.DrawImageOptions{} cm := colorm.ColorM{} cm.Translate(1.0, 1.0, 1.0, 0.0) colorm.DrawImage(screen, PlayerSprite, cm, op) The first three arguments of Translate are red, green, and blue values and go from 0.0 to 1.0 (0% to 100%). The last one is alpha (transparency). You can control the alpha field to make sprites transparent, creating cool effects like adding a shadow. To keep the transparent background unchanged, use Scale instead of Translate. Keep the first three arguments at 1.0, so they don’t change. op := &colorm.DrawImageOptions{} cm := colorm.ColorM{} cm.Scale(1.0, 1.0, 1.0, 0.5) colorm.DrawImage(screen, PlayerSprite, cm, op) Combining options You can do more with DrawImageOptions, but this should be all you need to start making games. For example, here’s an evolution animation I created in airplanes: I really like how it turned out. The best part is that I didn’t use graphic assets for the animation. It’s all done with DrawImageOptions and goes like this: Change the color of the sprite to white. Create a new sprite (also white) on top of the current one, with the scale set to (0, 0) (invisible). Scale the new sprite to scale (1, 1) over time. Scale the old sprite to scale (0, 0) over time. Remove the white color. Can you notice how the shadow evolves as well? It’s the same idea but using a gray sprite and some transparency. Are you experienced engineer who wants to learn Go basics? You don't become an engineer by watching videos. Learn Go hands-on by building real projects. The logic To make the game do anything interesting, you need to fill in the Update method. You keep all logic there while the Draw method draws images (on the screen or on top of each other). The basic idea of the game logic is simple: the Game struct holds some state, and the Update method changes this state. Draw reads the state and draws images based on it. A classic example of a game state is the player’s position. In 2D games, this will usually be a pair of (X, Y) values, also known as a vector. type Vector struct { X float64 Y float64 } type Game struct { playerPosition Vector } The zero-value of Vector is (0, 0), like any struct in Go. Let’s initialize the game state with a predefined value: g := &Game{ playerPosition: Vector{X: 100, Y: 100}, } All left to do is to update the Draw method so it draws the player’s sprite at the given position. This shouldn’t be surprising: op := &ebiten.DrawImageOptions{} op.GeoM.Translate(g.playerPosition.X, g.playerPosition.Y) screen.DrawImage(PlayerSprite, op) Note how we call Translate always with the current position of the player. Each time Draw is called, the screen is empty, and the images are drawn from scratch. (Ebitengine might optimize this, but we don’t need to care now.) Movement To move the player to the right, keep increasing the X position in Update. Do you remember that Update is called in an infinite loop? This becomes important now, as you need to decide how fast to move the player. func (g *Game) Update() error { speed := 5.0 g.playerPosition.X += speed return nil } speed is the number of pixels the position changes in a single tick (one Update call). With the default 60 ticks per second, the image would move 300 pixels per second to the right. It may be easier to keep the speed as “pixels per second” instead of “pixels per tick”. In this case, calculate it like this: // Move 300 pixels per second speed := float64(300 / ebiten.TPS()) Controls Let’s now allow moving the player using the arrow keys. The idea stays the same: we need to update the position. Not every tick, though, but only if a key is pressed. func (g *Game) Update() error { speed := 5.0 if ebiten.IsKeyPressed(ebiten.KeyDown) { g.playerPosition.Y += speed } if ebiten.IsKeyPressed(ebiten.KeyUp) { g.playerPosition.Y -= speed } if ebiten.IsKeyPressed(ebiten.KeyLeft) { g.playerPosition.X -= speed } if ebiten.IsKeyPressed(ebiten.KeyRight) { g.playerPosition.X += speed } return nil } ebiten.IsKeyPressed returns true if the given key is pressed in the current “tick” (remember, Update is called about 60 times per second by default). The other function you might need is ebitenutil.IsKeyJustPressed, which returns true just in the tick the player presses the key. It’s the difference between “hold space to keep jumping” and “even if you keep holding space, you jump just once”. Note it’s in the ebitenutil package not ebiten, in contrast to IsKeyPressed. Note how we don’t use if-else — thanks to this, the player can move diagonally by holding two arrows at a time. Tip Moving diagonally this way is naive and faster than moving in one direction at a time. If that’s not what you want, you need to adjust for it. var delta Vector if ebiten.IsKeyPressed(ebiten.KeyDown) { delta.Y = -speed } if ebiten.IsKeyPressed(ebiten.KeyUp) { delta.Y = speed } if ebiten.IsKeyPressed(ebiten.KeyLeft) { delta.X = -speed } if ebiten.IsKeyPressed(ebiten.KeyRight) { delta.X = speed } // Check for diagonal movement if delta.X != 0 && delta.Y != 0 { factor := speed / math.Sqrt(delta.X*delta.X+delta.Y*delta.Y) delta.X *= factor delta.Y *= factor } g.playerPosition.X += delta.X g.playerPosition.Y += delta.Y Timers A very common need in games is changing the logic with time. For example, you want the enemy object to move for two seconds to the right and then back to the left. Or you want the boss to cast a spell every five seconds. You can’t use a regular way to measure time, using time.Now() and time.Since() due to how the Update method works (running at a constant speed of 60 ticks per second). The idea is to keep counting ticks that have happened since the timer started. Then, you take some action and reset it. I like to keep the Timer as a separate struct with a nice API. type Timer struct { currentTicks int targetTicks int } func NewTimer(d time.Duration) *Timer { return &Timer{ currentTicks: 0, targetTicks: int(d.Milliseconds()) * ebiten.TPS() / 1000, } } func (t *Timer) Update() { if t.currentTicks < t.targetTicks { t.currentTicks++ } } func (t *Timer) IsReady() bool { return t.currentTicks >= t.targetTicks } func (t *Timer) Reset() { t.currentTicks = 0 } And here’s a short example of using it: g := &Game{ attackTimer: NewTimer(5 * time.Second), } // ... func (g *Game) Update() error { g.attackTimer.Update() if g.attackTimer.IsReady() { g.attackTimer.Reset() // Execute the attack! } } Game Objects Keeping the player’s position directly in the Game struct is like using global variables. It would be challenging to maintain non-trivial games this way. Encapsulation is a good improvement: having building blocks you can use instead of dealing with the details. A simple game object (or entity or whatever you want to call it) can be a struct with a position and sprite. There are many approaches to organizing the game logic, and you can use whatever makes sense to you. An easy way to start is to have each game object expose its own Update and Draw methods, and call them in the Game’s Update and Draw. type Player struct { position Vector sprite *ebiten.Image } func NewPlayer() *Player { return &Player{ position: Vector{X: 100, Y: 100}, sprite: PlayerSprite, } } func (p *Player) Update() { } func (p *Player) Draw(screen *ebiten.Image) { op := &ebiten.DrawImageOptions{} op.GeoM.Translate(p.position.X, p.position.Y) screen.DrawImage(p.sprite, op) } Now the Game methods don’t concern about the details of the Player: type Game struct { player *Player } func (g *Game) Update() error { g.player.Update() } func (g *Game) Draw(screen *ebiten.Image) { g.player.Draw(screen) } Let’s finish the Player implementation for this Asteroids clone. The player should be spawned in the center of the screen. It’s easy to calculate if we know the screen’s width and height. First, define the screen size as constants. const ( ScreenWidth = 800 ScreenHeight = 600 ) Now is the time we use the third method of ebiten.Game’s interface: the Layout. It should return the size of the game window. func (g *Game) Layout(outsideWidth, outsideHeight int) (int, int) { return ScreenWidth, ScreenHeight } Now, back to the NewPlayer constructor to set the initial position. The center of the screen is half of the screen’s width and height (X and Y). But remember, the player’s sprite top-left corner will be drawn at the given position. To keep it precisely in the center, we must move it left and up by half the sprite’s width and height. func NewPlayer() *Player { sprite := PlayerSprite bounds := sprite.Bounds() halfW := float64(bounds.Dx()) / 2 halfH := float64(bounds.Dy()) / 2 pos := Vector{ X: ScreenWidth/2 - halfW, Y: ScreenHeight/2 - halfH, } return &Player{ position: pos, sprite: sprite, } } Instead of moving the spaceship, the player should be able to rotate it. We can keep the rotation as float64 in the Player struct. func (p *Player) Update() { speed := math.Pi / float64(ebiten.TPS()) if ebiten.IsKeyPressed(ebiten.KeyLeft) { p.rotation -= speed } if ebiten.IsKeyPressed(ebiten.KeyRight) { p.rotation += speed } } Remember, the rotation unit is radians. 2π is a full rotation. I’ll stick to π / TPS, which means the player can rotate 180° per second. Now, we need to update the Draw struct to take rotation into account (using the trick of rotating the image around its center). func (p *Player) Draw(screen *ebiten.Image) { bounds := p.sprite.Bounds() halfW := float64(bounds.Dx()) / 2 halfH := float64(bounds.Dy()) / 2 op := &ebiten.DrawImageOptions{} op.GeoM.Translate(-halfW, -halfH) op.GeoM.Rotate(p.rotation) op.GeoM.Translate(halfW, halfH) op.GeoM.Translate(p.position.X, p.position.Y) screen.DrawImage(p.sprite, op) } Spawning objects Following the game object idea, let’s create a Meteor object. We can load many assets and pick one randomly to make it more interesting. var MeteorSprites = mustLoadImages("meteors/*.png") type Meteor struct { position Vector sprite *ebiten.Image } func NewMeteor() *Meteor { sprite := MeteorSprites[rand.Intn(len(MeteorSprites))] return &Meteor{ position: Vector{}, sprite: sprite, } } The Draw method is pretty much the same as for Player. (As you can imagine, there’s room for improvement here to keep the common code in one place — more on this later). Now, Game needs to keep track of meteors. It’s the same idea as with Player, except we now need a slice of objects. Let’s also add a spawn timer. type Game struct { player *Player meteorSpawnTimer *Timer meteors []*Meteor } Update and Draw iterate over all meteors and call their respective methods. And every time the timer is ready, a new meteor is added to the slice. func (g *Game) Update() error { g.player.Update() g.meteorSpawnTimer.Update() if g.meteorSpawnTimer.IsReady() { g.meteorSpawnTimer.Reset() m := NewMeteor() g.meteors = append(g.meteors, m) } for _, m := range g.meteors { m.Update() } return nil } func (g *Game) Draw(screen *ebiten.Image) { g.player.Draw(screen) for _, m := range g.meteors { m.Draw(screen) } } Tip Note that the order of drawing matters. The entities you draw later in Draw will appear on top of the ones drawn earlier (if their positions overlap). Details of the meteor The meteor should spawn at the edge of the screen. Here’s one approach to do it. // Figure out the target position — the screen center, in this case target := Vector{ X: ScreenWidth / 2, Y: SreenHeight / 2, } // The distance from the center the meteor should spawn at — half the width r := ScreenWidth / 2.0 // Pick a random angle — 2π is 360° — so this returns 0° to 360° angle := rand.Float64() * 2 * math.Pi // Figure out the spawn position by moving r pixels from the target at the chosen angle pos := Vector{ X: target.X + math.Cos(angle)*r, Y: target.Y + math.Sin(angle)*r, } As I mentioned, math is often handy for making games! You don’t need to know how it works in depth; just know the basic operations like “move 100 pixels from position (100, 200) at 30° angle”. (There’s absolutely no shame in having to consult the internet or AI every time you do it. 😅) Next, the meteor should keep moving toward the player’s position (the screen center). // Randomized velocity velocity := 0.25 + rand.Float64()*1.5 // Direction is the target minus the current position direction := Vector{ X: target.X - pos.X, Y: target.Y - pos.Y, } // Normalize the vector — get just the direction without the length normalizedDirection := direction.Normalize() // Multiply the direction by velocity movement := Vector{ X: normalizedDirection.X * velocity, Y: normalizedDirection.Y * velocity, } NewMeteor can store the movement in the struct, and then the Update becomes trivial: func (m *Meteor) Update() { m.position.X += m.movement.X m.position.Y += m.movement.Y } A final touch is adding a random rotation to each meteor. You can calculate it in the constructor: rotationSpeed := -0.02 + rand.Float64()*0.04, Then, keep moving and rotating the meteor. func (m *Meteor) Update() { m.position.X += m.movement.X m.position.Y += m.movement.Y m.rotation += m.rotationSpeed } Shooting Bullets Shooting bullets shouldn’t be surprising by now, so I won’t describe it in detail. You can see the complete source in the repository. We need to: Keep a timer with a shooting cooldown. Spawn a new bullet when a button is pressed. Rotate the bullet and keep it moving in the direction it faces. p.shootCooldown.Update() if p.shootCooldown.IsReady() && ebiten.IsKeyPressed(ebiten.KeySpace) { p.shootCooldown.Reset() // Spawn the bullet } One new thing is having Player spawn bullets instead of Game. But Game needs to keep track of the bullets, similar to meteors. The simplest way is to pass Game to Player’s constructor and keep it in the struct. Then, expose an AddBullet method from Game. It’s not ideal, as it creates a cross-dependency between Player and Game, but it’s good enough for now. I decided to calculate the spawn point on the Player side and make the Bullet’s constructor take it as an argument. bulletSpawnOffset := 50.0 bounds := p.sprite.Bounds() halfW := float64(bounds.Dx()) / 2 halfH := float64(bounds.Dy()) / 2 spawnPos := Vector{ p.position.X + halfW + math.Sin(p.rotation)*bulletSpawnOffset, p.position.Y + halfH + math.Cos(p.rotation)*-bulletSpawnOffset, } bullet := NewBullet(spawnPos, p.rotation) p.game.AddBullet(bullet) Collisions Detecting collisions is a complex topic, although, at the basic level, the concept is as simple as iterating over all objects and checking if they intersect. To make things easier, let’s introduce a Rect struct that represents a rectangle and makes it easy to check intersections. type Rect struct { X float64 Y float64 Width float64 Height float64 } func NewRect(x, y, width, height float64) Rect { return Rect{ X: x, Y: y, Width: width, Height: height, } } func (r Rect) MaxX() float64 { return r.X + r.Width } func (r Rect) MaxY() float64 { return r.Y + r.Height } func (r Rect) Intersects(other Rect) bool { return r.X <= other.MaxX() && other.X <= r.MaxX() && r.Y <= other.MaxY() && other.Y <= r.MaxY() } Next, each game object exposes a Collider() Rect method. func (p *Player) Collider() Rect { bounds := p.sprite.Bounds() return NewRect( p.position.X, p.position.Y, float64(bounds.Dx()), float64(bounds.Dy()), ) } In the Game’s Update, iterate over all objects and check for collisions. for i, m := range g.meteors { for j, b := range g.bullets { if m.Collider().Intersects(b.Collider()) { // A meteor collided with a bullet } } } for _, m := range g.meteors { if m.Collider().Intersects(g.player.Collider()) { // A meteor collided with the player } } When a bullet collides with a meteor, we want to destroy both objects. In our case, this means just dropping them using slice operations. If they’re gone from the list, the game won’t call their Update and Draw methods, so they effectively disappear (and the garbage collector handles the rest for us). if m.Collider().Intersects(b.Collider()) { g.meteors = append(g.meteors[:i], g.meteors[i+1:]...) g.bullets = append(g.bullets[:j], g.bullets[j+1:]...) } A meteor colliding with the player means the game is over. Let’s restart the game in this case. func (g *Game) Reset() { g.player = NewPlayer(g) g.meteors = nil g.bullets = nil } UI Drawing UI is similar to drawing other sprites. Two rules of thumb: It’s usually the last layer you draw, so it always stays on top. Sometimes, you want it on a separate section of the screen. In this case, you want to draw the game not directly on the screen but on a smaller canvas that you eventually draw next to the UI. (You can create an empty image with ebiten.NewImage and use it as a “panel”). A classic UI example is the score the player gets after shooting down a meteor. We can keep the score as an integer in the Game struct, increment it when a meteor collides with a bullet, and reset it to zero on the game over. What’s left is drawing the score on the screen. First, you need to load a font, which is similar to loading a sprite. (I’m using Kenney Fonts.) var ScoreFont = mustLoadFont("font.ttf") func mustLoadFont(name string) font.Face { f, err := assets.ReadFile(name) if err != nil { panic(err) } tt, err := opentype.Parse(f) if err != nil { panic(err) } face, err := opentype.NewFace(tt, &opentype.FaceOptions{ Size: 48, DPI: 72, Hinting: font.HintingVertical, }) if err != nil { panic(err) } return face } Then, call text.Draw in Game’s Draw: text.Draw(screen, fmt.Sprintf("%06d", g.score), ScoreFont, ScreenWidth/2-100, 50, color.White) If you want to be precise with the position, use font.BoundString to get the bounds. In the example above, I just estimated something that looks good enough. If you like this article, check out our materials on topics other than games.You can start with our e-book on modern Go applications! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Other concepts This article is already quite long, but I want to mention some ideas you may find helpful once you start making games. Scenes A scene is a standalone part of your game. For example, the welcome screen, the main menu, or the game itself. In the meteors project, there’s a single Game scene. This could be extended so it’s possible to switch between scenes, and each has its own Update and Draw methods. (And manages its own game objects.) Camera I was initially intimidated by the camera concept, but it turned out to be a simple idea. Your game “world” can be a big area, and you often don’t want to show all of it at once. With the camera concept, you draw only part of the bigger image. You can move the camera, make it follow the player, or scale it up and down to achieve the zoom effect. A trivial example needs just two things: A Camera struct that keeps the position (and optionally scale and other fields). An offscreen image (initialized with ebiten.NewImage). type Game struct { camera *Camera offscreen *ebiten.Image player *Player enemies []*Enemy } func (g *Game) Draw(screen *ebiten.Image){ g.offscreen.Clear() // Draw the game on the offscreen image g.player.Draw(g.offscreen) for _, e := range g.enemies { e.Draw(g.offscreen) } // Draw the offscreen image on the screen op := &ebiten.DrawImageOptions{} op.GeoM.Translate(-g.camera.X, -g.camera.Y) screen.DrawImage(g.offscreen, op) } As you update the camera’s position, the part of the offscreen image will also change. Animations An animation means changing the sprite of an object over time. Sometimes, just two or three sprites can create an animation that makes the game look more alive. To implement it, create a slice of sprites, a timer, and a “current sprite index”. Then, increase the index as the timer resets. func (p *Player) Update() { p.timer.Update() if p.timer.IsReady() { p.timer.Reset() p.index++ if p.index >= len(p.sprites) { p.index = 0 } p.sprite = p.sprites[p.index] } } Animation sprites often come as a single file with frames next to each other. You can use the SubImage method to extract a single frame from the loaded image. You can also “animate” other parameters of an object, like the sprite’s color or scale. Deploying to the web It’s easy to deploy your game with WebAssembly and host it anywhere. See more details in the docs and on GitHub. Entity Component System You probably noticed many common parts between the Player, Meteor, and Bullet game objects. It’s tempting to extract the common code so it’s easier to create new objects, and it definitely makes sense. One way to do it is by keeping a generic GameObject struct from which other game objects inherit. In Go, you could use struct embedding with similar results. Entity Component System (ECS) is another approach that prefers composition over inheritance. The idea is to keep common components like position, sprite, and collider as separate structs and then create game objects (entities) by combining them. All the logic is kept in systems that iterate over all entities and update them. There are many ECS libraries for Go. My pick is donburi, and my airplanes game is based on it. Debugging As with any code, you will sometimes get stuck with hard-to-debug bugs. While using prints for debugging sometimes helps, I like adding visual helpers in place, sort of “debug mode.” It often shows in a very explicit way where the issue is. You can make it appear and hide after a key so it doesn’t interfere with regular gameplay. Here’s an example debug mode I introduced in airplanes: Ebitengine provides some utils for this: ebitenutil.DebugPrintAt — to quickly draw a text. The vector package has functions like StrokeLine, StrokeRect, and StroneCircle. Go build something! All this barely scratches the surface of Gamedev. I picked topics that should let you make games and learn the basics. Let me know if you’d like to see more posts on some topics. To learn more about Ebitengine, the official examples are a great starting point. My example project is on GitHub, and you can play it online. Feel free to fork the repository and iterate over it! Here are some ideas to implement: Add support for controls using a mouse or gamepad. Make big meteors “break” into small ones instead of disappearing. Have meteors drop power-ups that the player can collect. Allow switching weapons. Add AI spaceships that follow a more complex pattern than flying towards the player. Add more scenes like the main menu and high scores. Add sound effects. (It really makes a difference!) If you can’t figure out how to create some mechanic you came up with (and it will happen a lot), try to figure out each step that needs to happen. For me, sketching using pen and paper often helps. Now, it’s time to go and build something. Have fun!

Hey, it’s been a long time! We’re happy to share that Watermill v1.3 is now out! What is Watermill Watermill is an open-source library for building message-driven or event-driven applications the easy way in Go. Our definition of “easy” is as easy as building an HTTP server in Go. With all that, it’s a library, not a framework. So your application is not tied to Watermill forever. Currently, Watermill has over 6k stars on GitHub, has over 50 contributors, and has been used by numerous projects in the last 4 years. It makes Watermill the most popular open-source event-driven library in Go. Watermill provides support for 12 different Pub/Sub implementations, including Kafka, Redis Streams, NATS, Google Cloud Pub/Sub, Amazon SQS/SNS (alpha), SQL, RabbitMQ and more. It’s also possible to provide a custom Pub/Sub implementation if needed. We believe that the foundation of every good open-source is excellent documentation together with real-life examples showing all features. Yesterday, we just wrapped up the v1.3 release. We included multiple features that make Watermill even more powerful and easy to use. What we’ve been up to over the last year In March, we kicked off the work on our second training, Go Event-Driven. To check the interest, we organized a pre-sale with the goal of selling at least 50 copies. To our amazement, 323 copies sold within a week and generated over $54k of income. Even though we didn’t have any content ready yet. Working on the training limited our time for Watermill, but on the other hand, it created an excellent synergy. While coming up with examples for the exercises, we also found a few improvements to Watermill. Finally, once we finished the training, we could wrap up the work on Watermill v1.3. Celebrating the Launch You can watch the release party recording on YouTube. Let’s dive deeper into the changes! What’s new? For the complete list of changes, see the GitHub releases page. Big thanks go to everybody who contributed to this release! 💪 @AlexCuse @tjnet @hlubek @vladtenlive @dkotik New CQRS API There are three APIs for handling messages in Watermill: The raw Subscribe() calls on a Subscriber. The Router and message handlers. The cqrs component and event/command handlers. Each is a more high-level abstraction than the previous one. There’s no best API — depending on your use case, you may want to use one or the other. During the last release, we mentioned we wanted to introduce a generic API to eliminate conversions from the interface{} (now: any). At the same time, we noticed some design issues with the CQRS component. The way to use it was via the Facade, which wired everything up using a big config. If you wanted to create a simple setup, it felt like too much effort compared to the Router. Here’s what changed in v1.3. The cqrs.Facade is now deprecated We won’t remove it, so your current code will keep working as before. We recommend using the new API instead of the Facade for new use cases. For consuming messages, you want to use the CommandProcessor and EventProcessor: cqrs.NewCommandProcessorWithConfig cqrs.NewEventProcessorWithConfig For publishing messages, you want to use the CommandBus and EventBus: cqrs.NewCommandBusWithConfig cqrs.NewEventBusWithConfig The bus and processor concepts are not new. The previous API also had them, so this change is backward-compatible. Their API is now more cohesive and easier to use. Generic handlers With the new API, you can use the same event or command handlers as before. There’s also the new generic API that eliminates the type assertion and is easier to write. This is the original API (still supported): type OrderPlacedHandler struct {} func (h OrderPlacedHandler) HandlerName() string { return "OrderPlacedHandler" } func (h OrderPlacedHandler) NewCommand() interface{} { return &OrderPlaced{} } func (h OrderPlacedHandler) Handle(ctx context.Context, e interface{}) error { event := e.(*OrderPlaced) return nil } And this is the new generic API: cqrs.NewEventHandler( "OrderPlacedHandler", func(ctx context.Context, event *OrderPlaced) error { // No more type assertions! return nil }, ) Of course, you can still use structs for dependency injection: type OrderPlacedHandler struct { repository OrderRepository } func (h OrderPlacedHandler) Handle(ctx context.Context, event *OrderPlaced) error { return h.repository.Save(event) } h := OrderPlacedHandler{ repository: NewRepository(), } eventHandler := cqrs.NewEventHandler("OrderPlacedHandler", h.Handle) Event handler groups The EventProcessor creates one subscriber for each event handler. This works great out of the box for the common use case of one event type per topic. If you keep all your events on one topic, you can use the new AddHandlersGroup() method to create a single subscriber for all specified handlers. This is especially important if you care about the message ordering. Otherwise, your events may be processed in parallel by different handlers. err := eventProcessor.AddHandlersGroup( "events", UserSignedUpHandler{}, OrderPlacedHandler{}, OrderCancelledHandler{}, ) In this setup, set the AckOnUnknownEvent config option to true so the processor skips events that don’t have a handler assigned. Circuit Breaker Middleware A new CircuitBreaker middleware implements the circuit breaker pattern. It uses the gobreaker library. Like the Throttle middleware, it can help to avoid overloading your downstream services in case of a failure. If processing the message fails several times, the middleware immediately returns an error without calling the handler for the following messages. After some time, it tries to rerun the handler. If it succeeds, it returns to normal. Otherwise, it keeps failing. The number of fails, the time it needs to fail within, and the time to wait before retrying are configurable, among other things. Request-Reply support for commands While communication over messages is asynchronous most of the time, it sometimes makes sense to receive a confirmation or result from a command. Many Pub/Subs support this use case with the request-reply pattern. We introduced a similar feature that works on top of any Pub/Sub. The new requestreply package works on top of the cqrs.CommandBus. It allows you to send a command and wait for a reply. The Reply can contain an error or, optionally, a (generic) result. If you are returning just an error from the handler, the function signature is compatible with the generic handler signature. err := commandProcessor.AddHandlers( requestreply.NewCommandHandler( "hotel_room_booking", ts.RequestReplyBackend, func(ctx context.Context, cmd *BookHotelRoom) error { return fmt.Errorf("some error") }, ), ) // ... After running the command processor with the handler, you can send a command and wait for a reply: reply, err := requestreply.SendWithReply[requestreply.NoResult]( context.Background(), ts.CommandBus, ts.RequestReplyBackend, &BookHotelRoom{ID: "1"}, ) // ... fmt.Println(reply.Error) // it's equal to `fmt.Errorf("some error")` You can also send a result from the handler: err := commandProcessor.AddHandlers( requestreply.NewCommandHandlerWithResult[PayForRoom, PayForRoomResult]( "pay_for_room", ts.RequestReplyBackend, func(ctx context.Context, cmd *PayForRoom) (PayForRoomResult, error) { return PayForRoomResult{PaymentReference: "1234"}, nil }, ), ) // ... After running the command processor with the handler, you can send a command and wait for a reply: reply, err := requestreply.SendWithReply[requestreply.NoResult]( context.Background(), ts.CommandBus, ts.RequestReplyBackend, &TestCommand{ID: "1"}, ) // ... fmt.Println(reply.Result.PaymentReference) // it's equal to "1234" fmt.Println(reply.Error) // it's nil We have also used generic types here, so your code is type-safe. By default, we don’t recommend using the request-reply pattern because it introduces coupling between services. Usually, using RPC is a better choice. But if you have a good argument for using request-reply (for example, you want to ensure that the request is processed, even if there was a transient error, and you wish to return a response to the customer), we’ve got you covered. Slog adapter Watermill now supports a logger backed by the new slog API. watermill-sql: v2 released We discovered a low chance of losing messages in watermill-sql in some specific cases. This is due to how transactions work — if you’re interested in the details, see this blog post. We had to change the default schema and the public API to fix this, so this is a major version bump. We recommend using watermill-sql v2 from now on. Note the minimal PostgreSQL version to use it is now 13. Migrating the existing database schema would be pretty complicated. Instead, we recommend using new tables. If you use watermill-sql just for the outbox pattern and there is one topic you care about, you can rename it. For example, if you’ve used the topic events-to-forward, you can rename it to events-to-forward-v2 while bumping watermill-sql to v2. This will create new tables with the new schema. If you use more topics, you can use the GenerateMessagesTableName option to customize the table name. The default prefix is watermill_. You can change it to something else, like watermill_v2_, while bumping watermill-sql to v2. GenerateMessagesTableName: func(topic string) string{ return fmt.Sprintf("watermill_v2_%s", topic) } Remember to pass this option to both the schema and the offsets schema! Usually, these tables are used just for delivering the messages and are not needed after it’s done. But you need to ensure all messages are processed before switching to new tables. In high-traffic environments, you may need to run both versions of watermill-sql for the migration period. Amazon SQS/SNS — looking for testers! There’s some work going on within PRs of watermill-amazonsqs (thank you, @Mariscal6!). If you’re familiar with this Pub/Sub or are running a project using it, we’d love to hear your feedback! Live Event Recording You can watch the release party recording on YouTube.

After almost three years since the last stable release, Watermill v1.2 is finally out! If you’re new here, Watermill is our open-source library for building event-driven applications in Go, the easy way. A lot has happened since v1.1. We’ve been running Watermill on production in many projects, using it in new ways and adding features to support new use cases. We also received a lot of feedback and contributions from the community. Watermill grew to over 5.2k stars on GitHub and 39 contributors, counting just the main repository. We’ve added 4 new official Pub/Subs, and there are many unofficial integrations. Celebrating the Launch Meet the Gopher at our live event! To celebrate the release, we plan to host a live event on March 1st at 5pm UTC. Join us on YouTube. The planned agenda is as follows: Watermill 1.2 Showcase. Using Consumer Groups in Watermill - a live demo with examples. Announcing our Event-Driven Training. Coming next to Watermill (generics!). A chance to win a unique crotchet Gopher. What’s new? You can find the full list of changes on the releases page. It’s long and includes many bug fixes and minor improvements, often from outside contributors (thank you, everyone!). Below are highlights of the bigger changes. The Forwarder Component One of the best use cases for Watermill is publishing messages in transactions (AKA the outbox pattern). I wrote about it before in When an SQL database makes a great Pub/Sub featuring a Watermill example. The Forwarder component provides a universal mechanism for this behavior: forwarding messages from one Pub/Sub to another (e.g., from SQL to Kafka). It’s a high-level API on top of the Router, so it’s even easier to forward messages. Forwarder has its own documentation entry and a complete example. It was contributed by Grzegorz Burzyński. The FanIn Component We already had the FanOut component before. It takes a single stream of messages and re-publishes them on a few topics. FanIn does the opposite: it takes a collection of topics and merges the messages from them into a single stream. FanIn has its own documentation entry. It was contributed by Michał Sokołowski. I’m thrilled to see it’s possible to write such components on top of existing structures. It shows that Watermill’s abstractions follow the UNIX philosophy of modular components that work well together. New Supported Pub/Subs Firestore - a NoSQL document database from Google. Contributed by Grzegorz Burzyński and Filip Borkiewicz. Bolt - an embedded key/value database for Go. Contributed by Filip Borkiewicz. NATS Jetstream - NATS’s built-in distributed persistence system. We released v2.0.0 in the existing NATS repository. Contributed by Alex Ullrich. Redis Stream - Redis-based Pub/Sub. Contributed by Hao-Ming Hsu. New Examples Server-Sent Events example SSE is our preferred way of pushing updates from the server. It’s simpler than WebSockets and plays nicely with Watermill’s abstractions. See the example on the Watermill repository. Exactly-once delivery example Exactly-once delivery is not possible in most setups, but there’s one specific scenario where you can achieve it. See the example on the Watermill repository. Adding handlers while the Router is running It’s now possible to add handlers to a running Router. Previously, all handlers had to be set up before starting it. New CI We moved the CI from CircleCI to GitHub Actions. It’s better integrated with GitHub’s PRs, and the configuration is simpler. All Pub/Subs now use the workflows defined in the main repository. All repositories now use golangci-lint for static analysis. Awesome Watermill We’ve added the Awesome Watermill page with a list of unofficial libraries. Please add a PR if you know a repository that should appear there! Watermill Contributors Without contributors, Watermill wouldn’t be where it is today. Writing a Pub/Sub integration is an especially high-impact project. To support the authors, we now keep a dedicated private channel on our discord server where they can reach us directly. The authors of Pub/Sub integrations keep write access to the repository after it’s transferred to the ThreeDotsLabs organization. Let us know if you want to join this community and have your Pub/Sub integration officially supported. Support As always, if something’s not clear, check out watermill.io. If you can’t find the answer there, join our discord server and ask your questions on the #watermill channel. If you’d like to get some open-source experience and contribute to Watermill, we’ve prepared a list of good first issues.

Did you have a situation when you lost a ton of time finding a Go library for your need? In theory, you can check lists like Awesome Go or make a choice based on GitHub stars. But Awesome Go contains over 2600 libraries, and popularity is not always the best indicator of library quality. I often thought that it would be great to have a place where I could find just the best and battle-tested libraries I could use in my project. Because we didn’t find such a place with Miłosz, we decided to create it. Based on our experience leading multiple Go teams and working on various projects, including complex financial, health, and security, we will recommend tools that could work well for different projects. In addition to providing a list of libraries, we also want to show you some non-obvious uses for those tools and libraries. However, it’s important to note that most of these tools can be misused. We’ve included some common anti-patterns to help you avoid making those mistakes. Note This list is intended to be opinionated. We only wanted to include libraries we used on real production systems. Thanks to that, we recommend just libraries that we are 100% sure about. Unfortunately, our day is limited to 24 hours, so checking all available libraries is impossible. If you know of any libraries we should include on this list, please let us know in the comments! We will continue to update the list with new findings over time. Table of Contents HTTP Routers Echo [GitHub] [Docs] [Examples] chi [GitHub] [Docs] [Examples] Middlewares Serving static content OpenAPI Generating Go server and clients deepmap/oapi-codegen [GitHub] [Docs] [Example] Bonus: Client for JavaScript/TypeScript openapi-generator-cli [GitHub] [Docs] Alternative types of communication gRPC protoc [Docs] Messaging Watermill [GitHub] [Docs] [Examples] Database SQL sqlx [GitHub] [Docs] sqlc [GitHub] [Docs] [Examples] SQLBoiler [GitHub] [Docs] [Examples] Migrations sql-migrate [GitHub] [Docs] goose [GitHub] [Docs] Observability Logging Logrus [GitHub] [Docs] zap [GitHub] [Docs] Metrics and tracing opencensus-go [GitHub] [Docs] Configuration Env variables caarlos0/env [GitHub] [Docs] Multi-format configuration koanf [GitHub] [Docs] Building CLI Building CLI libraries urfave/cli [GitHub] [Docs] [Examples] Testing Assertions testify [GitHub] [Docs] go-cmp [GitHub] [Docs] [Examples 1] [Examples 2] gofakeit [GitHub] [Docs] Mocking Writing mocks by hand Misc Extra types support google/uuid [GitHub] [Docs] oklog/ulid [GitHub] [Docs] shopspring/decimal [GitHub] [Docs] Errors hashicorp/go-multierror [GitHub] [Docs] Useful tools Misc samber/lo [GitHub] [Docs] Task [GitHub] [Docs] Live code reloading reflex [GitHub] [Docs] [Example] Linter golangci-lint [GitHub] [Docs] go-cleanarch [GitHub] [Docs] Formatters go fmt goimports [Docs] gofumpt [GitHub] [Docs] Example projects DDD & Clean Architecture Wild Workouts Go DDD Example application [GitHub] General purpose Modern Go Application by Márk Sági-Kazár [GitHub] Summary HTTP Routers As I mentioned in my previous article, it’s generally better to use libraries instead of frameworks for long-term projects. One of the most fundamental components of any service is an HTTP router. While it’s technically possible to build an application without one by using the standard library’s http package, its routing capabilities are limited. Using a dedicated router will make your life much easier. ❌ Anti-pattern: Frameworks in Go If the library you consider using impacts how you write your domain models, it’s probably a framework, not a router. We recommend using lightweight routers instead. Learn more about the risks of using the framework in my previous article. By design, router functionality is limited to routing requests to a proper handler. All non-standard functionalities like CORS, CSRF, error handling, HTTP logging, and authorization (that frameworks usually provide) are provided by reusable middlewares. I recommend some in the section on middlewares. I use one of two router libraries in most projects: Echo or chi. Both of them are great routers, with different characteristics. They work perfectly with OpenAPI code generation. ✅ Echo [GitHub] [Docs] [Examples] Compared to chi, Echo does offer a custom *http.Request handler signature. Some people may find it a downside, but I think it helps to write less error-prone HTTP handlers. If you have been writing Go for a while, you probably made this mistake at least once: func someHandler(w http.ResponseWriter, r *http.Request) { err := foo() if err != nil { w.WriteHeader(http.StatusBadRequest) // you forgot the return here, bar() will be executed } bar() } Echo makes you return an error: func someHandler(c echo.Context) error { err := foo() if err != nil { return err } bar() return c.NoContent(http.StatusNoContent) } The advantage of Echo is the ability to define a custom error handler. It’s not possible to do it in the same way with chi. For detailed usage and examples, please check Echo docs. ✅ chi [GitHub] [Docs] [Examples] Compared to Echo, chi’s handler functions are compatible with the standard library. For some people, it may be an upside; for some, it may be a downside – you should make your own judgment. What chi does better than Echo is the format of defining routes and grouping. It gives you better control over middleware per path or sub-path. r.Route("/articles", func(r chi.Router) { r.With(paginate).Get("/", ListArticles) r.Post("/", CreateArticle) // POST /articles r.Get("/search", SearchArticles) // GET /articles/search r.Route("/{articleID}", func(r chi.Router) { r.Use(ArticleCtx) // Load the *Article on the request context r.Get("/", GetArticle) // GET /articles/123 r.Put("/", UpdateArticle) // PUT /articles/123 r.Delete("/", DeleteArticle) // DELETE /articles/123 }) // GET /articles/whats-up r.With(ArticleCtx).Get("/{articleSlug:[a-z-]+}", GetArticle) }) Full source: github.com/go-chi/chi/_examples/rest/main.go Full source ❌ Anti-pattern: You should not choose tools based just on benchmarks Some developers choose libraries based on the benchmark results. It’s a risky approach because extreme performance optimizations lead to worse API and limited functionalities set. In most cases, performance differences are negligible in real-life use cases. Even if, for some applications, it may make a difference, for most applications, it doesn’t matter that much. Making just one extra database query or up-scaling a service can make a much more significant difference in performance. If performance is not absolutely critical for you, you should prefer other characteristics, like the ease of use and number of features. Middlewares HTTP middlewares can give you functionalities like CORS, CSRF, error handling, HTTP logging, authorization, etc. Echo and chi provide their set of middlewares: Echo middlewares chi middlewares Echo middlewares have a different interface, so they can’t be used in chi. Generally speaking, all standard-library compatible middlewares are compatible with chi and echo. To use standard library-compatible middleware with echo, you need to call echo.WrapMiddleware: package main import ( "github.com/go-chi/chi/v5/middleware" "github.com/labstack/echo/v4" ) // echo version func main() { e := echo.New() // You can use a middleware from chi with echo. e.Use( echo.WrapMiddleware(middleware.BasicAuth("realm", map[string]string{ "admin": "password", })), ) e.Logger.Fatal(e.Start(":8080")) } If none of them provides the middleware you are looking for, you can check the Awesome Go list. All of them will be compatible with chi, Echo, and servers built just with the standard library. You can also write your own middleware. Check example middlewares for inspiration! Serving static content You don’t need any library to serve static content in Go. Since Go 1.16, you can easily embed static files into your Go binary. Here’s how to do it for Echo and chi: package main import ( "embed" "net/http" "github.com/go-chi/chi" "github.com/go-chi/chi/v5" "github.com/labstack/echo/v4" ) // your static files should be in the static/ directory, for example static/index.html, static/main.js etc. // //go:embed static var staticFs embed.FS // chi version func main() { r := chi.NewRouter() r.Handle("/static/*", http.StripPrefix("/", http.FileServer(http.FS(staticFs)))) log.Fatal(http.ListenAndServe(":8080", r)) } // echo version func main() { e := echo.New() e.GET("/static/*", echo.WrapHandler(http.StripPrefix("/", http.FileServer(http.FS(staticFs))))) e.Logger.Fatal(e.Start(":8080")) } After running the server, assets will be available under http://localhost:8080/static/index.html, http://localhost:8080/static/main.js etc. ❌ Anti-pattern: Do not use no-name libraries for trivial functionalities Do you remember the leftpad JavaScript library? It was 11 lines of code adding padding on the left side of a string. Some day, the author decided to remove that library. It wouldn’t be a big problem if it wasn’t a dependency of thousands of projects, including Node and Babel. Serving static content from your web server is one of such trivial functionalities. Are you experienced engineer who wants to learn Go basics? You don't become an engineer by watching videos. Learn Go hands-on by building real projects. OpenAPI Nobody likes to maintain API contracts manually. It’s annoying and counterproductive to keep multiple boring JSON’s up-to-date. OpenAPI solves this problem with a JavaScript HTTP client and Go HTTP server generated from the provided specification. This is how an example specification looks like. If you didn’t work with OpenAPI before, you can read more details in my previous article. Here, I focus on tools that we recommend for OpenAPI spec-generated code. Generating Go server and clients We do not recommend using the official OpenAPI generator for the Go code. We recommend the oapi-codegen tool instead because of the higher quality of the generated code. It also has more functionalities. ❌ Anti-pattern: Don't try to generate OpenAPI spec from Go code There are tools that can generate an OpenAPI spec from Go code. We don’t recommend using them. The entire OpenAPI specification is very rich, and it will be hard to generate everything from Go code. It’s likely that you will need to add something to the OpenAPI spec at some point, and it may be impossible to do it from the Go code. It’s much easier to generate it the other way around: Go code from OpenAPI spec. ✅ deepmap/oapi-codegen [GitHub] [Docs] [Example] oapi-codegen is a great tool that doesn’t just generate models but also the entire router definition, header’s validation, and proper parameters parsing. It works with chi and Echo. To generate a server, run the following: oapi-codegen -generate types -o "<OUTPUT DIR>/openapi_types.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml" oapi-codegen -generate <TYPE> -o "<OUTPUT DIR>/openapi_api.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml" Where <TYPE> for chi should be chi-server, and for Echo just server. To generate clients: oapi-codegen -generate types -o "<OUTPUT DIR>/$service/openapi_types.gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml" oapi-codegen -generate client -o "<OUTPUT DIR>/$service/openapi_client_gen.go" -package "<GO PACKAGE>" "api/openapi/service.yml" Don’t forget to change <GO PACKAGE> to the desired Go package name and <OUTPUT DIR> to the desired output dir. 😉 Your job on the server side is just to implement the ServerInterface interface, like: // ServerInterface represents all server handlers. type ServerInterface interface { // (GET /trainer/calendar) GetTrainerAvailableHours(w http.ResponseWriter, r *http.Request, params GetTrainerAvailableHoursParams) // (PUT /trainer/calendar/make-hour-available) MakeHourAvailable(w http.ResponseWriter, r *http.Request) // (PUT /trainer/calendar/make-hour-unavailable) MakeHourUnavailable(w http.ResponseWriter, r *http.Request) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/ports/openapi_api.gen.go Full source You can see it in action in the Wild Workouts project. Bonus: Client for JavaScript/TypeScript Even if it’s the list of recommended Go libraries, you may need to generate code for the browser. ✅ openapi-generator-cli [GitHub] [Docs] In that case, we also recommend a non-official library instead of the official one. In contrast to oapi-codegen, openapi-generator-cli is a Java tool. To avoid any JVM-related issues, we recommend generating clients using Docker: docker run --rm --env "JAVA_OPTS=-Dlog.level=error" -v "${PWD}:/local" \ "openapitools/openapi-generator-cli:v6.2.1" generate \ -i "/local/api/openapi/service.yml" \ -g javascript \ -o "/local/web/src/clients/service" It assumes that the spec is available locally in api/openapi/service.yml. You can use openapi-generator-cli for TypeScript and other languages as well. Alternative types of communication gRPC gRPC is a technology that can help you with building robust, internal communication between your services (but not only!). I already described in detail why it’s worth using gRPC for internal communication and how to do it. I’ll not repeat it here and will focus on the tooling you need. With gRPC, you have little choice for generating server and client: you should use official tooling. The good news is that you don’t need anything more because it does its job! ✅ protoc [Docs] To generate Go code from .proto files, you need to install protoc and protoc Go Plugin. A list of supported types can be found in Protocol Buffers Version 3 Language Specification. More complex built-in types like Timestamp can be found in Well-Known Types list. Messaging ✅ Watermill [GitHub] [Docs] [Examples] About four years ago, when working on one of our projects, we found that there is no library that can simplify building message-driven or event-driven applications easily. To make our lives easier, we decided to write a library that will allow us to write event-driven code as easily as writing HTTP services. This is how Watermill was born. Today, Watermill is one of the most popular Go libraries with almost 5k GitHub stars, +35 contributors, and 10 officially supported Pub/Subs. Usually, message-broker libraries are very low-level. With Watermill, publishing messages may be as simple as: publisher.Publish("example.topic", msg) And subscribing like: messages, err := subscriber.Subscribe(ctx, "example.topic") if err != nil { panic(err) } for msg := range messages { fmt.Printf("received message: %s, payload: %s\n", msg.UUID, string(msg.Payload)) msg.Ack() } Compared to using just the message broker’s library, Watermill provides support for some higher level functionalities like middlewares, CQRS support, or event-forwarder component (that can be used to stream your messages from an SQL database to the message broker). Today, Watermill officially supports Kafka, GCP Pub/Sub, NATS, RabbitMQ message brokers (Pub/Subs). It can also listen to and emit events as HTTP hooks, from databases like MySQL/Postgres, BoltDB or Firestore. It can also work with in-memory Go-channel based Pub/Sub. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Database SQL There is no golden hammer solution for interacting with SQL databases. The reason is simple: it depends greatly on what kind of data you store. In some projects, data models are relatively simple. In some, they are very complex. Because of that, I have two libraries to recommend. You should choose one of them based on the requirements of your project. For projects with straightforward data models, you should check sqlx. For a bit more complex, you should look at SQLBoiler. ✅ Tactic: Using ORM I hear more and more that using ORM is not a good idea. I understand a reason for such thinking: many people are hurt by the improper use of ORMs. It’s like with a knife: I have an extremally sharp Japanese knife without which I can’t imagine cooking. On another side, I need to be very careful with using it. But that fact doesn’t make this knife a bad tool! If you are using it properly, it’s making your life much easier. It’s the same situation with ORMs. Writing queries by hand may be time-consuming and error-prone when your models are complex. ORMs were invented to solve that problem. If you have a bad experience using ORMs, you should check Things to know about DRY article. The tactics presented in that article will help you to avoid all common problems with ORMs. ❌ Anti-pattern: Avoid weakly typed ORMs Most ORMs depend heavily on reflection and interface{}/any. The type system is one of the biggest strengths of Go. It helps you build applications efficiently. Resigning from strict typing makes your application more error-prone. ✅ sqlx [GitHub] [Docs] The standard library’s database/sql package is rather a low-level one. sqlx provides a more convenient and powerful API to work with databases. It includes helper functions for common tasks like inserting and querying data and support for more advanced features like prepared statements and transactions. sqlx also has more advanced support for data unmarshaling (for example to structs, lists of structs, json data, etc.). As a nice bonus, sqlx’s interface is compatible with interfaces from database/sql. But even if sqlx is a great library, it works well for relatively simple database models. At some level of complexity, you should consider migration to an ORM. ✅ sqlc [GitHub] [Docs] [Examples] sqlc is now our default choice for new projects. Like SQLBoiler, it generates type-safe Go code, but from your SQL queries rather than just the schema. You write standard SQL, and sqlc generates fully type-safe Go code for your queries. The key advantages of sqlc: Query-first approach: You write SQL queries directly, and sqlc generates Go functions for them. No new query DSL to learn. Compile-time safety: Catches SQL errors before runtime, including column mismatches and type errors. Excellent PostgreSQL support: First-class support for PostgreSQL features like CTEs, window functions, and arrays. Active development: The project receives regular updates and has a growing community. Here’s an example workflow. -- queries/users.sql -- name: GetUser :one SELECT id, name, email FROM users WHERE id = $1; -- name: ListUsers :many SELECT id, name, email FROM users ORDER BY name; sqlc generates type-safe Go functions you can call directly: user, err := queries.GetUser(ctx, userID) users, err := queries.ListUsers(ctx) sqlc supports PostgreSQL, MySQL, and SQLite. ✅ SQLBoiler [GitHub] [Docs] [Examples] Note Update (2026): SQLBoiler is now in maintenance mode. It still receives updates and works well in existing projects, but we recommend sqlc for new projects. SQLBoiler is an ORM that generates Go models from your database schema. At first, how you define SQLBoiler models may surprise you. Most ORMs generate the database schema out of your Go models. SQLBoiler does the opposite: it generates Go models from your database schema. This approach has multiple advantages. One of the most important features is stricter typing than other libraries. Thanks to that, many checks are done during compilation. You don’t need to depend on a ton of reflection and magic struct tags. In most cases, as long as the code compiles, it will work correctly. Generating code from the database schema helps with migration from an existing database because you don’t need to re-write DB models: SQLBoiler generates them for you. So if you start with sqlx and move to SQLBoiler later, the migration should be pretty easy. SQLBoiler supports PostgreSQL, MySQL, MSSQLServer 2012+, SQLite3, and CockroachDB. ❌ Anti-pattern: Using database models in the API responses As long as you’re not writing a stupid simple CRUD application (and the chances are you’re not), you should not couple your database models with the API responses. At some point, requirements will force you to return data in a format different from the format you have in the database. Instead of trying to follow DRY at all costs, it’s time to split your models. You can read more on this in “Business Applications in Go: Things to know about DRY” article and “Common Anti-Patterns in Go Web Applications”. Migrations SQLBoiler and sqlx don’t provide out-of-the-box support for migrations. It’s okay because you are not forced to use any particular solution. In my recent projects, I used both sql-migrate and goose, and I was happy about them. ✅ sql-migrate [GitHub] [Docs] ✅ goose [GitHub] [Docs] We like sql-migrate and goose because of their simplicity and flexibility. sql-migrate and goose can be executed as CLI tools and as part of your service. I like to embed it into the binary of the service. Thanks to that, the migration is executed when the service starts, and it keeps the setup simple. It’s also much less complex to run. For example, sql-migrate with go:embed: // migrations/run.go package migrations import ( "database/sql" "embed" migrate "github.com/rubenv/sql-migrate" ) //go:embed * var migrationsFiles embed.FS func Run(postgresConn string) error { db, err := sql.Open("postgres", postgresConn) if err != nil { return err } migrations := &migrate.EmbedFileSystemMigrationSource{ FileSystem: migrationsFiles, Root: ".", } if _, err := migrate.Exec(db, "postgres", migrations, migrate.Up); err != nil { return err } return nil } Put your migrations in migrations/, for example: migrations/1_init.sql, migrations/2_alter_some_table.sql, etc. Then run Run in your main. Observability Logging The standard library’s logger doesn’t provide essential features like log levels and output formatting. For logging, we can recommend two libraries: Logrus and zap. In contrast to zap, Logrus provides a bit nicer user API, but zap is faster. You can check detailed benchmarks in zap’s readme. ❌ Anti-pattern: You should not choose tools based just on benchmarks Some developers tend to choose libraries based on the benchmark results. It’s a risky approach because extreme performance optimizations lead to worse API and limited functionalities set. In most cases, performance differences are negligible in real-life use cases. Even if, for some applications, it may make a difference, for most applications, it doesn’t matter that much. Making just one extra database query or up-scaling a service can make a much more significant difference in performance. If performance is not absolutely critical for you, you should prefer other characteristics, like the ease of use and number of features. ✅ Logrus [GitHub] [Docs] ✅ zap [GitHub] [Docs] Metrics and tracing ✅ opencensus-go [GitHub] [Docs] OpenCensus Go is a library that helps you add metrics and tracing to your endpoints or database queries. The integration uses middleware/decorator patterns, and it doesn’t require a lot of custom code. It supports HTTP endpoints, gRPC endpoints, SQL databases, MongoDB, etc. You can export traces and metrics to Prometheus, OpenZipkin, GCP Stackdriver Monitoring, Jaeger, AWS X-Ray, Datadog, Graphite, Honeycomb, or New Relic. Configuration Go’s standard library doesn’t support much more configuration options than the flag package. Even if it’s enough for simple CLI tools, you may need a bit more for building services. Env variables ✅ caarlos0/env [GitHub] [Docs] This library should provide everything you need for configuration for most applications. Compared to the standard library, it does support loading envs to structs and setting env defaults. It helps to save a lot of boilerplate for bigger configurations. It also supports embedded structs, so you can compose bigger a configuration from independent components. ✅ Tactic: Use env variables for your services configuration For most applications, environment variables should be good enough as configuration. Configuration is where you should keep secrets and things that differ between environments. If your configuration is massive and does not change often, it may be worth hardcoding it instead. It’s much more pragmatic than having tens of never-changing configuration options. Multi-format configuration ✅ koanf [GitHub] [Docs] Koanf is an excellent tool if your project requires multiple configuration formats. It’s often the case when you write tools that are used externally (for example, CLI tools). This is my most recent finding. Compared to other more popular libraries, koanf just does loading multi-format configuration right. Bonus points for a nice abstraction that allows extending parsing and loading. Koanf does support the most important configuration formats, like json, yaml, dotenv, env vars, or hcl. They can be loaded from the filesystem, flags, and multiple external sources like s3, vault, etcd, or consul. Building CLI Building CLI libraries ✅ urfave/cli [GitHub] [Docs] [Examples] We like urfave/cli because of its simple interface and extensibility. We used it in multiple projects without any issues. Compared to other alternatives, it offers a big-enough feature set while keeping the library lightweight. Testing Assertions ✅ testify [GitHub] [Docs] testify became the standard assertion library, and I’ve seen it in every project I worked on. It provides asserts for the most common cases and also some more complex. One of testify’s key features are friendly messages for all failed asserts. It makes writing and debugging tests much faster. The library provides two ways of asserting: assert from github.com/stretchr/testify/assert - the test continues after failure. You should use it when you want to see multiple errors (not just the first one). Works when called in a goroutine. require from github.com/stretchr/testify/require - the test is interrupted after failure. You should use it when some critical condition was not met and continuing doesn’t make any sense (for example: storing to database failed). Doesn’t work when called in a goroutine. Some example asserts: Equal - good enough in most cases Eventually - useful for asserting asynchronous conditions ElementsMatch - useful for unsorted slices WithinDuration - useful when comparing time that is not exactly equal ErrorIs JSONEq Panics ✅ Tactic: Use assert messages just if it is really needed I’ve seen people who obsessively write messages for all failed assert. For example: assert.Equal(t, 123, 321, "123 is not equal to 321") will give output: Error: Not equal: expected: 123 actual : 321 Test: TestEqual Messages: 123 is not equal to 321 As you can see, the message doesn’t add anything more than testify would figure out. It can even be harmful because with time, you will need to spend a lot of time to keep the message up to date. In most cases, the message provided by testify will be good enough. If the test fails, the person who sees the failure will navigate to this test and will understand the reason from the surrounding code. ✅ Tactic: Do not write basic asserts by hand Many people advocate for writing all asserts by hand. It won’t give you much advantage in the end. testify is also very smart in showing the difference between the expected and actual value. For example: assert.Equal(t, []byte("foo bar baz"), []byte("foo bar 42")) prints: Error: Not equal: expected: []byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x62, 0x61, 0x7a} actual : []byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x34, 0x32} Diff: --- Expected +++ Actual @@ -1,3 +1,3 @@ -([]uint8) (len=11) { - 00000000 66 6f 6f 20 62 61 72 20 62 61 7a |foo bar baz| +([]uint8) (len=10) { + 00000000 66 6f 6f 20 62 61 72 20 34 32 |foo bar 42| } Test: TestEqual --- FAIL: TestEqual (0.00s) Expected :[]byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x62, 0x61, 0x7a} Actual :[]byte{0x66, 0x6f, 0x6f, 0x20, 0x62, 0x61, 0x72, 0x20, 0x34, 0x32} It makes no sense to reinvent the wheel and write it from scratch. ❌ Anti-pattern: Do not use test suites from testify Testify is an excellent library for assertions, but we don’t recommend its test suites. They don’t support parallel sub-tests. They may be fine for unit tests, but for integration/API/E2E tests it’s a deal-breaker. The standard library can achieve most of the functionalities provided by testify’s test suites. You can see specific examples in this article on testing microservices. ✅ go-cmp [GitHub] [Docs] [Examples 1] [Examples 2] Sometimes, you must assert a complex struct in your tests skipping some fields. Or the struct contains fields that should be compared in a specific way. Or you need to ignore the slice order or time delta. It’s where go-cmp can help you! import ( "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" ) diff := cmp.Diff( want, got, // FieldToIgnore and AnotherFieldToIgnore will be ignored in SomeStruct cmpopts.IgnoreFields(SomeStruct{}, "FieldToIgnore", "AnotherFieldToIgnore"), // when comparing time, truncate it to one second // can be written for any type opt := cmp.Comparer(func(x, y time.Time) bool { return x.Truncate(time.Second).Equal(y.Truncate(time.Second)) }) // sort all []int cmpopts.SortSlices(func(x, y int) bool { return x < y })) ) // cmp returns diff if two objects are different // to check if objects are equal, you can assert if the diff is empty assert.Empty(t, diff) To see the list of all available options, I recommend checking the godoc of cmp and cmpopts package. go-cmp can also be used outside of tests, but be careful – it’s another tool that, used irresponsibly, may hurt your project. ✅ gofakeit [GitHub] [Docs] If you need more realistic data for your tests, gofakeit helps. Mocking Writing mocks by hand Initially, I recommended one popular mocking tool here. But after some thinking, we decided that the tool is not good enough to recommend. Instead, consider an alternative mocking strategy 👇 ✅ Tactic: Consider writing mocks by hand Even if it sounds like a waste of time, writing mocks yourself may be good enough. Objectively speaking, writing them by hand doesn’t require much more code and time. As a bonus, it gives you much more flexibility. This is how an example mock can look like: type BalanceUpdate struct { UserID string AmountChange int } type UserServiceMock struct { BalanceUpdates []BalanceUpdate balanceUpdatesLock sync.Mutex } func (u *UserServiceMock) UpdateTrainingBalance(ctx context.Context, userID string, amountChange int) error { u.balanceUpdatesLock.Lock() defer u.balanceUpdatesLock.Unlock() u.BalanceUpdates = append(u.BalanceUpdates, BalanceUpdate{userID, amountChange}) return nil } It took me literally 1 minute to write it. ✅ Tactic: Keep your interfaces small, so it's easier to mock them It’s hard to mock complex types by hand. But if your interface is so complex you can’t write a mock for it, you should reconsider if it needs to be that big. Using mocking libraries obfuscates the real problem. Try to simplify the type that you are mocking. Maybe the interface segregation principle will help? It could be possible to split this type into multiple smaller types. It will not only simplify your mocks but will improve your codebase. Misc Extra types support ✅ google/uuid [GitHub] [Docs] This library generates UUIDs. ✅ oklog/ulid [GitHub] [Docs] UUIDs may be slow to store at a larger scale in relational databases. A solution may be using Universally Unique Lexicographically Sortable Identifier: ULIDs. ULIDs are compatible with UUIDs, are unique enough for large scale, and have shorter string representation (Crockford’s base32). ULIDs are lexicographically sortable, thanks to what building indexes should be much faster. Note It’s worth mentioning that UUID v6, v7, and v8 will also be lexicographically sortable. But its spec is still draft when during the release of the article. If you want to try UUID v6 or v7, you can check github.com/gofrs/uuid which does already implement them. ✅ shopspring/decimal [GitHub] [Docs] Go doesn’t have built-in support for decimals. shopspring/decimal does the job. We have used this library for a couple of years to build a large financial system. ✅ Tactic: Use decimals for monetary values Floats are not designed to accurately store decimal numbers. For example: fmt.Printf("%.16f", 12.1+0.03) > Output: 12.1300000000000008 To make sure your money calculations are correct (and you are not losing or getting extra cents in calculations), we recommend using a decimal type. It’s also a good idea to use the string representation of decimals instead of floats in transport (in events, API requests and responses, etc.). Errors ✅ hashicorp/go-multierror [GitHub] [Docs] Did you ever need to handle an error while you were handling another error? hashicorp/go-multierror is here to help you! It’s also helpful if an operation can return multiple errors, and you don’t want to return just the first one (for example, validation). Example use cases: func validate() { var resultErr error if err := validateFoo(); err != nil { resultErr = multierror.Append(resultErr, err) } if err := validateBar(); err != nil { resultErr = multierror.Append(resultErr, err) } return resultErr } or func ExecuteStuff() error { if err := makeStuff(); err != nil { if cleanupErr := cleanup(); cleanupErr != nil { err = multierror.Append(err, cleanupErr) } return err } return nil } Note: Go 1.20 will introduce errors.Join function. After release of Go 1.20 you should consider using it instead. Useful tools Misc ✅ samber/lo [GitHub] [Docs] Lodash-style Go library based on Go 1.18+ Generics. It may be especially useful for you if you are coming to Go from Python and missing some basic slice/map functions. Some functions that I’m using the most: Filter Map Keys Values Find Max Must 😈 please use it just for tests or if you really have a good reason Even if some may find it “non-idiomatic”, I find it useful in some cases. It’s similar to using an ORM – as long as such libraries are used responsibly and don’t obfuscate code, they are useful. So if you find yourself writing code like: lo.Map( lo.Filter(someSlice, func(v SomeType, _ int) bool { return v.IsSpecial }), func(t SomeType, _ int) string { return t.SpecialName() }, ) …it’s just better to convert it to a simple, more readable loop. 😉 ✅ Task [GitHub] [Docs] Task is not really a Go library, but it’s a tool written in Go that may be useful for your projects. It’s an excellent alternative to Makefile. The most important features that it offers are: Parallel tasks execution (supported by task dependencies) Preventing unnecessary work Loading .env Dynamic variables Forwarding CLI arguments Templating It’s a must-have for each of my new projects. Live code reloading ✅ reflex [GitHub] [Docs] [Example] Go doesn’t provide code live-reloading out of the box. But you can achieve it quickly with the reflex library. Some time ago, Miłosz wrote an article that shows how to create a local environment with Docker and reflex. Linter ✅ golangci-lint [GitHub] [Docs] golangci-lint is a linter that aggregates multiple linters and runs them in parallel and does it very fast. Here’s an example configuration that we use in our projects. ✅ go-cleanarch [GitHub] [Docs] If you use Clean/Hexagonal Architecture in your project, you can use this linter to ensure that The Dependency Inversion Rule and interaction between packages are kept. Formatters ✅ go fmt The standard formatter provided by Go toolchain. ✅ goimports [Docs] Goimports does all that go fmt does, but it also sorts imports of your Go files. It’s one of the tools that you will see widely adopted in most Go projects. Not everybody knows, but you can also separately group your local imports with the -local flag. goimports -local "github.com/ThreeDotsLabs/some-repository" -l -w . ✅ gofumpt [GitHub] [Docs] Just for the biggest formatting freaks! Does all that go fmt and goimports do and more! Personally, I like gofumpt’s formatting decisions. Example projects DDD & Clean Architecture ✅ Wild Workouts Go DDD Example application [GitHub] Wild Workouts is an example Go DDD project that we created to show how to build Go applications that are easy to develop, maintain, and fun to work with, It shows a project developed over time and with complex problems to solve. In contrast to other example projects, it was not blindly copied from other languages. This is the way how we build our services daily. Highly recommended if you are looking for patterns that will allow you to build more complex projects! ❌ Anti-pattern: Low-quality example repositories Avoid projects that look like over-engineered copies from other programming languages. People who write such “DDD” projects often just read a couple of articles about it without understanding it correctly and without using it in real-life projects. If you see DDD/Clean Architecture examples without encapsulated domain models (with public fields) and json tags: run! It’s definitely not DDD nor Clean Architecture. General purpose ✅ Modern Go Application by Márk Sági-Kazár [GitHub] Another example repository that we can recommend. It doesn’t cover patterns like DDD or Clean Architecture but emphasizes infrastructure beats like observability. Summary Should we check some library that is not listed here? Please let us know in the comments!

While writing this blog and leading Go teams for a couple of years, the most common question I heard from beginners was “What framework should I use?”. One of the worst things you can do in Go is follow an approach from other programming languages. Other languages have established, “default” frameworks. Java has Spring, Python has Django and Flask, Ruby has Rails, C# has ASP.NET, Node has Express, and PHP has Symfony and Laravel. Go is different: there is no default framework. What’s even more interesting, many suggest you shouldn’t use a framework at all. Are they insane? Building Go service from libraries may give a feeling of building Frankenstein’s Monster The Philosophy of Go Go frameworks exist, but none provides a feature set like frameworks from other languages. This won’t change soon. You may think that it’s because the Go ecosystem is younger. But there is a more important factor. Go is built around the Unix Philosophy that says: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams because that is a universal interface. This philosophy originated from Ken Thompson, the designer of the B programming language (a precursor of C) and also… Go! In practice, the Unix philosophy favors building small independent pieces of software that do one thing well rather than big chunks that do everything. You may have seen it in your terminal. For example: cat example.txt | sort | uniq. cat reads the text from the file, sort sorts the lines, and uniq removes the duplicates. All commands are independent and do one thing. It comes directly from the Unix philosophy. Thanks to such design, you can independently develop much smaller, autonomous commands. In Go, the Unix philosophy is visible in the standard library. The best examples are the most widespread interfaces: the io.Reader and the io.Writer. The best libraries follow this philosophy as well. Frameworks are designed against this philosophy. Often, they try to cover all possible use cases within one framework. They are not designed to work with other tools and often can’t be reused. It means that it’s not possible to transfer development efforts to other non-compatible frameworks. If the framework adoption is low (or it just dies), all the effort is lost. Are you experienced engineer who wants to learn Go basics? You don't become an engineer by watching videos. Learn Go hands-on by building real projects. What is important for you Every technical decision has tradeoffs. You need to choose tradeoffs that make more sense for you and your project. Some approaches make sense when you work on a one-weekend proof of concept and throw it away (you will, right?). In that case, the most critical factor is how quick you can finish it. But if you work on a project that will last a long time and you build it with multiple people, the impact of that decision is tremendous. For most projects, the most important parameters are: how fast can you start the project, how fast will you be able to develop this project in the long term, how flexible for future changes the project will be (it’s strongly connected with the previous point). Let’s start assessing our decisions. Time saving One of the biggest promises given by frameworks is time saving. You run one command, and you get a fully functional project. Frameworks usually provide an opinionated structure of the project, and it helps if you don’t know how to do it. But as with most other technical decisions, it’s not free. With time, when the project grows, you’ll quickly hit the framework’s wall of conventions and limitations. The framework’s author requirements were likely different from yours. The decision taken by the framework creator may work nicely for simple CRUD applications but may not handle more complex scenarios. It’s easy to quickly lose all your time saved on the project bootstrap just to fight with one framework limitation. Over time, it can lead to a lot of frustration in your team. A couple of years ago, I worked in a company that initially started with one Go framework (I’ll skip the name of the framework). The company was growing and creating new services. With time, we started to feel more pain when we wanted to support more complex use cases. It was also a source of severe bugs. Unfortunately, getting rid of the framework wasn’t easy. At one point, some framework components became unmaintained and incompatible with the rest of the ecosystem. We were forced to get rid of it. The framework has already become very tightly coupled to the entire system. Removing it from tens of services was a non-trivial task. It required a cross-team initiative that took multiple person-months and a few incidents to get rid of. Even if the project was successful in the end, I didn’t see it in such a way. All the time spent could be used in a much better way if someone took a different decision earlier. The entire project would not be needed. It’s not surprising that many companies suffer from the lack of trust in the development team. It’s a great example how such a small decision can become a costly rescue project after a couple of years. Maintainability of the project Measuring the maintainability of projects is a controversial topic — it’s hard to compare two projects. Some people say that frameworks are great and they don’t feel the pain of using them. For others, frameworks can be the biggest nightmare in the long term. Some projects are much more challenging than others. Many people think that fighting with the framework is just a part of the job. That’s why it’s hard to objectively measure the impact of frameworks on the project’s maintainability. Fortunately, we can help ourselves understand it with a bit of science. More precisely, with the excellent Accelerate: The Science of Lean Software and DevOps book based on scientific research. The book is focused on finding the characteristics of the best and worst-performing teams. What’s important for us, one of the most significant factors for good performance is loosely-coupled architecture. The teams I led often asked me how to know if our architecture is loosely coupled. One of the simplest ways is ensuring that it’s easy to replace or delete parts of your application. If it’s hard to remove parts of your application, your application is tightly coupled. Touching one thing stars a domino effect of changes in different places. Why is the loosely coupled architecture that important? Let’s admit it. We are humans, and even after the best research, we make mistakes. When you choose the wrong framework or library, it should be easy to replace it without rewriting the entire project. If we want to save time, we should think what helps in the long term, not only at the beginning of the project. Consider a scenario when you want to remove the framework entirely. Will it require a lot of code rewrites? Can it be done on multiple services independently? If not, you could put some effort to separate the framework from your core logic. But it requires sacrificing the “time-saving” it gives in the first place. The alternative? Building services without a framework You may feel that building your services without a framework will take ages. Especially if you are coming from other programming languages. I understand that. I had the same feeling a couple of years ago when I started writing in Go. It was an unjustified fear. Not using a framework doesn’t mean that you will need to build everything yourself. There are many proven libraries that provide the functionality you need. You need to put a bit more effort into research. You’re reading this, so you’re already doing that! Even a couple hours of research are nothing in the entire project’s lifetime. You’ll also get that time back very soon thanks to the flexibility it gives you. What should you do if you decide not to use a framework? The biggest blocker at the beginning may be how you build a service. The easiest way is to start by putting everything into one file. You can start simple, defer some decisions, and evolve your project with time. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! It’s helpful to have example projects you can use as a reference. You can take a look at the project I used for my Let’s build event-driven application in 15 minutes with Watermill presentation at GoRemoteFest – github.com/roblaszczak/goremotefest-livecoding. This example needed just two external libraries to make it functional. Feel free to clone this repository and adopt to your needs. I’m sure this example doesn’t have all libraries required by your project. To help you with more specific use-cases, we wrote a next article with a list of Go libraries you can use to build your Go services. We have been using them for a couple of years. We are explaining why we use those libraries and how to know if a similar library is good or bad. It's practical to change parts of your project without killing it. When your project becomes more complex, and you already know how your libraries work together, you can start to refactor it. In the end, you may not need most of the framework features that seemed critical. Thanks to that, you can end up with a simpler project and less research. If you are looking for a reference on how more advanced projects can look like, you should check Wild Workouts – our fully functional example Go project. We released a +200 pages free e-book describing how we built that application. Summary Deciding how you build services is not where you should take shortcuts. Making a wrong decision may have a very negative impact on your time in the long term. It negatively affects your team’s velocity and, more importantly, morale. After making a wrong decision, you can quickly fall into the sunk cost fallacy trap. Instead of becoming heroes who solve problems they created, we should avoid creating them. After that article’s lecture, you should know each way’s tradeoffs and implications. You can now make a responsible decision. I hope this article will help avoid at least one company a couple-man-months painful refactoring project. Do you have any framework horror stories (even from a different programming language)? Let us know in the comments!

Cohesion is part of the low coupling, high cohesion principle that’s supposed to keep your code maintainable. While low coupling means few dependencies, high cohesion roughly translates to single responsibility. Highly cohesive code (a module or a function) is focused on a single purpose. Low cohesion means it does many unrelated things. I’ve written about coupling in the previous article on anti-patterns. Here are some tips on increasing cohesion in Go applications using the recently released generics. The Application Logic There’s a place in your project where your application logic lives. We call it the application layer and split it into commands and queries, CQRS style. Perhaps you call it handlers, services, controllers, or use cases. Hopefully, it’s not mixed with implementation details, like HTTP handlers. Note If you don’t keep your logic separate yet, check our articles on Clean Architecture and CQRS. Whatever you call this code, it’s probably not only logic. Some common operations need to happen for each request your application handles. They’re often related to observability, like logging, metrics, and tracing. Other examples are retries, timeouts, or an authorization mechanism. Code sections in a typical handler. func (h SubscribeHandler) Handle(ctx context.Context, cmd Subscribe) (err error) { start := time.Now() h.logger.Println("Subscribing to newsletter", cmd) defer func() { end := time.Since(start) h.metricsClient.Inc("commands.subscribe.duration", int(end.Seconds())) if err == nil { h.metricsClient.Inc("commands.subscribe.success", 1) h.logger.Println("Subscribed to newsletter") } else { h.metricsClient.Inc("commands.subscribe.failure", 1) h.logger.Println("Failed subscribing to newsletter:", err) } }() } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/01-low-cohesion/subscribe.go Full source Keeping such operations close to the logic decreases the cohesion. It mixes different concepts. Eventually, it gets hard to grasp the logic itself. Even if you move some code to helper functions, you’ll end up passing many arguments and handling errors. You risk that someone copy-pastes the entire header of the function and forgets to change one of the fields. Updating the supporting code requires changing the logic parts. The User Context An interesting scenario is when you need to handle requests differently depending on the caller. A typical example is checking the user’s permissions or attributes. You need to pass some kind of user context to the handler — a token, a user ID, or a complete User object. The logic can’t proceed without it. A middleware transforming an authorization token into a user context. user, err := UserFromContext(ctx) if err != nil { return err } if !user.Active { return errors.New("the user's account is not active") } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/01-low-cohesion/subscribe.go Full source But often, there are a few entry points to do the same action. For example, in a system subscribing users to a newsletter, we could have: A public HTTP API that uses the user context coming from a session token. An internal gRPC API that other services call. An admin CLI tool also uses it. A message handler that reacts to an event that already happened (a user signed up who agreed to receive a newsletter). A migration that runs on the service’s startup. It backfills subscriptions that weren’t there because of a bug. Except for the first entry point, all others miss a proper user context the logic could use Multiple entry points to the same command handler. You must manually craft and pass a fake user context for entry points other than the HTTP API. fakeUser := User{ ID: "1", // Missing ID in the context, let's assume it's the root user making changes Active: true, } ctx = ContextWithUser(ctx, fakeUser) err := handler.Handle(ctx, cmd) if err != nil { return err } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/01-low-cohesion/main.go Full source Even if your application exposes only a public HTTP API, testing this function isn’t a good experience. You need to prepare a fake user every time you call the handler or mock it somewhere in the dependencies. func TestSubscribe(t *testing.T) { logger := log.New(os.Stdout, "", log.LstdFlags) metricsClient := nopMetricsClient{} handler := NewSubscribeHandler(logger, metricsClient) user := User{ ID: "1000", Active: true, } ctx := ContextWithUser(context.Background(), user) cmd := Subscribe{ Email: "user@example.com", NewsletterID: "product-news", } err := handler.Handle(ctx, cmd) if err != nil { t.Fatal(err) } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/01-low-cohesion/subscribe_test.go Full source ❌ Anti-pattern: Low cohesion Don’t mix your application’s logic with other code directly in a handler. The Decorator Pattern Decorators are one of the least controversial design patterns. After all, HTTP middlewares are widely used, and it’s the same idea. You can think of decorators as application-level middlewares. Decorators let you separate things that don’t belong together, but you can still explicitly use them where needed. You wrap the logic with other operations. The logic handler doesn’t know what decorates it. Decorators wrapping the command handler. type subscribeAuthorizationDecorator struct { base SubscribeHandler } func (d subscribeAuthorizationDecorator) Handle(ctx context.Context, cmd Subscribe) error { user, err := UserFromContext(ctx) if err != nil { return err } if !user.Active { return errors.New("the user's account is not active") } return d.base.Handle(ctx, cmd) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/02-decorators/decorators.go Full source The logic part stays super short now. func (h subscribeHandler) Handle(ctx context.Context, cmd Subscribe) error { // Subscribe the user to the newsletter return nil } You can pick which decorators to use depending on the use case. You can set them up in constructors. Different decorator sets for different purposes. func NewAuthorizedSubscribeHandler(logger Logger, metricsClient MetricsClient) SubscribeHandler { return subscribeLoggingDecorator{ base: subscribeMetricsDecorator{ base: subscribeAuthorizationDecorator{ base: subscribeHandler{}, }, client: metricsClient, }, logger: logger, } } func NewUnauthorizedSubscribeHandler(logger Logger, metricsClient MetricsClient) SubscribeHandler { return subscribeLoggingDecorator{ base: subscribeMetricsDecorator{ base: subscribeHandler{}, client: metricsClient, }, logger: logger, } } func NewSubscribeHandler() SubscribeHandler { return subscribeHandler{} } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/02-decorators/subscribe.go Full source You can write a unit test for the exact same code but without things not related directly to the logic. You don’t need to mock the logger or the metrics client. Tests become shorter and easier to follow. func TestSubscribe(t *testing.T) { handler := NewSubscribeHandler() cmd := Subscribe{ Email: "user@example.com", NewsletterID: "product-news", } err := handler.Handle(context.Background(), cmd) if err != nil { t.Fatal(err) } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/02-decorators/subscribe_test.go Full source Note You still need to test the authorization mechanism. I suggest doing this on the component tests level. You can learn more about different test types in Microservices test architecture. ✅ Tactic: Decorators Wrap your logic with decorators. Choose which to use depending on the use case. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Generic decorators Splitting code into decorators comes with some boilerplate. Usually, we like to deal with boilerplate using code generation. For example, you can easily generate API or storage models. We used to generate our decorators as well. But with Go 1.18 out there, we can now replace them with generic decorators. Because all our command handlers follow the same convention, the command is the only generic type we need. (A command is a struct containing all inputs for the command handler.) type CommandHandler[C any] interface { Handle(ctx context.Context, cmd C) error } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/03-generics/subscribe.go Full source type authorizationDecorator[C any] struct { base CommandHandler[C] } func (d authorizationDecorator[C]) Handle(ctx context.Context, cmd C) error { user, err := UserFromContext(ctx) if err != nil { return err } if !user.Active { return errors.New("the user's account is not active") } return d.base.Handle(ctx, cmd) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/03-generics/decorators.go Full source You can use the generic decorators with any other command handler. You could even move them to a separate library. func NewUnauthorizedSubscribeHandler(logger Logger, metricsClient MetricsClient) CommandHandler[Subscribe] { return loggingDecorator[Subscribe]{ base: metricsDecorator[Subscribe]{ base: SubscribeHandler{}, client: metricsClient, }, logger: logger, } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/03-generics/subscribe.go Full source ✅ Tactic: Generic decorators Use generics to reduce boilerplate. One Caveat In most cases in Go, it makes sense to accept interfaces and return structs. But to make wrapping possible, you need to return an interface from constructors. type SubscribeHandler interface { Handle(ctx context.Context, cmd Subscribe) error } type subscribeHandler struct{} func NewSubscribeHandler() SubscribeHandler { return subscribeHandler{} } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/03-cohesion/02-decorators/subscribe.go Full source The downside is that it’s more challenging to navigate the project. You can’t just click the Handle method to land in the definition. You should be fine if you use a solid IDE that shows what implements an interface. Also, be aware that it can surprise someone trying to follow the general rule. It’s OK to break the rules but understand (and be ready to explain) why you make an exception. Another thing to watch out for is how you wrap your decorators. Outer decorators get executed first, and sometimes the order makes a difference. For example, you want to keep logs and metrics even if the authorization fails. The Complete Example Check the complete source code on the anti-patterns repository. It features logging and metrics decorators ready to use with minimal changes. Wild Workouts In parallel, Robert introduced decorators to Wild Workouts, the example project you might know from our previous posts. Check the commit to see what this looks like in a real application. There are two helper functions not covered in this post: ApplyCommandDecorators and ApplyQueryDecorators. Robert also added support for running Wild Workouts locally on M1 CPUs. If you didn’t have a chance to do it yet, it might be a good moment to play with the project.

Note Hello! Please give Krzysztof a warm welcome in the first guest post on our blog. 🎉 We’ve been working with Krzysztof for the past two years, and we’re excited to share his work here. Miłosz & Robert We all struggle with software architecture diagrams, don’t we? Have you ever wondered why? If you ask yourself that question, why maintenance of up-to-date and detailed software architecture diagrams is so painful, you will come up with a long list of valid answers. Our software changes all the time. We update it daily, starting with a simple change of names, and ending with nuke-refactoring, reshaping the entire application. Each of those changes requires careful alignment of the software architecture diagrams. Most likely, you have also thought that drawing diagrams with even the most convenient tools is time-consuming. It requires a lot of effort. Those diagrams notoriously get out of control, storing outdated naming and already-dropped modules while missing their initial readable and well-organised structure. All of that makes this particularly frustrating. If an ideal world ever existed… Everyone appreciates a well-structured diagram when joining a new, unexplored project. Even better if it is an easy-to-navigate, map-like C4 model diagram. Auto-generated C4 diagram example If this concept doesn’t sound familiar to you, there is a simple idea behind it. C4 model introduces four layers of software architecture visualisation: Context, Containers, Components, and Code. Depending on the information you are looking for, you can drill down to the specific part of the implementation, the same way you look into geographical maps. When searching for a particular address, you usually use a city or even a district map, not a map of the whole country. Consequently, when you check how single service modules talk to each other, you will find it in the diagram of that service Components layer. When you look for services and runtimes interactions, you will go to the Containers one. Do you get the idea? For more information, go to C4 model site. In my opinion, this is the best approach proposed so far. Is maintaining detailed and up-to-date diagrams of your applications’ architecture worth all that effort? And does it have to be done manually? What if we could… Automate it! If you work within a Go ecosystem, you are used to code generation. What if we also could generate software diagrams from the code? Moreover, what if we could do this automatically, within continuous integration pipelines? Driven by this idea, I wrote a library that generates diagrams out of code with just a single configuration file. Let me show you how to use it. Let’s start simple To demonstrate the feature, I selected a repository that most readers are familiar with – wild-workouts-go-ddd-example application. There are a couple of services that we could try to generate diagrams for. Let’s start with the trainer one. At the moment, the library provides a set of components to code your own diagram auto-generation command. I did that in the wild-workouts-go-ddd-example repository. Within a separate tools/c4 directory I created a simple command file. func main() { s, err := scraper.NewScraperFromConfigFile("scraper.yml") if err != nil { ... } app := trainerService.NewApplication(context.Background()) structure := s.Scrape(app) v, err := view.NewViewFromConfigFile("scraper.yml") if err != nil { ... } outFile, err := os.Create("out/view.plantuml") if err != nil { ... } defer outFile.Close() err = v.RenderStructureTo(structure, outFile) if err != nil { ... } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/main.go Full source In this command, I use a couple of components provided by the library. The first one, the scraper, crawls down any Go structure and collects visited components following provided configuration and scraping rules. I started with a minimal configuration file: configuration: pkgs: - "github.com" rules: - name_regexp: ".*" pkg_regexps: - ".*" Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/scraper.yml Full source In the configuration, I instruct the scraper to crawl all components from any package starting with a github.com prefix and then to collect the ones that match the rule with the provided name and package regular expressions. The expression .* would match any string, so basically, I want to capture everything. For those of you who are YAML haters, the very same configuration may be done programmatically. Moving next, I need to instantiate the service application context as the “content” to scrape. Robert and Miłosz had implemented a convenient constructor, which I invoke here. Then I can pass it to the scraper and get the returned Structure model containing all the collected data. I instantiate another library component to render the acquired structure into the output file – the view. It consumes the same configuration file as the scraper, enhanced with just a single line of the view definition. view: title: Trainer service components Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/view.yml Full source …and I render it. The output file is of type *.plantuml. I need to render it into a *.png image file with the plantuml CLI tool. plantuml out/view.plantuml Here is what I got: Raw C4 diagram Getting fancier So far, so good. But there is at least a couple of issues here. It works, but it is all flat! All I can see in the diagram is a map of components without any details, description, or color-coding. Which of the components belong to the application or domain layer? Also, we do not know much about the infrastructure around. Is there any database being used? If yes, which of the components are using it? Let’s iterate. I need to extend the scraping range by adding the package with the firestore client. The application uses the firestore database, so I included its client in the diagram. configuration: pkgs: - "github.com" - "cloud.google.com/go/firestore" Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/scraper.yml Full source Then, I create several rules that would instruct the scraper on interpreting the components from specified packages. With rules, I could tag components, add descriptions, define name functions, and much more. Here are a few examples of rules where I define application, domain, and database components. rules: - name_regexp: ".*" pkg_regexps: - ".*/app/command.*" component: description: "application command" tags: - APP - name_regexp: ".*" pkg_regexps: - ".*/domain/.*" component: description: "domain component" tags: - DOMAIN - name_regexp: ".*Client$" pkg_regexps: - "cloud.google.com/go/firestore$" component: name: "Firestore" description: "firestore client" tags: - DB Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/scraper.yml Full source Finally, I can add some view styles and assign those to the tags defined above. view: title: Trainer service components line_color: 000000ff styles: - id: APP background_color: 1a4577ff font_color: ffffffff border_color: 000000ff - id: DOMAIN background_color: ffffffff font_color: 000000ff border_color: 000000ff - id: DB background_color: c8c8c8ff font_color: 000000ff border_color: 000000ff shape: database Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/tools/c4/view.yml Full source When I reran the scraper, I got the following result. Doesn’t it look much better now? Styled C4 diagram Want more? If you want to have a deeper look into this example, go to wild-workouts-go-ddd-example application repo and check the implementation where I generate complete diagrams for all of the microservices. You can also learn more about the library itself. Visit go-structurizr repo and dive deep into the documentation and provided examples. Having a pretty powerful way to auto-generate diagrams out of the golang code, every time you change your code, you can just regenerate diagrams reusing the very same configuration! Finally, it may sound tempting to fully automate it as part of our regular CI pipelines. If you want to know how I do that, let me know in the comments.

Enums are a crucial part of web applications. Go doesn’t support them out of the box, but there are ways to emulate them. Many obvious solutions are far from ideal. Here are some ideas we use that make enums safer by design. iota Go lets you enumerate things with iota. const ( Guest = iota Member Moderator Admin ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/01-iota/role/role.go Full source While Go is explicit, iota seems relatively obscure. If you sort the const group any other way, you introduce side effects. In the example above, you could accidentally create moderators instead of members. You can explicitly assign a number to each value to avoid this issue, but it makes iota obsolete. iota works well for flags represented by powers of two. const ( Guest = 1 << iota // 1 Member // 2 Moderator // 4 Admin // 8 ) // ... user.Roles = Member | Moderator // 6 Bitmasks are efficient and sometimes helpful. However, it’s a different use case than enums in most web applications. Often, you’ll be fine storing all roles in a list. It’ll also be more readable. The main issue with iota is it works on integers that don’t guard against passing invalid values. func CreateUser(role int) error { fmt.Println("Creating user with role", role) return nil } func main() { err := CreateUser(-1) if err != nil { fmt.Println(err) } err = CreateUser(42) if err != nil { fmt.Println(err) } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/01-iota/main.go Full source CreateUser will happily accept -1 or 42 even though there are no corresponding roles. Of course, we could validate this within the function. But we use a language with strong types, so let’s take advantage of it. In our application’s context, a user role is much more than a vague number. ❌ Anti-pattern: Integer enums Don’t use iota-based integers to represent enums that are not sequential numbers or flags. We could introduce a type to improve the solution. type Role uint const ( Guest Role = iota Member Moderator Admin ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/02-typed-iota/role/role.go Full source It looks better, but it’s still possible to pass any arbitrary integer in place of Role. The Go compiler doesn’t help us here. func CreateUser(role Role) error { fmt.Println("Creating user with role", role) return nil } func main () { err := CreateUser(0) if err != nil { fmt.Println(err) } err = CreateUser(role.Role(42)) if err != nil { fmt.Println(err) } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/02-typed-iota/main.go Full source The type is an improvement over a bare integer, but it’s still an illusion. It doesn’t give us any guarantee that the role is valid. Sentinel values Because iota starts from 0, the Guest is also the Role’s zero-value. It makes it hard to detect if the role is empty or someone passed a Guest value. You can avoid this by counting from 1. Even better, keep an explicit sentinel value you can compare and can’t mistake for an actual role. const ( Unknown Role = iota Guest Member Moderator Admin ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/02-typed-iota/role/role.go Full source func CreateUser(r role.Role) error { if r == role.Unknown { return errors.New("no role provided") } fmt.Println("Creating user with role", r) return nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/02-typed-iota/main.go Full source ✅ Tactic: Explicit sentinels Keep an explicit variable for the enum’s zero-value. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Slugs Enums seem to be about sequential integers, but it’s rarely a valid representation. In web applications, we use enums to group possible variants of some type. They don’t map well to numbers. It’s hard to understand the context when you see a 3 in the API response, a database table, or logs. You have to check the source or the outdated documentation to know what it’s about. String slugs are more meaningful than integers in most scenarios. Wherever you see it, a moderator is obvious. Since iota doesn’t help us anyway, we can as well use human-readable strings. type Role string const ( Unknown Role = "" Guest Role = "guest" Member Role = "member" Moderator Role = "moderator" Admin Role = "admin" ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/03-slugs/role/role.go Full source ✅ Tactic: Slugs Use string values instead of integers. Avoid whitespace for easier parsing and logging. Use camelCase, snake_case, or kebab-case. Slug are especially useful for error codes. An error response like {"error": "user-not-found"} is obvious in contrast to {"error": 4102}. However, the type can still hold any arbitrary string. err = CreateUser("super-admin") if err != nil { fmt.Println(err) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/03-slugs/main.go Full source Struct-based Enums The final iteration uses structs. It lets us work with code secure by design. We don’t need to check if the passed value is correct. type Role struct { slug string } func (r Role) String() string { return r.slug } var ( Unknown = Role{""} Guest = Role{"guest"} Member = Role{"member"} Moderator = Role{"moderator"} Admin = Role{"admin"} ) Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/04-structs/role/role.go Full source Because the slug field is unexported, it’s not possible to fill it from outside the package. The only invalid role you can construct is the empty one: Role{}. We can add a constructor to create a valid role based on a slug: func FromString(s string) (Role, error) { switch s { case Guest.slug: return Guest, nil case Member.slug: return Member, nil case Moderator.slug: return Moderator, nil case Admin.slug: return Admin, nil } return Unknown, errors.New("unknown role: " + s) } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/02-enums/04-structs/role/role.go Full source ✅ Tactic: Struct-based enums Encapsulate enums in structs for extra compile-time safety. This approach is perfect when you work with business logic. Keeping structures always in a valid state in memory makes your code easier to work with and understand. It’s enough to check if the enum type isn’t empty, and you’re sure it’s a correct value. There’s one potential issue with this approach. Structs cannot be constants in Go, so it’s possible to overwrite the global variables like this: roles.Guest = role.Admin There is no sane reason to do this, though. You’re much more likely to pass an invalid integer by accident. Another disadvantage is you have to make updates in two places: the enums list and the constructor. However, it’s easy to spot, even if you miss it at first. How do you do it? When there’s a missing Go feature, we tend to develop our ways of doing things. What I described is just one possible approach. Do you use another pattern to store your enums? Please share it in the comments. 🙂 For complete source code, see the anti-patterns repository.

At one point in my career, I was no longer excited about the software I was building. My favorite part of the job were low-level details and complex algorithms. After switching to user-facing applications, they were mostly gone. It seemed programming was about moving data from one place to another using existing libraries and tools. What I’ve learned so far about software wasn’t that useful anymore. Let’s face it: most web applications don’t solve tough technical challenges. They should correctly model the product and allow improving it faster than the competition. It seems boring at first, but then you realize supporting this goal is harder than it sounds. There’s an entirely different set of challenges. Even if they’re not as complex in the technical sense, solving them has a massive impact on the product and is deeply satisfying. The biggest challenge web apps face is not becoming an unmaintainable Big Ball of Mud. It slows you down and can put you out of business. Here’s why it happens in Go and how I’ve learned to avoid it. Loose Coupling Is the Key A big reason applications get hard to maintain is strong coupling. In tightly coupled applications, anything you touch has unexpected side effects. Each refactoring attempt uncovers new issues. Eventually, you decide it’s best to rewrite the project from scratch. In a fast-growing product, you can’t afford to freeze all development to re-do what you’ve built. You have no guarantee you’ll get everything right this time. In contrast, loosely coupled applications keep clear boundaries. They allow replacing a broken part without effect on the rest of the project. They’re easier to build and maintain. So why are they so rare? Microservices promised us loose coupling, but we’re after their hype era, and unmaintainable applications still exist. Sometimes it’s even worse, and we fall into the trap of distributed monoliths, dealing with the same issues as before with added network overhead. From a tightly coupled monolith to a distributed monolith. ❌ Anti-pattern: The Distributed Monolith Don’t split your application into microservices before you know the boundaries. Microservices don’t lower coupling because it’s not important how many times you split the application. What matters is how you connect the pieces. From a modular monolith to loosely-coupled microservices. ✅ Tactic: Loose Coupling Aim for loosely coupled modules. How you deploy them (as a modular monolith or microservices) is an implementation detail. DRY introduces coupling Strong coupling is common because we’re taught early about the “Don’t Repeat Yourself” (DRY) rule. Short rules are easy to remember, but it’s hard to capture all the details in three words. The Pragmatic Programmer offers a longer version: Every piece of knowledge must have a single, unambiguous, authoritative representation within a system. “Every piece of knowledge” is quite extreme. The answer to most programming dilemmas is “it depends”, and DRY is no different. When you make two things use a common abstraction, you introduce coupling. If you follow DRY too strictly, you add abstractions before they’re needed. Being DRY in Go Compared to other modern languages, Go is explicit and stripped from features. There’s not much syntactic sugar to hide complexity. We’re used to shortcuts, so at first, it’s hard to accept Go’s verbosity. It’s like we’ve developed an instinct to find “smarter” ways of writing code. The best example is error handling. If you have any experience writing Go, this snippet feels natural: if err != nil { return err } For newcomers, repeating these three lines over and over seems like breaking the DRY rule. They often look for a way to avoid this boilerplate, but it never ends well. Eventually, everyone accepts that’s how Go works. It makes you repeat yourself, but it’s not the duplication DRY tells you to avoid. A Single Model Couples Your Application There’s one feature in Go that introduces strong coupling and makes you think you’re following DRY. It’s using multiple tags in a single struct. It seems like a good idea because we often use similar models for different things. Here’s a popular way of keeping a single User model. type User struct { ID int `json:"id" gorm:"autoIncrement primaryKey"` FirstName string `json:"first_name" validate:"required_without=LastName"` LastName string `json:"last_name" validate:"required_without=FirstName"` DisplayName string `json:"display_name"` Email string `json:"email,omitempty" gorm:"-"` Emails []Email `json:"emails" validate:"required,dive" gorm:"constraint:OnDelete:CASCADE"` PasswordHash string `json:"-"` LastIP string `json:"-"` CreatedAt *time.Time `json:"-"` UpdatedAt *time.Time `json:"-"` } type Email struct { ID int `json:"-" gorm:"primaryKey"` Address string `json:"address" validate:"required,email" gorm:"size:256;uniqueIndex"` Primary bool `json:"primary"` UserID int `json:"-"` } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/01-tightly-coupled/internal/user.go Full source This approach takes a few lines of code and makes you maintain just a single structure. However, fitting everything in a single model requires many tricks. The API shouldn’t expose some fields, so they’re hidden using json:"-". Only one API endpoint uses the Email field, so the ORM skips it, and the omitempty tag hides it from regular JSON responses. Most importantly, this solution gives you one of the worst issues: strong coupling between the API, storage, and logic. When you want to update anything in the struct, you have no idea what else can change. You can break the API contract by changing the database schema or corrupt the stored data when updating validation rules. The more complex the model, the more issues you face. For example, the json tag means JSON, not HTTP. What happens when you introduce events that also marshal to JSON but with a bit different format than the API response? You will keep adding hacks to make it work. Eventually, your team avoids any changes to the structure because they don’t know what can break after you touch it. ❌ Anti-pattern: The Single Model Don’t give a single model more than one responsibility. Don’t use more than one tag per structure field. Duplication removes coupling The easiest way to reduce coupling is using separate models. We extract the API-specific part as HTTP models: type CreateUserRequest struct { FirstName string `json:"first_name" validate:"required_without=LastName"` LastName string `json:"last_name" validate:"required_without=FirstName"` Email string `json:"email" validate:"required,email"` } type UpdateUserRequest struct { FirstName *string `json:"first_name" validate:"required_without=LastName"` LastName *string `json:"last_name" validate:"required_without=FirstName"` } type UserResponse struct { ID int `json:"id"` FirstName string `json:"first_name"` LastName string `json:"last_name"` DisplayName string `json:"display_name"` Emails []EmailResponse `json:"emails"` } type EmailResponse struct { Address string `json:"address"` Primary bool `json:"primary"` } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/02-loosely-coupled/internal/http.go Full source And the database-related part as storage models: type UserDBModel struct { ID int `gorm:"column:id;primaryKey"` FirstName string `gorm:"column:first_name"` LastName string `gorm:"column:last_name"` Emails []EmailDBModel `gorm:"foreignKey:UserID;constraint:OnDelete:CASCADE"` PasswordHash string `gorm:"column:password_hash"` LastIP string `gorm:"column:last_ip"` CreatedAt *time.Time `gorm:"column:created_at"` UpdatedAt *time.Time `gorm:"column:updated_at"` } type EmailDBModel struct { ID int `gorm:"column:id;primaryKey"` Address string `gorm:"column:address;size:256;uniqueIndex"` Primary bool `gorm:"column:primary"` UserID int `gorm:"column:user_id"` } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/02-loosely-coupled/internal/db.go Full source At first, it seemed we would use the same “user” model everywhere. Now, it’s clear we’ve avoided duplication too early. The API and storage structures are similar but different enough to use separate models. In web applications, the views your API returns (read models) are not the same thing you store in the database (write models). The storage code should know nothing about the HTTP models, so we need to convert the structs. func userResponseFromDBModel(u UserDBModel) UserResponse { var emails []EmailResponse for _, e := range u.Emails { emails = append(emails, emailResponseFromDBModel(e)) } return UserResponse{ ID: u.ID, FirstName: u.FirstName, LastName: u.LastName, DisplayName: displayName(u.FirstName, u.LastName), Emails: emails, } } func emailResponseFromDBModel(e EmailDBModel) EmailResponse { return EmailResponse{ Address: e.Address, Primary: e.Primary, } } func userDBModelFromCreateRequest(r CreateUserRequest) UserDBModel { return UserDBModel{ FirstName: r.FirstName, LastName: r.LastName, Emails: []EmailDBModel{ { Address: r.Email, }, }, } } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/02-loosely-coupled/internal/http.go Full source That’s all the code you need: functions mapping one type to another. Writing such banal code may seem boring, but it’s essential for decoupling. It’s tempting to create a generic solution to map the structs, like marshaling or using reflect. Resist it. Writing the boilerplate takes less time and effort than debugging mapping edge cases. Plain functions are easy to understand for everyone in your team. Arcane converters will be hard to grasp, even for you, after some time. ✅ Tactic: One Model, One Responsibility. Aim for loose coupling by using separate models. Write plain and obvious functions to convert between them. If you’re afraid of too much duplication, consider the worst-case scenario. If you end up with a few more structures that stay the same as the application grows, you can merge them back into one. In contrast to tightly-coupled code, fixing duplicated code is trivial. Generate the Boilerplate If you worry about writing all this code by hand, there’s an idiomatic way to avoid it. Use libraries that generate the boilerplate for you. You can generate things like: HTTP models and routes out of the OpenAPI definition (oapi-codegen and other libraries). Database models and related code out of the SQL schema (sqlboiler and other ORMs). gRPC models out of ProtoBuf files. Generated code gives you strong types, so you no longer pass interface{} to generic functions. You keep compile-time checks and don’t need to write the code manually. Here’s how the generated models look. // PostUserRequest defines model for PostUserRequest. type PostUserRequest struct { // E-mail Email string `json:"email"` // First name FirstName string `json:"first_name"` // Last name LastName string `json:"last_name"` } // UserResponse defines model for UserResponse. type UserResponse struct { DisplayName string `json:"display_name"` Emails []EmailResponse `json:"emails"` FirstName string `json:"first_name"` Id int `json:"id"` LastName string `json:"last_name"` } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/03-loosely-coupled-generated/internal/http_types.go Full source type User struct { ID int64 `boil:"id" json:"id" toml:"id" yaml:"id"` FirstName string `boil:"first_name" json:"first_name" toml:"first_name" yaml:"first_name"` LastName string `boil:"last_name" json:"last_name" toml:"last_name" yaml:"last_name"` PasswordHash null.String `boil:"password_hash" json:"password_hash,omitempty" toml:"password_hash" yaml:"password_hash,omitempty"` LastIP null.String `boil:"last_ip" json:"last_ip,omitempty" toml:"last_ip" yaml:"last_ip,omitempty"` CreatedAt null.Time `boil:"created_at" json:"created_at,omitempty" toml:"created_at" yaml:"created_at,omitempty"` UpdatedAt null.Time `boil:"updated_at" json:"updated_at,omitempty" toml:"updated_at" yaml:"updated_at,omitempty"` R *userR `boil:"-" json:"-" toml:"-" yaml:"-"` L userL `boil:"-" json:"-" toml:"-" yaml:"-"` } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/03-loosely-coupled-generated/models/users.go Full source Sometimes, you may even want to write a code-generating tool. It’s not that hard, and the result is regular Go code that everyone can read and understand. The common alternative is reflect, which is terrible to grasp and debug. Of course, first, consider if the effort is worth it. In most cases, writing the code by hand will be quick enough. ✅ Tactic: Generate the Repetitive Parts Generated code gives you strong types and compile-time safety. Choose it over reflect. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Don’t overuse libraries Use the generated code only for what it’s supposed to do. You want to avoid writing the boilerplate by hand, but you should still keep a few dedicated models. Don’t end up with the single model anti-pattern. It’s easy to fall into this trap when you want to follow DRY. For example, both sqlc and sqlboiler projects generate code out of SQL queries. sqlc allows adding JSON tags to the generated models and even lets you choose between camelCase and snake_case. sqlboiler adds json, toml, and yaml tags to all models by default. It’s clear people use these models not only for storage. Looking through sqlc’s issues, I’ve found developers asking for even more flexibility, like renaming the generated fields or skipping some JSON fields entirely. Someone even mentions they Need some way to hide sensitive fields in REST API. All this encourages keeping a single model for many responsibilities. It lets you write less code, but always consider if the coupling is worth it. Similarly, watch out for magic hidden in struct tags. For example, consider the permissions model that gorm supports: type User struct { Name string `gorm:"<-:create"` // allow read and create Name string `gorm:"<-:update"` // allow read and update Name string `gorm:"<-"` // allow read and write (create and update) Name string `gorm:"<-:false"` // allow read, disable write permission Name string `gorm:"->"` // readonly (disable write permission unless it configured ) Name string `gorm:"->;<-:create"` // allow read and create Name string `gorm:"->:false;<-:create"` // createonly (disabled read from db) Name string `gorm:"-"` // ignore this field when write and read with struct } Full source: gorm.io/docs/models.html#Field-Level-Permission Full source You can also use quite complex comparisons using the validator library, like referencing other fields: type User { FirstName string `validate:"required_without=LastName"` LastName string `validate:"required_without=FirstName"` } It saves you a bit of time writing the code, but you give up the compile-time checks. It’s easy to make a typo in a struct tag, and it’s a risk using it for sensitive areas like validation and permissions. It’s also confusing to anyone not familiar with the library’s arcane syntax. I don’t mean to pick on the mentioned libraries. They all have their uses, but these examples show how we tend to take DRY to the extremes, so we don’t have to write more code. ❌ Anti-pattern: Choosing magic to save time writing code Don’t overuse the libraries to avoid verbosity. Avoid implicit tag names Most libraries don’t require the tags to be present and use field names by default. While refactoring the project, someone can rename a field, unaware they edit an API response or a database model. If there are no tags, this can break your API contract or even corrupt your storage. Always fill all the tags. Even if you have to type the same name twice, it’s not against DRY. ❌ Anti-pattern: Omitting Structure Tags Don’t skip struct tags if a library uses them. type Email struct { ID int `gorm:"primaryKey"` Address string `gorm:"size:256;uniqueIndex"` Primary bool UserID int } ✅ Tactic: Explicit Structure Tags Always fill the struct tags, even if the field names are the same. type Email struct { ID int `gorm:"column:id;primaryKey"` Address string `gorm:"column:address;size:256;uniqueIndex"` Primary bool `gorm:"column:primary"` UserID int `gorm:"column:user_id"` } Separate Logic from Implementation Details Decoupling the API from the storage and using generated models is a good start. But we still keep validation in the HTTP handlers. type createRequest struct { Email string `validate:"required,email"` FirstName string `validate:"required_without=LastName"` LastName string `validate:"required_without=FirstName"` } validate := validator.New() err = validate.Struct(createRequest(postUserRequest)) if err != nil { log.Println(err) w.WriteHeader(http.StatusBadRequest) return } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/03-loosely-coupled-generated/internal/http.go Full source Validation is just one part of the business logic you find in most web applications. Often, there will be more of it, like: showing fields only in particular cases, checking permissions, hiding fields depending on the role, calculating the price, taking action depending on a few factors. Mixing logic with implementation details (like keeping it in the HTTP handlers) is a quick way to deliver an MVP. But it also introduces the worst kind of technical debt. It’s why you get vendor lock-in and why you keep adding hacks to support new features. ❌ Anti-pattern: Mixing logic and details Don’t mix your application logic with implementation details. Business logic deserves its own layer. Changing the implementation (database engine, HTTP library, infrastructure, Pub/Sub, etc.) should be possible without any changes to the logic parts. You’re not doing this separation because you expect to change the database. It rarely happens. But the separation of concerns makes your code easy to understand and modify. You know what you’re changing, and there are no side effects. It’s harder to introduce bugs in the most crucial parts. To separate the application layer, we need to add additional models and mappings. type User struct { id int firstName string lastName string emails []Email } func NewUser(firstName string, lastName string, emailAddress string) (User, error) { if firstName == "" && lastName == "" { return User{}, ErrNameRequired } email, err := NewEmail(emailAddress, true) if err != nil { return User{}, err } return User{ firstName: firstName, lastName: lastName, emails: []Email{email}, }, nil } type Email struct { address string primary bool } func NewEmail(address string, primary bool) (Email, error) { if address == "" { return Email{}, ErrEmailRequired } // A naive validation to make the example short, but you get the idea if !strings.Contains(address, "@") { return Email{}, ErrInvalidEmail } return Email{ address: address, primary: primary, }, nil } Full source: github.com/ThreeDotsLabs/go-web-app-antipatterns/01-coupling/04-loosely-coupled-app-layer/internal/user.go Full source It’s the code I’d like to work with when I need to update the business logic. It’s boring, obvious, and I know exactly what’s going to change. We do the same thing when adding another API, like gRPC, or an external system, like a Pub/Sub. Each part uses separate models, and we use those from the application layer to map them. Because the application models hold all validations and other business rules, it makes no difference whether we use them from an HTTP or gRPC API. The API is just an entry point to the application. ✅ Tactic: Application Layer Dedicate a separate layer to your product’s most important code. The code snippets above come from the same repository and implement a classic “users” domain. All examples expose the same API and pass the same test suite. Here’s how they compare: Tightly Coupled Loosely Coupled Loosely Coupled Generated Loosely Coupled App Layer Coupling Strong Medium Medium Weak Boilerplate Manual Manual Generated Generated Lines of code 292 345 298 408 Generated code 0 0 2154 2154 The Standard Go Project Structure If you’ve seen the repository, you may be surprised there’s just a single package in each example. There’s no official directory structure in Go. You can find many “example microservice” or “REST boilerplate” repositories that propose how to split the packages. They usually have a well-designed directory structure. Some even mention they follow “Clean Architecture” or “Hexagonal Architecture”. The first thing I check is how the example keeps models. Most often, it uses structures with JSON and database tags combined. It’s an illusion: the packages look nicely separated on the outside, but in reality, the single model couples them. It’s common even for popular examples that newcomers use to learn. Ironically, the “Standard Go Project Structure” discussion continues in the community, while the single model anti-pattern is widespread. If types couple your application, no directory structure will change it. While you look at the example structures, keep in mind they could be designed for a different kind of application. No approach works equally well for an open-source infrastructure tool, a web application’s backend, and the standard library. The problem with package hierarchy is similar to splitting microservices. The important part is not how you split them but how they’re connected. When you focus on loose coupling, the structure becomes obvious. You separate the implementation details from the business logic. You group things that refer to each other and keep separate things that don’t. In the examples I’ve prepared, I could easily move HTTP-related and database-related code to separate packages. It would make the namespace less polluted. There’s already no coupling between the models, so it’s just a detail. ❌ Anti-pattern: Overthinking the directory structure Don’t start the project by separating directories. However you do it, it’s a convention. You’re unlikely to do it right before you’ve written any code. ✅ Tactic: Loosely Coupled Code The important part isn’t the directory structure but how packages and structures reference each other. Keeping it simple Let’s say you want to create a user with an ID field. The simplest approach can look like this: type User struct { ID string `validate:"required,len=32"` } func (u User) Validate() error { return validate.Struct(u) } This code works. However, you can’t tell if the struct is correct at any point. You rely on something to call the validation and handle the error correctly. Another approach follows the good old encapsulation. type User struct { id UserID } type UserID struct { id string } func NewUserID(id string) (UserID, error) { if id == "" { return UserID{}, ErrEmptyID } if len(id) != 32 { return UserID{}, ErrInvalidLength } return UserID{ id: id, }, nil } func (u UserID) String() string { return u.id } This snippet is more explicit and verbose. If you create a new UserID and receive no error, you’re sure it’s valid. Otherwise, you can easily map the error to a proper response specific to your API. Whatever approach you choose, you need to model the essential complexity of the user’s ID. From the pure implementation point of view, keeping the ID in a string is the simplest solution. Go is supposed to be simple, but it doesn’t mean you should use only primitive types. For complex behaviors, use code that reflects how the product works. Otherwise, you’ll end up with a simplified model of it. ❌ Anti-pattern: Over-simplification Don’t model complex behavior with trivial code. ✅ Tactic: Write obvious code Be explicit, even if it’s verbose. Use encapsulation to ensure your structs are always in a valid state. Note It’s possible to create an empty structure outside the package, even if all fields are unexported. It’s the only thing you have to check when accepting UserID as a parameter. You can either use if id == UserID{} or write a dedicated IsZero() method that does it. Starting with the database schema Let’s consider adding teams that users create and join. Following the relational approach, we would add a teams table and another one that joins it with users. Let’s call it membership. We already keep UserStorage, so it’s natural to add two more structs: TeamStorage and MembershipStorage. They expose CRUD methods for each table. A snippet adding a new team could look like this: func CreateTeam(teamName string, ownerID int) error { teamID, err := teamStorage.Create(teamName) if err != nil { return err } return membershipStorage.Create(teamID, ownerID, MemberRoleOwner) } This approach has one issue: we don’t create the team and the membership entry within a transaction. We might end up with a team without an owner assigned if things go wrong. The first solution that comes to mind is passing a transaction object between methods. func CreateTeam(teamName string, ownerID int) error { tx, err := db.Begin() if err != nil { return err } defer func() { if err == nil { err = tx.Commit() } else { rollbackErr := tx.Rollback() if rollbackErr != nil { log.Error("Rollback failed:", err) } } }() teamID, err := teamStorage.Create(tx, teamName) if err != nil { return err } return membershipStorage.Create(tx, teamID, ownerID, MemberRoleOwner) } However, this approach leaks the implementation details (transaction handling) to the logic layer. It pollutes a readable function with defer-based error handling. Here’s an exercise: consider how we would model this in a document database. For example, we could keep all Members inside the Team document. In this scenario, adding members would be done in TeamStorage. We wouldn’t need a separate MembershipStorage. Isn’t it weird that switching the database changes our assumption about the models? It’s now clear we’ve leaked the implementation details by introducing the “membership” concept. By saying “create a new membership,” we would only confuse our sales or customer service colleagues. It’s a major red flag when you start speaking a different language than the rest of the company. ❌ Anti-pattern: Starting with the database schema Don’t base your models on the database schema. You will end up exposing implementation details. TeamStorage stores teams, but it’s not about the teams SQL table. It’s about our product’s team concept. When you start modeling from the domain, you see actual behaviors instead of CRUD methods. You also notice the transaction boundaries. Everyone understands creating a team requires an owner, and we can expose a method for it. The method executes all queries within a transaction. teamStorage.Create(teamName, ownerID, MemberRoleOwner) Similarly, we could keep a method for joining a team. teamStorage.JoinTeam(teamID, memberID, MemberRoleGuest) The membership table is still there, but it’s an implementation detail hidden in the TeamStorage. ✅ Tactic: Start with the domain Your storage methods should follow the product’s behavior. Don’t leak transactions out of them. Your Web Application is not a CRUD Tutorials often feature “simple CRUDs”, so they seem to be the building block of any web application. It’s a myth. If all your product needs is a CRUD, you waste time and money writing it from scratch. Frameworks and no-code tools make it easy to bootstrap CRUDs, but we still pay developers to build custom software. Even GitHub’s Copilot won’t know how your product works besides the boilerplate. It’s the special rules and weird details that make your application different. It’s not some logic you sprinkle on top of the four CRUD operations. It’s the core of the product you sell. In the MVP stage, it’s tempting to start with a CRUD to build a working version quickly. But it’s like using a spreadsheet instead of dedicated software. You get similar results at first, but each new feature requires more hacks. ❌ Anti-pattern: Starting with a CRUD Don’t design your application around the idea of four CRUD operations. ✅ Tactic: Understand your domain Spend time to understand how your product works and model it in the code. Many of the tactics I’ve described are ideas behind well-known patterns: Single Responsibility Principle from SOLID (a single model responsible for one thing). Clean Architecture (loosely coupled packages, separating logic from details). CQRS (using different read models and write models). Some are even close to Domain-Driven Design: Value Objects (keeping structures always in a valid state). Aggregate and Repository (saving domain objects transactionally regardless of the number of database tables). Ubiquitous Language (using a language everyone understands). These patterns seem linked mostly to enterprise applications. But most of them are about straightforward core ideas, like the tactics from this article. They apply just as well in web applications, which often deal with complex business behaviors. You don’t need to read heavy books or copy how things work in other languages to follow these patterns. It’s possible to write idiomatic Go code together with battle-proven techniques. If you want to learn more about them, check out our free e-book. Note Let us know in the comments if you’d like to see more examples added to the anti-patterns repository and on what topics.

A couple of years ago, I worked in a SaaS company that suffered from probably all possible issues with software development. Code was so complex that adding simples changes could take months. All tasks and the scope of the project were defined by the project manager alone. Developers didn’t understand what problem they were solving. Without an understanding the customer’s expectations, many implemented functionalities were useless. The development team was also not able to propose better solutions. Even though we had microservices, introducing one change often required changes in most of the services. The architecture was so tightly coupled that we were not able to deploy these “microservices” independently. The business didn’t understand why adding “one button” may take two months. In the end, stakeholders didn’t trust the development team anymore. We all were very frustrated. But the situation was not hopeless. I was lucky enough to be a bit familiar with Domain-Driven Design. I was far from being an expert in that field at that time. But my knowledge was solid enough to help the company minimize and even eliminate a big part of mentioned problems. Some time has passed, and these problems are not gone in other companies. Even if the solution for these problems exists and is not arcane knowledge. People seem to not be aware of that. Maybe it’s because old techniques like GRASP (1997), SOLID (2000) or DDD (Domain-Driven Design) (2003) are often forgotten or considered obsolete? It reminds me of the situation that happened in the historical Dark Ages, when the ancient knowledge was forgotten. Similarly, we can use the old ideas. They’re still valid and can solve present-day issues, but they’re often ignored. It’s like we live in Software Dark Ages. Another similarity is focusing on the wrong things. In the historical Dark Ages, religion put down science. In Software Dark Ages, infrastructure is putting down important software design techniques. I’m not claiming that religion is not important. Spiritual life is super important, but not if you are suffering from hunger and illness. It’s the same case for the infrastructure. Having awesome Kubernetes cluster and most fancy microservices infrastructure will not help you if your software design sucks. Let me try to put another flowered and useless tool in the production... Software Dark Ages as a system problem Software Dark Ages is a very strong self-perpetuating system. You can’t fix the systemic problem without understanding the big picture. System thinking is a technique that helps to analyze such complex issues set. I used this technique to visualize the Software Dark Ages. The ultimate guide to creating the worst team ever.See the full version. You can see one big loop of things that accelerate each other. Without intervention, problems become bigger and bigger. How can Domain-Driven Design fix that? Unlike most famous programming gurus, we don’t want you to just believe in our story. We could just make it up. Fortunately, we can explain why it works by science. More precisely, with the excellent Accelerate: The Science of Lean Software and DevOps book based on scientific research. The research described in the book mentions the characteristics of the best, and worst-performing teams. One of the most critical factors is loosely coupled architecture. If you think that microservices may give you loosely coupled architecture – you are very wrong. I’ve seen microservices that are more coupled than a monolith multiple times. This is why we need something more than just infrastructure solutions. This is the time when Domain-Driven Design (DDD) comes into play. This key architectural property enables teams to easily test and deploy individual components or services even as the organization and the number of systems it operates grow—that is, it allows organizations to increase their productivity as they scale. (…) employing the latest whizzy microservices architecture deployed on containers is no guarantee of higher performance if you ignore these characteristics. (…) Architectural approaches that enable this strategy include the use of bounded contexts and APIs as a way to decouple large domains into smaller, more loosely coupled units, and the use of test doubles and virtualization as a way to test services or components in isolation. Accelerate: The Science of Lean Software and DevOps DDD doesn’t work Maybe you know someone who tried DDD, and it didn’t work for them? Maybe you worked with a person who didn’t understand it well, tried to force these techniques, and made everything too complex? Maybe you’ve seen on Twitter that some famous software engineer said that DDD doesn’t work? Maybe for you, it’s a legendary Holy Grail that someone claims work for them – but nobody has seen it yet. We know what we are doing... kinda Let’s not forget that we are living in the Software Dark Ages. There’s one problem with ideas from the previous epoch – there is a chance that some people may miss the initial point of DDD. That’s not surprising in the context of 2003 when DDD was proposed for the first time. DDD in 2003 It’s not easy to enter the DDD world. A lot of books and articles are missing the most important DDD points by oversimplifying them. They are also often explained in abstract examples that are detached from reality. It’s also not rare to see too long and too complex examples that are impossible to understand. Let me try to explain DDD in the simplest way. From the Dark Ages to the Renaissance DDD techniques can be split into two parts. Tactical and strategic patterns. Tactical Domain-Driven Design patterns are about how to implement the solution in the code. There is no rocket science in the Tactical DDD – it’s all about Object-Oriented Programming good practices. But before writing the code, you need to know what to implement. That’s where Strategic DDD patterns come into the game. Many sources describing DDD spend most of the time covering tactical patterns. Sometimes, they even skip strategic patterns. You can practice DDD by using just the strategic patterns. In some projects using tactical DDD patterns is even overkill. Unfortunetly, most people are doing the totally opposite thing. They use just tactical patterns without the strategic part. That’s super horrifying. If anybody asked me if any silver bullet exists in product software engineering, I’d have only one candidate: Domain-Driven Design Strategic Patterns. Strategic DDD helps us to get the answer about: what problem you are solving? will your solution meet stakeholders and users expectations? how complex is the project? what features are not necessary? how to separate services to support fast development in the long term? These questions are essential while implementing a new project, adding new functionality, or doing refactoring. Strategic DDD patterns give us a way to answer these questions consistently and predictably. Some engineers tell me they are “just engineers”. They don’t care too much about who uses their software or why. They are just implementing, say, JIRA tasks – building services with some bytes on the input and some bytes on the output. Such a mindset leads to a big disconnection between engineers and their clients whose problems we, as engineers, are trying to solve. Without proper communication, it’s much harder to create solutions that help clients in the right way. This is the goal in the end – not simply to process bytes. It’s tempting to spend a small amount of time on the planning phase of the project. To start coding as soon as possible and finish earlier. When anybody has such doubt, I like to say that “With 5 days of coding, you can save 1 day of planning”. The unasked questions will not magically disappear. The best way to overcome the Software Dark Ages is to attack it from multiple angles. Let’s see how DDD patterns can attack the system. See the full version. Event Storming Event Storming is a game-changer for Strategic DDD patterns and software development in general. I can’t believe why it’s not adopted by every team in the world yet. Event Storming is a workshop during which people with questions (often developers) meet with people with answers (often stakeholders). During the session, they can quickly explore complex business domains. In the beginning, you are focusing on building an entirely working flow based on Domain Events (orange sticky notes). Event Storming is a super flexible method. Thanks to that, you can verify if your solution meets expected requirements. You can also explore data flow, potential problems, or UX depending on the session’s goal. Your browser does not support the video tag. Verifying if the solution has no gaps and is about what users asked takes minutes. Introducing changes and verifying ideas in the developed and deployed code is hugely more expensive. Changing a sticky note on the board is extremely cheap. A civil engineer or a rocket engineer can quickly see the result of a mistake. They can see that something is clearly wrong before finishing the building process. It is not so simple with software because it’s not easily seen. Most of our critical decisions will not hurt anybody. Problems with features development and maintenance will not appear in a day. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Event Storming works when you are planning both a big project or just a single story. It’s just a matter of how much time you would like to spend. When we are using it for one story, it can be something between 10 minutes to a couple of hours. We tend to spend between one day to a couple of days of sessions for a bigger functionality. After the session, you should have the correct answer for the questions about: what problems are you trying to solve – instead of guessing what may be useful for the end-user or assuming that “we know everything better”, if stakeholders are happy with a proposed solution – rather than verifying that half-year later with the implementation, complexity of the problem is easily visible – it makes clear why adding one button may require a ton of work, initial idea of how you can split your microservices by responsibilities – instead of blindly grouping “similar things”. In the end, you will end up with much more trust from stakeholders because you are planning a solution together. It’s a much better approach than isolated coding in the basement. What’s excellent about Event Storming is that the outcome of a properly done session can be mapped directly to the code. It should help you avoid many discussions during development and speed up your work a lot. Transforming Event Storming directly to the code. You have to start with a clear purpose. Time can fly on a project, and before you know it, you have spent half a year on a project only to find that it’s not useful to anyone. Have you experienced that? It happens more often than you might think, which is why some people lose trust in “engineers” and how we can end up as developers without any autonomy. It’s common to fear how much time we need to “lose” for running the session. Thinking about time lost for running session is not the right approach. You should instead think about the benefits that you will lose if you will not run the session. I heard the story when running one Event Storming session stopped the implementation of the project for a couple months. It may sound bad, but during the session the team found that current assumptions are totally invalid. Continuation of the project would lead to a complete failure. Even if the session may look time-consuming in the short term, the company avoided a couple of months of useless development. Event Modeling In 2018 Adam Dymitruk proposed the Event Modeling technique. The notation and idea are heavily based on the Event Storming technique but adds a couple of new features. It also puts an extra emphasis on the UX part of the session. In general, these techniques are pretty compatible. Even if you stay with Event Storming, you may find some valuable approaches from Event Modeling that you may use. You can read more about the technique on eventmodeling.org. Bounded Context and transaction boundaries (aggregates) Bounded Context is another Strategic DDD pattern that helps us split big models into smaller, logical pieces. It’s a key for achieving proper services separation. If you need to touch half of the system to implement and test new functionality, your separation is wrong. Alternative to the wrong separation is lack of separation. Often, a symptom of lack of separation is god objects (huge objects that know too much or do too much). In that case, changes will primarily affect one service. The cost of that approach is a higher risk of big system failure and higher complexity of changes. In other words – in both cases, it will be harder to develop your project. A great tool that helps with the discovery of Bounded Context and Aggregates is (of course) Event Storming. Event Storming artifact As a result of the session, you can visually see how you should split your services and touchpoints between them. Ubiquitous Language Ubiquitous Language is a Strategic DDD pattern that covers building up a common language between developers, operations, stakeholders, and users. It is the most underestimated Strategic DDD pattern. Because who cares about the language, right? It took me time to see how many communication issues between developers and non-developers are because of using a different language. And how painful it is. I’d encourage you to pay attention to that as well. Because of miscommunication, developers aren’t solving the right problem as nobody understands what’s expected of them. Would you be surprised if I told you that Event Storming will help you develop Ubiquitous Language? Running the session together with stakeholders forces you to talk with them. It’s hard to build a solution together when you can’t understand each other. That’s why it’s critical to not miss your stakeholders at the workshop! From brainstorm to working flow Does DDD solve all problems? Even if DDD is great, it doesn’t solve all problems that we have. It’s important to manage your expectations. Even if we use these techniques on a high level in my team, we still have doubts if the created design is good enough. Sometimes we don’t know how to approach the problem. Sometimes we go back from the code to the design phase. Sometimes we make bad decisions. All of these are perfectly fine situations. There is no team in the world without these issues. It’s better to assume that it will happen and not be surprised. But we know that without DDD, these issues would be much more significant. While using DDD, you should pay special attention to avoid: Big Design Up Front, implementing code “for the future”, trying to create anything perfect, Instead, you should: Focus on delivering the MVP to the user in a short time (by short, I mean rather 1 month than 6 months). If you need to implement something “for the future” because it will be harder to add it later – that’s a very bad sign. You should think about how to make it easy to add it later. Reconcile that even if you do your best, your design will not be perfect from the beginning – it’s much better to improve it over time. For some teams, it may need a lot of work to go to this level. But I can promise you from my experience that it’s possible. And the fun that you will have from delivering software again is worth it! Note If you feel that you should be a tech leader to propose such improvements – you’re wrong! In the early days, when I was not a leader, I was already proposing many improvements in the teams in which I worked. You need to have good arguments with your teammates. We always explain “why” the techniques work in our articles. When you will use these arguments, they should be enough to convince them. If it will not work because your team is close-minded, it’s a good reason to consider changing the job. Software Renaissance It’s hard to go very deep with presented techniques within one article. My goal was rather to inspire you to question the status quo. This is not how our industry should work. If you are not okay with the status quo, I hope that I inspired you to learn new techniques. This is the best way to fight with the Software Dark Ages. If you want to learn about Tactical DDD patterns, you should check our previous articles. The examples are written in Go, but they can be easily ported to any language. Go has a very low entry point as well. Who knows, maybe you will find a new favorite language? 😉 What about Strategic DDD patterns? This article is actually an introduction to the next part of our article series. In the following months, we will deeply cover the most important Strategic DDD patterns on a fully working example project. Who are you in the Software Dark Ages? Just an ordinary hick who blindly follows the rules imposed by others? Inquisition, who will try to hate and stifle any unusual approach? Alchemist, trying to create gold? Even if it’s not scientifically backed? Or maybe you are secretly reading forbidden books in your cellar? Maybe you are looking with us to finish the Software Dark Ages and to begin the Software Renaissance? Let us know in the comments!

This post is a direct follow-up to Microservices test architecture where I introduced new kinds of tests to our example project. Wild Workouts uses Google Cloud Build as its CI/CD platform. It’s configured for continuous deployment, meaning changes land on production as soon as the pipeline passes. Note State of this article in 2026 This article is kept as an archive. As of 2026, GitHub Actions is our default choice for CI/CD. It’s simpler to set up, more widely adopted, and integrates seamlessly with GitHub repositories. The core concepts in this article—running integration tests with docker-compose, testing against real dependencies, and structuring your test pipeline—remain relevant regardless of which CI/CD platform you choose. The patterns shown here translate directly to GitHub Actions or any other CI system that supports Docker. If you consider our current setup, it’s both brave and naive. We have no tests running that could save us from obvious mistakes (the not-so-obvious mistakes can rarely be caught by tests, anyway). In this article I will show how to run integration, component, and end-to-end tests on Google Cloud Build using docker-compose. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ The current config Let’s look at the current cloudbuild.yaml file. While it’s simple, most steps run several times because we keep three microservices in a single repository. I’ll focus on the backend part and skip all config related to frontend deployment. steps: - id: trainer-lint name: golang entrypoint: ./scripts/lint.sh args: [trainer] - id: trainings-lint name: golang entrypoint: ./scripts/lint.sh args: [trainings] - id: users-lint name: golang entrypoint: ./scripts/lint.sh args: [users] - id: trainer-docker name: gcr.io/cloud-builders/docker entrypoint: ./scripts/build-docker.sh args: ["trainer", "$PROJECT_ID"] waitFor: [trainer-lint] - id: trainings-docker name: gcr.io/cloud-builders/docker entrypoint: ./scripts/build-docker.sh args: ["trainings", "$PROJECT_ID"] waitFor: [trainings-lint] - id: users-docker name: gcr.io/cloud-builders/docker entrypoint: ./scripts/build-docker.sh args: ["users", "$PROJECT_ID"] waitFor: [users-lint] - id: trainer-http-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainer, http, "$PROJECT_ID"] waitFor: [trainer-docker] - id: trainer-grpc-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainer, grpc, "$PROJECT_ID"] waitFor: [trainer-docker] - id: trainings-http-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainings, http, "$PROJECT_ID"] waitFor: [trainings-docker] - id: users-http-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [users, http, "$PROJECT_ID"] waitFor: [users-docker] - id: users-grpc-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [users, grpc, "$PROJECT_ID"] waitFor: [users-docker] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source Notice the waitFor key. It makes a step wait only for other specified steps. Some jobs can run in parallel this way. Here’s a more readable version of what’s going on: We have a similar workflow for each service: lint (static analysis), build the Docker image, and deploy it as one or two Cloud Run services. Since our test suite is ready and works locally, we need to figure out how to plug it into the pipeline. Docker Compose We already have one docker-compose definition, and I would like to keep it this way. We will use it for: running the application locally, running tests locally, running tests in the CI. These three targets have different needs. For example, when running the application locally, we want hot code reloading. But that’s pointless in CI. On the other hand, we can’t expose ports on localhost in CI, which is the easiest way to reach the application in the local environment. Luckily, docker-compose is flexible enough to support all these use cases. We’ll use a base docker-compose.yml file and an additional docker-compose.ci.yml file with overrides just for CI. You can run it by passing both files using the -f flag (notice there’s one flag for each file). Keys from the files merge in the order provided. docker-compose -f docker-compose.yml -f docker-compose.ci.yml up -d Note Typically, docker-compose looks for the docker-compose.yml file in the current directory or parent directories. Using the -f flag disables this behavior, so only specified files are parsed. To run this on Cloud Build, we can use the docker/compose image. - id: docker-compose name: 'docker/compose:1.19.0' args: ['-f', 'docker-compose.yml', '-f', 'docker-compose.ci.yml', 'up', '-d'] env: - 'PROJECT_ID=$PROJECT_ID' waitFor: [trainer-docker, trainings-docker, users-docker] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source Since we filled waitFor with proper step names, we can be sure the correct images are present. Here’s what we just added: The first override we add to docker-compose.ci.yml makes each service use Docker images by tag instead of building from docker/app/Dockerfile. This ensures our tests check the same images we’re going to deploy. Note the ${PROJECT_ID} variable in the image keys. This needs to be the production project, so we can’t hardcode it in the repository. Cloud Build provides this variable in each step, so we pass it to the docker-compose up command (see the definition above). services: trainer-http: image: "gcr.io/${PROJECT_ID}/trainer" trainer-grpc: image: "gcr.io/${PROJECT_ID}/trainer" trainings-http: image: "gcr.io/${PROJECT_ID}/trainings" users-http: image: "gcr.io/${PROJECT_ID}/users" users-grpc: image: "gcr.io/${PROJECT_ID}/users" Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/docker-compose.ci.yml Full source Network Many CI systems use Docker today, typically running each step inside a container with the chosen image. Using docker-compose in CI is a bit trickier, as it usually means running Docker containers from within a Docker container. On Google Cloud Build, all containers live inside the cloudbuild network. Adding this network as the default for our docker-compose.ci.yml is enough for CI steps to connect to the docker-compose services. Here’s the second part of our override file: networks: default: external: name: cloudbuild Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/docker-compose.ci.yml Full source Environment variables Using environment variables for configuration seems simple at first, but it quickly becomes complex considering how many scenarios we need to handle. Let’s list all of them: Running the application locally Running component tests locally Running component tests in CI Running end-to-end tests locally Running end-to-end tests in CI I didn’t include running the application on production, as it doesn’t use docker-compose. Why are component and end-to-end tests separate scenarios? Component tests spin up services on demand, while end-to-end tests communicate with services already running within docker-compose. This means both types use different endpoints to reach the services. Note For more details on component and end-to-end tests, see the previous article. The short version: we focus coverage on component tests, which don't include external services. End-to-end tests just confirm the contract isn't broken at a high level, and only for the most critical path. This is the key to decoupled services. We already keep a base .env file that holds most variables. It’s passed to each service in the docker-compose definition. Additionally, docker-compose loads this file automatically when it finds it in the working directory. This lets us use the variables inside the YAML definition as well. services: trainer-http: build: context: docker/app ports: # The $PORT variable comes from the .env file - "127.0.0.1:3000:$PORT" env_file: # All variables from .env are passed to the service - .env # (part of the definition omitted) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/docker-compose.yml Full source We also need these variables loaded when running tests. That’s pretty easy to do in bash: source .env # exactly the same thing . .env However, the variables in our .env file have no export prefix, so they won’t be passed to applications running in the shell. We can’t use the prefix because it’s incompatible with the syntax docker-compose expects. Additionally, we can’t use a single file for all scenarios. We need variable overrides, just as we did with the docker-compose definition. My idea is to keep one additional file for each scenario, loaded together with the base .env file. Let’s see the differences between all scenarios. For clarity, I’ve included only users-http, but the idea applies to all services. Scenario MySQL host Firestore host users-http address File Running locally localhost localhost localhost:3002 .env Local component tests localhost localhost localhost:5002 .test.env CI component tests mysql firestore-component-tests localhost:5002 .test.ci.env Local end-to-end tests - - localhost:3002 .e2e.env CI end-to-end tests - - users-http:3000 .e2e.ci.env Services run by docker-compose use ports 3000+, and component tests start services on ports 5000+. This way, both can run at the same time. I created a bash script that reads the variables and runs tests. Please don’t try to define such a complex scenario directly in the Makefile. Make is terrible at managing environment variables. Your mental health is at stake. Another reason for creating a dedicated script is that we keep three services in one repository and end-to-end tests in a separate directory. If I need to run the same command multiple times, I prefer calling a script with two variables rather than a long incantation of flags and arguments. A third argument in favor of separate bash scripts: they can be linted with shellcheck. #!/bin/bash set -e readonly service="$1" readonly env_file="$2" cd "./internal/$service" env $(cat "../../.env" "../../$env_file" | grep -Ev '^#' | xargs) go test -count=1 ./... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/scripts/test.sh Full source The script runs go test in the given directory with environment variables loaded from .env and the specified file. The env / xargs trick passes all variables to the following command. Notice how we remove comments from the file with grep. Note Testing cache go test caches successful results, as long as the related files are not modified. With tests that use Docker, you might change something at the infrastructure level, like the docker-compose definition or some environment variables. go test won’t detect this, and you can mistake a cached test for a successful one. It’s easy to get confused by this, and since our tests are fast anyway, we can disable the cache. The -count=1 flag is an idiomatic (though not obvious) way to do it. Running tests I have the end-to-end tests running after tests for all services pass. This should resemble how you would typically run them. Remember, end-to-end tests should serve as a double-check, and each service’s own tests should have the most coverage. Because our end-to-end tests are small in scope, we can run them before deploying the services. If they ran for a long time, this could block our deployments. A better approach in that scenario would be to rely on each service’s component tests and run the end-to-end suite in parallel. - id: trainer-tests name: golang entrypoint: ./scripts/test.sh args: ["trainer", ".test.ci.env"] waitFor: [docker-compose] - id: trainings-tests name: golang entrypoint: ./scripts/test.sh args: ["trainings", ".test.ci.env"] waitFor: [docker-compose] - id: users-tests name: golang entrypoint: ./scripts/test.sh args: ["users", ".test.ci.env"] waitFor: [docker-compose] - id: e2e-tests name: golang entrypoint: ./scripts/test.sh args: ["common", ".e2e.ci.env"] waitFor: [trainer-tests, trainings-tests, users-tests] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source The last thing we add is running docker-compose down after all tests pass. This is just cleanup. - id: docker-compose-down name: 'docker/compose:1.19.0' args: ['-f', 'docker-compose.yml', '-f', 'docker-compose.ci.yml', 'down'] env: - 'PROJECT_ID=$PROJECT_ID' waitFor: [e2e-tests] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source The second part of our pipeline looks like this now: Here’s how running the tests locally looks (I introduced this make target in the previous article). The commands are exactly the same as in CI, just with different .env files. test: @./scripts/test.sh common .e2e.env @./scripts/test.sh trainer .test.env @./scripts/test.sh trainings .test.env @./scripts/test.sh users .test.env Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/Makefile Full source Separating tests Looking at the table from the previous article, we could split tests into two groups based on whether they use Docker. Feature Unit Integration Component End-to-End Docker database No Yes Yes Yes Unit tests are the only category not using a Docker database, while integration, component, and end-to-end tests do. Even though we made all our tests fast and stable, setting up Docker infrastructure adds some overhead. It’s helpful to run all unit tests separately as a first guard against mistakes. We could use build tags to separate tests that don’t use Docker. You can define the build tags in the first line of a file. // +build docker We could now run unit tests separately from all tests. For example, the command below would run only tests that need Docker services: go test -tags=docker ./... Another way to separate tests is using the -short flag and checking testing.Short() in each test case. In the end, I decided not to introduce this separation. Our tests are stable and fast enough that running them all at once is not an issue. However, our project is small, and the test suite covers just the critical paths. As it grows, it might be a good idea to introduce build tags in component tests. Digression: A short story about CI debugging While I was making changes for this post, the initial test runs on Cloud Build kept failing. According to the logs, tests couldn’t reach services from docker-compose. I started debugging and added a simple bash script that would connect to the services via telnet. To my surprise, connecting to mysql:3306 worked correctly, but firestore:8787 didn’t, and the same for all Wild Workouts services. I thought this was because docker-compose takes a long time to start, but any number of retries didn’t help. Finally, I decided to try something crazy and set up a reverse SSH tunnel from one of the containers in docker-compose. This let me SSH into one of the containers while the build was still running. I then tried telnet and curl, and they worked correctly for all services. Finally, I spotted a bug in the bash script I used. readonly host="$1" readonly port="$1" # (some retries code) telnet "$host" "$port" The typo in the variable definition caused the telnet command to run like this: telnet $host $host. So why did it work for MySQL? It turns out telnet recognizes ports defined in /etc/services. So telnet mysql mysql got translated to telnet mysql 3306 and worked fine, but it failed for any other service. But why did the tests fail? Well, it turned out to be a totally different reason. Originally, we connected to MySQL like this: config := mysql.Config{ Addr: os.Getenv("MYSQL_ADDR"), User: os.Getenv("MYSQL_USER"), Passwd: os.Getenv("MYSQL_PASSWORD"), DBName: os.Getenv("MYSQL_DATABASE"), ParseTime: true, // with that parameter, we can use time.Time in mysqlHour.Hour } db, err := sqlx.Connect("mysql", config.FormatDSN()) if err != nil { return nil, errors.Wrap(err, "cannot connect to MySQL") } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/adapters/hour_mysql_repository.go Full source I looked into environment variables, and all of them were filled correctly. After adding some fmt.Println() debugs, I found that the config’s Addr part is completely ignored by the MySQL client because we didn’t specify the Net field. Why did it work for local tests? Because MySQL was exposed on localhost, which is the default address. The other test failed to connect to one of the Wild Workouts services, and it turned out to be because I used an incorrect port in the .env file. Why am I sharing this at all? I think it’s a great example of how working with CI systems can often look. When multiple things can fail, it’s easy to draw wrong conclusions and dig deep in the wrong direction. When in doubt, I like to reach for basic tools for investigating Linux issues, like strace, curl, or telnet. That’s also why I set up the reverse SSH tunnel, and I’m glad I did because it seems like a great way to debug issues inside CI. I feel I’ll use it again sometime. 😄 Summary We managed to keep a single docker-compose definition for running tests both locally and in the pipeline. The entire Cloud Build run from git push to production takes 4 minutes. We used a few clever hacks, but that’s normal when dealing with CI. Sometimes you just can’t avoid adding some bash magic to make things work. Unlike domain code, hacks in your CI setup shouldn’t hurt too much, as long as there are only a few of them. Just make sure it’s easy to understand what’s going on, so you’re not the only person with all the knowledge. With this post, we wrap up the first refactoring session of Wild Workouts. We’re going to move on to strategic patterns now. Thanks for reading, and see you soon! 👋

Thanks to tests and code review, you can make your project bug-free. Right? Well… actually, probably not. That would be too easy. 😉 These techniques lower the chance of bugs, but they can’t eliminate them entirely. Does that mean we need to live with the risk of bugs until the end of our lives? Over a year ago, I found a pretty interesting PR in the harbor project. It was a fix for an issue that allowed a regular user to create an admin user. This was obviously a severe security issue. Of course, automated tests didn’t find this bug earlier. This is what the bugfix looks like: ua.RenderError(http.StatusBadRequest, "register error:"+err.Error()) return } + + if !ua.IsAdmin && user.HasAdminRole { + msg := "Non-admin cannot create an admin user." + log.Errorf(msg) + ua.SendForbiddenError(errors.New(msg)) + return + } + userExist, err := dao.UserExists(user, "username") if err != nil { One if statement fixed the bug. Adding new tests also should ensure that there will be no regression in the future. Is it enough? Did it secure the application from a similar bug in the future? I’m pretty sure it didn’t. The problem becomes bigger in complex systems with a large team working on them. What if someone is new to the project and forgets to add this if statement? Even if you aren’t hiring new people now, you may hire them in the future. You’ll probably be surprised how long the code you’ve written will live. We should not trust people to use our code the way we intended: they won’t. In some cases, the solution that protects us from issues like this is good design. Good design should not allow our code to be used in an invalid way. Good design should guarantee that you can modify existing code without fear. People new to the project will feel safer introducing changes. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ In this article, I’ll show how I ensured that only authorized people can see and edit a training. In our case, a training can only be seen by the training owner (the attendee) and the trainer. I will implement it in a way that prevents our code from being misused. By design. Our current application assumes that a repository is the only way to access data. Because of that, I will add authorization at the repository level. This ensures that unauthorized users cannot access this data. Note #### What is Repository (tl;dr) If you haven't had a chance to read our previous articles, a repository is a pattern that helps us abstract database implementation from our application logic. If you want to know more about its advantages and learn how to apply it in your project, read my previous article: The Repository pattern: a painless way to simplify your Go service logic. But wait, is the repository the right place to manage authorization? Well, I can imagine some people may be skeptical about this approach. Of course, we could start a philosophical discussion on what belongs in the repository and what doesn’t. Also, the actual logic of who can see the training will be placed in the domain layer. I don’t see any significant downsides, and the advantages are clear. In my opinion, pragmatism should win here. Tip What’s also interesting in this series is that we focus on business-oriented applications. But even though the Harbor project is a pure system application, most of the presented patterns can be applied as well. After introducing Clean Architecture to our team, our teammate used this approach in his game to abstract the rendering engine. 😉 (Cheers, Mariusz, if you are reading that!) Show me the code, please! To achieve our robust design, we need to implement three things: Logic for who can see the training (domain layer), Functions to get the training (GetTraining in the repository), Functions to update the training (UpdateTraining in the repository). Domain layer The first part is the logic responsible for deciding if someone can see the training. Because it is part of the domain logic (you can discuss who can see the training with your business or product team), it should go in the domain layer. It’s implemented with the CanUserSeeTraining function. It is also acceptable to keep it at the repository level, but it’s harder to reuse. I don’t see any advantage to that approach: especially since putting it in the domain doesn’t cost anything. 😉 package training // ... type User struct { userUUID string userType UserType } // ... type ForbiddenToSeeTrainingError struct { RequestingUserUUID string TrainingOwnerUUID string } func (f ForbiddenToSeeTrainingError) Error() string { return fmt.Sprintf( "user '%s' can't see user '%s' training", f.RequestingUserUUID, f.TrainingOwnerUUID, ) } func CanUserSeeTraining(user User, training Training) error { if user.Type() == Trainer { return nil } if user.UUID() == training.UserUUID() { return nil } return ForbiddenToSeeTrainingError{user.UUID(), training.UserUUID()} } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/user.go Full source Repository Now that we have the CanUserSeeTraining function, we need to use it. Simple as that. func (r TrainingsFirestoreRepository) GetTraining( ctx context.Context, trainingUUID string, + user training.User, ) (*training.Training, error) { firestoreTraining, err := r.trainingsCollection().Doc(trainingUUID).Get(ctx) if status.Code(err) == codes.NotFound { return nil, training.NotFoundError{trainingUUID} } if err != nil { return nil, errors.Wrap(err, "unable to get actual docs") } tr, err := r.unmarshalTraining(firestoreTraining) if err != nil { return nil, err } + + if err := training.CanUserSeeTraining(user, *tr); err != nil { + return nil, err + } + return tr, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/trainings_firestore_repository.go Full source Isn’t this too simple? Our goal is to create simple, not complex, design and code. This is an excellent sign that it’s dead simple. We are changing UpdateTraining in the same way. func (r TrainingsFirestoreRepository) UpdateTraining( ctx context.Context, trainingUUID string, + user training.User, updateFn func(ctx context.Context, tr *training.Training) (*training.Training, error), ) error { trainingsCollection := r.trainingsCollection() return r.firestoreClient.RunTransaction(ctx, func(ctx context.Context, tx *firestore.Transaction) error { documentRef := trainingsCollection.Doc(trainingUUID) firestoreTraining, err := tx.Get(documentRef) if err != nil { return errors.Wrap(err, "unable to get actual docs") } tr, err := r.unmarshalTraining(firestoreTraining) if err != nil { return err } + + if err := training.CanUserSeeTraining(user, *tr); err != nil { + return err + } + updatedTraining, err := updateFn(ctx, tr) if err != nil { return err } return tx.Set(documentRef, r.marshalTraining(updatedTraining)) }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/trainings_firestore_repository.go Full source And… that’s all! Is there any way someone can misuse this? As long as the User is valid: no. This approach is similar to the method presented in the DDD Lite introduction article. It’s all about creating code that can’t be misused. This is what using UpdateTraining now looks like: func (h ApproveTrainingRescheduleHandler) Handle(ctx context.Context, cmd ApproveTrainingReschedule) (err error) { defer func() { logs.LogCommandExecution("ApproveTrainingReschedule", cmd, err) }() return h.repo.UpdateTraining( ctx, cmd.TrainingUUID, cmd.User, func(ctx context.Context, tr *training.Training) (*training.Training, error) { originalTrainingTime := tr.Time() if err := tr.ApproveReschedule(cmd.User.Type()); err != nil { return nil, err } err := h.trainerService.MoveTraining(ctx, tr.Time(), originalTrainingTime) if err != nil { return nil, err } return tr, nil }, ) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/approve_training_reschedule.go Full source Of course, there are still some rules if Training can be rescheduled, but this is handled by the Training domain type. It’s covered in details in the DDD Lite introduction article. 😉 Handling collections Even though this approach works perfectly for operating on a single training, you need to ensure that access to a collection of trainings is properly secured. There’s no magic here: func (r TrainingsFirestoreRepository) FindTrainingsForUser(ctx context.Context, userUUID string) ([]query.Training, error) { query := r.trainingsCollection().Query. Where("Time", ">=", time.Now().Add(-time.Hour*24)). Where("UserUuid", "==", userUUID). Where("Canceled", "==", false) iter := query.Documents(ctx) return r.trainingModelsToQuery(iter) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/trainings_firestore_repository.go Full source Doing this at the application layer with the CanUserSeeTraining function would be very expensive and slow. It’s better to create a bit of logic duplication. If this logic is more complex in your application, you can try abstracting it in the domain layer to a format that you can convert to query parameters in your database driver. I did this once, and it worked pretty nicely. But in Wild Workouts, that would add unnecessary complexity. Let’s Keep It Simple, Stupid. Handling internal updates We often want to have endpoints that allow a developer or your company’s operations department to make some “backdoor” changes. The worst thing you can do in this case is create any kind of “fake user” and hacks. From my experience, this ends with a lot of if statements added to the code. It also obfuscates the audit log (if you have one). Instead of a “fake user”, it’s better to create a special role and explicitly define that role’s permissions. If you need repository methods that don’t require any user (for Pub/Sub message handlers or migrations), it’s better to create separate repository methods. In that case, naming is essential: we need to be sure that the person who uses that method understands the security implications. From my experience, if updates are becoming much different for different actors, it’s worth introducing separate CQRS Commands per actor. In our case, it might be UpdateTrainingByOperations. Passing authentication via context.Context As far as I know, some people are passing authentication details via context.Context. I highly recommend not passing anything required by your application to work correctly via context.Context. The reason is simple: when passing values via context.Context, we lose one of Go’s most significant advantages: static typing. It also hides what exactly the input for your functions is. If you need to pass values via context for some reason, it may be a symptom of bad design somewhere in your service. Maybe the function is doing too much, and it’s hard to pass all arguments there? Perhaps it’s time to decompose it? And that’s all for today! As you can see, the presented approach is straightforward to implement quickly. I hope that it will help you with your project and give you more confidence in future development. Do you see that it can help in your project? Do you think that it may help your colleagues? Don’t forget to share it with them!

Do you know the rare feeling when you develop a new application from scratch and can cover all lines with proper tests? I said “rare” because most of the time, you work with software that has a long history, multiple contributors, and a less-than-obvious testing approach. Even if the code uses good patterns, the test suite doesn’t always follow. Some projects have no modern development environment set up, so there are only unit tests for things that are easy to test. For example, they test single functions separately because testing the public API is hard. The team needs to manually verify all changes, probably on some kind of staging environment. You know what happens when someone introduces changes and doesn’t know they need to test them manually. Other projects have no tests from the beginning. This allows quicker development by taking shortcuts, for example, keeping dependencies in the global state. When the team realizes the lack of tests causes bugs and slows them down, they decide to add them. But now, it’s impossible to do so reasonably. So the team writes an end-to-end test suite with proper infrastructure in place. End-to-end tests might give you some confidence, but you don’t want to maintain such a test suite. It’s hard to debug, takes a long time to test even the simplest change, and releasing the application takes hours. Introducing new tests is also not trivial in this scenario, so developers avoid it if they can. I want to introduce some ideas that have worked for us so far and should help you avoid the scenarios above. This post is not about which testing library is best or what tricks you can use (although I will show a few tips). It’s closer to something I would call “test architecture”. It’s not only about “how”, but also “where”, “what”, and “why”. There’s been a lot of discussion on different types of tests, for example, the “test pyramid” (Robert mentioned it in this section about testing scenarios). It’s a helpful model to keep in mind. However, it’s also abstract, and you can’t easily measure it. I want to take a more practical approach and show how to introduce a few kinds of tests in a Go project. Why bother about tests? But isn’t test code less important than the rest of the application? Can’t we just accept that keeping tests in good shape is hard and move on? Wouldn’t it speed up development? If you’ve been following this series, you know we base all posts on the Wild Workouts application. When I started writing this article, running tests locally didn’t even work correctly for me, and this is a relatively new project. There’s one reason this happened: we weren’t running tests in the CI pipeline. It’s a shock, but it seems even a serverless, cloud-native application using the most popular, cutting-edge technologies can be a mess in disguise. We know we should now add tests to the pipeline. It’s common knowledge that this gives you the confidence to deploy changes to production safely. However, there’s also a cost. Running tests will likely take a significant part of your pipeline’s duration. If you don’t approach their design and implementation with the same quality as the application code, you can realize it too late, with the pipeline taking one hour to pass and randomly failing on you. Even if your application code is well designed, the tests can become a bottleneck of delivering the changes. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ The Layers We’re now a few refactoring sessions into the project. We introduced patterns like Repository, Clean Architecture, and CQRS. With solid separation of concerns, we can reason about particular parts of the project much more easily. Let’s revisit the concept of layers we introduced in previous posts. If you didn’t have a chance to read these earlier, I recommend doing so before you continue: it will help you better understand this article. Take a look at a diagram that will help us understand the project’s structure. Below is a generic service built with the approach used in Wild Workouts. All external inputs start on the left. The only entry point to the application is through the Ports layer (HTTP handlers, Pub/Sub message handlers). Ports execute relevant handlers in the App layer. Some of these will call the Domain code, and some will use Adapters, which are the only way out of the service. The adapters layer is where your database queries and HTTP clients live. The diagram below shows the layers and flow of a part of the trainer service in Wild Workouts. Let’s now see what types of tests we would need to cover all of it. Unit tests We kick off with the inner layers and something everyone is familiar with: unit tests. The domain layer is where the most complex logic of your service lives. However, the tests here should be some of the simplest to write and run super fast. There are no external dependencies in the domain, so you don’t need any special infrastructure or mocks (except for really complex scenarios, but let’s leave that for now). As a rule of thumb, you should aim for high test coverage in the domain layer. Make sure you test only the exported code (black-box testing). Adding the _test suffix to the package name is a great practice to enforce this. The domain code is pure logic and straightforward to test, so it’s the best place to check all corner cases. Table-driven tests are especially great for this. func TestFactoryConfig_Validate(t *testing.T) { testCases := []struct { Name string Config hour.FactoryConfig ExpectedErr string }{ { Name: "valid", Config: hour.FactoryConfig{ MaxWeeksInTheFutureToSet: 10, MinUtcHour: 10, MaxUtcHour: 12, }, ExpectedErr: "", }, { Name: "equal_min_and_max_hour", Config: hour.FactoryConfig{ MaxWeeksInTheFutureToSet: 10, MinUtcHour: 12, MaxUtcHour: 12, }, ExpectedErr: "", }, // ... } for _, c := range testCases { t.Run(c.Name, func(t *testing.T) { err := c.Config.Validate() if c.ExpectedErr != "" { assert.EqualError(t, err, c.ExpectedErr) } else { assert.NoError(t, err) } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/hour_test.go Full source We leave the domain and enter the application layer. After introducing CQRS, we split it further into Commands and Queries. Depending on your project, there could be nothing to test or some complex scenarios to cover. Most of the time, especially in queries, this code just glues together other layers. Testing it doesn’t add any value. But if there’s any complex orchestration in commands, it’s another good case for unit tests. Note Watch out for complex logic living in the application layer. If you start testing business scenarios here, it’s worth considering introducing the domain layer. On the other hand, it’s the perfect place for orchestration: calling adapters and services in a particular order and passing return values around. If you separate it like that, application tests should not break every time you change the domain code. The application’s commands and queries have many external dependencies, as opposed to the domain code. They will be trivial to mock if you follow Clean Architecture and use the Dependency Inversion Principle (your code depends on interfaces, not on structs). In most cases, a struct with a single method will make a perfect mock here. Note If you prefer to use mocking libraries or code-generated mocks, you can use them as well. Go lets you define and implement small interfaces, so we choose to define the mocks ourselves, as it’s the simplest way. The snippet below shows how an application command is created with injected mocks. func newDependencies() dependencies { repository := &repositoryMock{} trainerService := &trainerServiceMock{} userService := &userServiceMock{} return dependencies{ repository: repository, trainerService: trainerService, userService: userService, handler: command.NewCancelTrainingHandler(repository, userService, trainerService), } } // ... deps := newDependencies() tr := tc.TrainingConstructor() deps.repository.Trainings = map[string]training.Training{ trainingUUID: *tr, } err := deps.handler.Handle(context.Background(), command.CancelTraining{ TrainingUUID: trainingUUID, User: training.MustNewUser(requestingUserID, tc.UserType), }) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/cancel_training_test.go Full source Watch out for adding tests that don’t check anything relevant: you don’t want to end up testing the mocks. Focus on the logic, and if there’s none, skip the test altogether. We’ve covered the two inner layers. I guess this didn’t seem novel so far, as we’re all familiar with unit tests. However, the Continuous Delivery Maturity Model lists them only on the “base” maturity level. Let’s now look into integration testing. Integration tests After reading this header, did you just imagine a long-running test that you have to retry several times to pass? And it’s because of that 30-second sleep someone added that turns out to be too short when Jenkins is running under load? There’s no reason for integration tests to be slow and flaky. And practices like automatic retries and increasing sleep times should be absolutely out of the question. In our context, an integration test checks if an adapter works correctly with external infrastructure. Most of the time, this means testing database repositories. These tests don’t check whether the database works correctly, but whether you use it correctly (the integration part). It’s also an excellent way to verify that you know how to use the database internals, like handling transactions. Because we need real infrastructure, integration tests are more challenging than unit tests to write and maintain. Usually, we can use docker-compose to spin up all dependencies. Note Should we test our application with the Docker version of a database? The Docker image will almost always be slightly different than what we run on production. In some cases, like Firestore, there’s only an emulator available, not the real database. Indeed, Docker doesn’t reflect the exact infrastructure you run in production. However, you’re much more likely to mess up an SQL query in the code than to stumble on issues because of a minor configuration difference. A good practice is to pin the image version to the same one running in production. Using Docker won’t give you 100% production parity, but it eliminates the “works on my machine” issues and tests your code with proper infrastructure. Robert covered integration tests for databases in depth in 4 practical principles of high-quality database integration tests in Go. Keeping integration tests stable and fast When dealing with network calls and databases, test speed becomes super important. It’s crucial to run tests in parallel, which you can enable in Go by calling t.Parallel(). It seems simple to do, but we have to make sure our tests support this behavior. For example, consider this trivial test scenario: Check if the trainings collection is empty. Call repository method that adds a training. Check if there’s one training in the collection. If another test uses the same collection, you will get random fails because of the race condition. Sometimes, the collection will contain more than one training we’ve just added. The simplest way out of this is never to assert things like list length, but to check for the exact thing we’re testing. For example, we could get all trainings, then iterate over the list to check if the expected ID is present. Another approach is to isolate the tests somehow, so they can’t interfere with each other. For example, each test case can work within a unique user’s context (see component tests below). Of course, both patterns are more complex than a simple length assertion. When you stumble upon this issue for the first time, it may be tempting to give up and decide that “our integration tests don’t need to run in parallel”. Don’t do this. You will need to get creative sometimes, but it’s not that much effort in the end. In return, your integration tests will be stable and run as fast as unit tests. If you find yourself creating a new database before each run, it’s another sign that you could rework tests to not interfere with each other. Note A common gotcha in Go 1.21 and earlier (You can skip this section if running Go 1.22 or later.) When working with table-driven tests, you can see code like this: for _, c := range testCases { t.Run(c.Name, func(t *testing.T) { // ... }) } This is an idiomatic way to run tests over a slice of test cases. Let’s say you now want to run each test case in parallel. The solution seems trivial: for _, c := range testCases { t.Run(c.Name, func(t *testing.T) { t.Parallel() // ... }) } Sadly, this won’t work as expected. The Common Mistakes page on Go’s GitHub wiki lists just two items, and both are actually about the same thing. So it seems there’s only one mistake you should worry about in Go. However, this one is really hard to spot sometimes. It’s not obvious initially, but adding the parallel switch makes the parent test function not wait for the subtests spawned by t.Run. Because of this, you can’t safely use the c loop variable inside the func closure. Running tests like this will usually cause all subtests to work with the last test case, ignoring all others. The worst part is the tests will pass, and you will see correct subtest names when running go test with the -v flag. The only way to notice this issue is to change the code expecting tests to fail and see them pass instead. As mentioned in the wiki, one way to fix this is to introduce a new scoped variable: for _, c := range testCases { c := c t.Run(c.Name, func(t *testing.T) { t.Parallel() // ... }) } It’s just a matter of taste, but we don’t like this approach, as it looks like a magic spell to anyone who doesn’t know what it means. Instead, we choose the more verbose but obvious approach: for i := range testCases { c := testCases[i] t.Run(c.Name, func(t *testing.T) { t.Parallel() // ... }) } Even if you know about this behavior, it’s dangerously easy to misuse it. We made this mistake in the Watermill library, which caused some tests not to run at all. You can see the fix in this commit. We covered the database repository with tests, but we also have a gRPC client adapter. How should we test this one? It’s similar to the application layer in this regard. If your test would just duplicate the code it checks, it probably makes no sense to add it. It becomes additional work when changing the code. Let’s consider the users service gRPC adapter: func (s UsersGrpc) UpdateTrainingBalance(ctx context.Context, userID string, amountChange int) error { _, err := s.client.UpdateTrainingBalance(ctx, &users.UpdateTrainingBalanceRequest{ UserId: userID, AmountChange: int64(amountChange), }) return err } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/users_grpc.go Full source There’s nothing interesting to test here. We could inject a mock client and check whether a proper method has been called. But this won’t verify anything, and each change in the code will require a corresponding change in the test. Component tests So far, we’ve created mostly narrow, specialized tests for isolated parts of the application. Such tests work great for checking corner cases and specific scenarios, but that doesn’t mean each service works correctly. It’s easy enough to forget to call an application handler from a port. Also, unit tests alone won’t help us make sure the application still works after a major refactoring. Is it now the time to run end-to-end tests across all our services? Not yet. Since there is no standard naming for test types, I encourage you to follow Simon Stewart’s advice from his Test Sizes post. Create a table that makes it obvious to everyone on the team what to expect from a particular test. You can then cut all (unproductive) discussions on the topic. In our case, the table could look like this: Feature Unit Integration Component End-to-End Docker database No Yes Yes Yes Use external systems No No No Yes Focused on business cases Depends on the tested code No Yes Yes Used mocks Most dependencies Usually none External systems None Tested API Go package Go package HTTP and gRPC HTTP To ensure each service works correctly internally, we introduce component tests to check all layers working together. A component test covers a single service, isolated from other services in the application. We will call real port handlers and use the infrastructure provided by Docker. However, we will mock all adapters reaching external services. You might be wondering, why not test external services as well? After all, we could use Docker containers and test all of them together. The issue is the complexity of testing several connected services. If you have just a couple of them, that can work well enough. But remember, you need proper infrastructure in place for each service you spin up, including all databases it uses and all external services it calls. It can easily be tens of services in total, usually owned by multiple teams. You don't want this as your primary testing approach. We’ll cover this next in end-to-end tests. But for now, we add component tests because we need a fast way to know if a service works correctly. To better understand why it’s essential to have component tests in place, I suggest looking at some quotes from Accelerate. Note If you haven’t heard about Accelerate yet, it’s a book describing research on high-performing software teams. I recommend reading it to learn what can help your team get up to speed. According to the book, this is what the best software teams said about testability. We can do most of our testing without requiring an integrated environment. We can and do deploy or release our application independently of other applications/services it depends on. Accelerate Wait, weren’t microservices supposed to fix teams being dependent on each other? If you think it’s impossible to achieve this in the application you work on, it’s likely because of poor architecture choices. You can fix it by applying strategic DDD patterns that we plan to introduce in future posts. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Accelerate follows up with: Unfortunately, in real life, many so-called service-oriented architectures don’t permit testing and deploying services independently of each other, and thus will not enable teams to achieve higher performance. Accelerate We raise this point throughout the series: using microservices doesn’t make your application and teams less coupled by itself. Decoupling takes conscious design of both the application’s architecture and the system level. In component tests, our goal is to check the completeness of a single service in isolation, with all infrastructure it needs. We make sure the service accepts the API we agreed on and responds with expected results. These tests are more challenging technically, but still relatively straightforward. We won’t be running a real service binary because we need to mock some dependencies. We have to modify the way we start the service to make it possible. Note Once again, if you follow the Dependency Inversion Principle (just a reminder, it’s part of SOLID), injecting mocks at the service level should be trivial. I introduced two constructors for our app.Application struct, which holds all commands and queries. The first one works just like before, setting up real gRPC clients and injecting them. The second replaces them with mocks. func NewApplication(ctx context.Context) (app.Application, func()) { // ... trainerGrpc := adapters.NewTrainerGrpc(trainerClient) usersGrpc := adapters.NewUsersGrpc(usersClient) return newApplication(ctx, trainerGrpc, usersGrpc), // ... } func NewComponentTestApplication(ctx context.Context) app.Application { return newApplication(ctx, TrainerServiceMock{}, UserServiceMock{}) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/service/service.go Full source We can now simply run the service in a separate goroutine. We want to run just a single service instance for all tests, so we use the TestMain function. It’s a simple way to insert a setup before running tests. func startService() bool { app := NewComponentTestApplication(context.Background()) trainingsHTTPAddr := os.Getenv("TRAININGS_HTTP_ADDR") go server.RunHTTPServerOnAddr(trainingsHTTPAddr, func(router chi.Router) http.Handler { return ports.HandlerFromMux(ports.NewHttpServer(app), router) }) ok := tests.WaitForPort(trainingsHTTPAddr) if !ok { log.Println("Timed out waiting for trainings HTTP to come up") } return ok } func TestMain(m *testing.M) { if !startService() { os.Exit(1) } os.Exit(m.Run()) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/service/component_test.go Full source I created the WaitForPort helper function that waits until the specified port is open or times out. It’s crucial, as you need to ensure the service has properly started. Don’t replace it with sleeps. You will either add too much delay and make the test slow, or make it too short and it will fail randomly. What to test in component tests? Usually, the happy path should be enough. Don’t check for corner cases there. Unit and integration tests should already cover these. Make sure the correct payloads are processed, the storage works, and responses are correct. Calling the ports I use HTTP clients generated by openapi-codegen. Similar to the server part, this makes writing tests much easier. For example, you don’t need to specify the whole REST path and worry about marshaling JSON each time. Even though the generated clients save us a lot of boilerplate, I still added the tests/client.go file with client wrappers for test purposes. func (c TrainingsHTTPClient) CreateTraining(t *testing.T, note string, hour time.Time) string { response, err := c.client.CreateTrainingWithResponse(context.Background(), trainings.CreateTrainingJSONRequestBody{ Notes: note, Time: hour, }) require.NoError(t, err) require.Equal(t, http.StatusNoContent, response.StatusCode()) contentLocation := response.HTTPResponse.Header.Get("content-location") return lastPathElement(contentLocation) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/tests/clients.go Full source The tests become even more readable, and it’s easy to understand what’s going on. Making tests pass is easy enough, but it’s much harder to understand them in a code review, reading between all asserts and mocks. It’s twice as important when modifying tests. Instead of the snippet above, we can now use a single line that clearly states what’s happening. trainingUUID := client.CreateTraining(t, "some note", hour) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/service/component_test.go Full source Other helper methods are FakeAttendeeJWT and FakeTrainerJWT. They generate valid authorization tokens with the chosen role. As gRPC uses structs generated from protobuf, the clients are already straightforward to use. End-to-end tests Finally, we come to the most dreaded part of our test suite. We will now leave all mocks behind. End-to-end tests verify your whole system working together. They are slow, error-prone, and hard to maintain. You still need them, but make sure you build them well. In Wild Workouts, it will be similar to running component tests, except we’re going to spin up all services inside the docker-compose. We’ll then verify a few critical paths calling just the HTTP endpoints, as this is what we expose to the external world. Note If you can’t run the entire platform on docker-compose, you need to find a similar approach. It could be a separate Kubernetes cluster or namespace if you’re already using it, or some kind of staging environment. Now comes the part where you’re going to have more questions than answers. Where should you keep end-to-end tests? Which team should own them? Where should you run them? How often? Should they be part of the CI/CD pipeline or a separate thing run from a cronjob? I can’t give you clear answers, as this heavily depends on your team’s structure, organizational culture, and CI/CD setup. As with most challenges, try an approach that seems best and iterate until you’re happy. We’re lucky to have just three services for the entire application, all using the same database. As the number of services and dependencies grows, your application will become harder to test this way. Try to keep end-to-end tests short. They should test whether services connect together correctly, not the logic inside them. These tests work as a double-check. They shouldn’t fail most of the time, and if they do, it usually means someone broke the contract. Note I said we’ll use just HTTP endpoints, as this is what is exposed publicly. There’s one exception: we call the users service via gRPC to add training balance for our test attendee. As you would expect, this endpoint is not accessible to the public. The code is very close to what we did in component tests. The main difference is we no longer spin up the service in a separate goroutine. Instead, all services run inside docker-compose using the same binaries we deploy to production. user := usersHTTPClient.GetCurrentUser(t) originalBalance := user.Balance _, err = usersGrpcClient.UpdateTrainingBalance(context.Background(), &users.UpdateTrainingBalanceRequest{ UserId: userID, AmountChange: 1, }) require.NoError(t, err) user = usersHTTPClient.GetCurrentUser(t) require.Equal(t, originalBalance+1, user.Balance, "Attendee's balance should be updated") trainingUUID := trainingsHTTPClient.CreateTraining(t, "some note", hour) trainingsResponse := trainingsHTTPClient.GetTrainings(t) require.Len(t, trainingsResponse.Trainings, 1) require.Equal(t, trainingUUID, trainingsResponse.Trainings[0].Uuid, "Attendee should see the training") user = usersHTTPClient.GetCurrentUser(t) require.Equal(t, originalBalance, user.Balance, "Attendee's balance should be updated after a training is scheduled") Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/tests/e2e_test.go Full source Acceptance tests You can often find acceptance tests defined as the next level after unit and integration tests. We see them as orthogonal to the technical aspects of tests. An acceptance test focuses on a complete business feature instead of implementation details. As Martin Fowler puts it: Here’s the thing: At one point you should make sure to test that your software works correctly from a user’s perspective, not just from a technical perspective. What you call these tests is really not that important. Having these tests, however, is. Pick a term, stick to it, and write those tests. Acceptance Tests — Do Your Features Work Correctly? In our case, component tests and end-to-end tests can both be considered acceptance tests. If you like, you can use the BDD style for some of them: it makes them easier to read but adds some boilerplate. Can we sleep well now? A service is not really tested until we break it in production. Jesse Robbins, according to The DevOps Handbook A reliable test suite will catch most of your bugs so you can deliver consistently. But you want to be ready for outages anyway. We now enter the topic of monitoring, observability, and chaos engineering. However, this is out of scope for today. Just keep in mind that no test suite will give you total confidence. It’s also crucial to have a straightforward process in place that allows quick rollbacks, reverts, and undoing migrations. Moving on If you look through the full commit, you might notice how we inject dependencies now is not very elegant. We’re going to iterate on this in the future. We have the test suite in place, but we’re still missing a critical part: it’s not running in our Continuous Integration pipeline. Also, coming up with proper docker-compose and environment variables for tests wasn’t trivial. The current solution works but is not straightforward to understand if you don’t know what’s going on. And running the tests in an automated build will only make it more complicated. I will cover these topics in another post in a few weeks, focusing more on the infrastructure and CI/CD in general. We plan to write on more advanced topics in 2021, and Robert will start with a post about secure repositories. Stay safe, and see you soon!

In the previous articles, we introduced techniques like DDD Lite, CQRS, and Clean (Hexagonal) Architecture. Even when used alone, they are beneficial. But they work best together. Like Power Rangers. Unfortunately, using them together in a real project is not easy. In this article, I will show you how to connect DDD Lite, CQRS, and Clean Architecture in the most pragmatic and efficient way. Why should I care? Working on a programming project is similar to planning and building a residential district. If you know the district will expand in the near future, you need to keep space for future improvements. Even if at the beginning it may look like a waste of space. You should reserve room for future facilities like residential blocks, hospitals, and temples. Without that, you will be forced to destroy buildings and streets to make space for new ones. It’s much better to think about that earlier. The situation is the same with code. If you know the project will be developed for longer than one month, you should keep the long term in mind from the beginning. You need to write your code in a way that won’t block your future work. Even if at the beginning it may look like over-engineering and a lot of extra boilerplate, you need to keep the long term in mind. This doesn’t mean you need to plan every feature you will implement in the future – it’s actually the opposite. This approach helps you adapt to new requirements or a changing understanding of your domain. Big upfront design is not needed here. This is critical in current times, when the world is changing fast and those who can’t adapt can simply go out of business. This is exactly what these patterns give you when combined: the ability to maintain constant development speed. Without destroying or touching existing code too much. Does it require more thinking and planning? Is it a more challenging way? Do you need to have extra knowledge to do that? Sure! But the long term result is worth that! Fortunately, you are in the right place to learn that. 😉 But let’s leave the theory behind us. Let’s get to the code. In this article, we will skip the reasoning for our design choices. We described these already in the previous articles. If you haven’t read them yet, I recommend doing so – you will understand this article better. As in previous articles, we will base our code on refactoring a real open-source project. This should make the examples more realistic and applicable to your own projects. Are you ready? Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Let’s refactor Let’s start our refactoring with the Domain-First approach. We will begin by introducing a domain layer. This ensures that implementation details do not affect our domain code. We can also put all our efforts into understanding the business problem, not writing boring database queries and API endpoints. The Domain-First approach works well for both rescue (refactoring 😉) and greenfield projects. To start building my domain layer, I needed to identify what the application is actually doing. This article will focus on refactoring of trainings Wild Workouts microservice. I started with identifying use cases handled by the application. After previous refactoring to Clean Architecture, we can find it in the TrainingService. When I work with a messy application, I look at RPC and HTTP endpoints to find supported use cases. One of the functionalities I identified is the approval of training reschedule. In Wild Workouts, a training reschedule approval is required if it was requested less than 24 hours before its date. If an attendee requests a reschedule, the trainer needs to approve it. If a trainer requests a reschedule, the attendee needs to accept it. - func (c TrainingService) ApproveTrainingReschedule(ctx context.Context, user auth.User, trainingUUID string) error { - return c.repo.ApproveTrainingReschedule(ctx, trainingUUID, func(training Training) (Training, error) { - if training.ProposedTime == nil { - return Training{}, errors.New("training has no proposed time") - } - if training.MoveProposedBy == nil { - return Training{}, errors.New("training has no MoveProposedBy") - } - if *training.MoveProposedBy == "trainer" && training.UserUUID != user.UUID { - return Training{}, errors.Errorf("user '%s' cannot approve reschedule of user '%s'", user.UUID, training.UserUUID) - } - if *training.MoveProposedBy == user.Role { - return Training{}, errors.New("reschedule cannot be accepted by requesting person") - } - - training.Time = *training.ProposedTime - training.ProposedTime = nil - - return training, nil - }) - } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/commit/8d9274811559399461aa9f6bf3829316b8ddfb63 Full source Start with the domain Even if this doesn’t look like the worst code you’ve ever seen, functions like ApproveTrainingReschedule tend to get more complex over time. More complex functions mean more potential bugs during future development. This is even more likely if you are new to the project and lack the “shaman knowledge” about it. You should always consider all the people who will work on the project after you, and make it resistant to being accidentally broken by them. That will help your project avoid becoming the legacy code that everybody is afraid to touch. You probably know that feeling when you’re new to a project and afraid to touch anything for fear of breaking the system. It’s not uncommon for people to change jobs more often than every two years. This makes it even more critical for long-term project development. If you don’t believe this code may become complex, I recommend checking the Git history of the worst place in the project you work on. In most cases, that worst code started with “just a couple of simple ifs”. 😉 The more complex the code becomes, the more difficult it will be to simplify later. We should be sensitive to emerging complexity and try to simplify it as soon as we can. Training domain entity While analyzing the current use cases handled by the trainings microservice, I found that they are all related to a training. It’s natural to create a Training type to handle these operations. Methods of refactored TrainingService Note noun == entity Is this a valid approach for discovering entities? Well, not really. DDD provides tools that help us model complex domains without guessing (Strategic DDD Patterns, Aggregates). We don’t want to guess what our aggregates look like – we want tools to discover them. The Event Storming technique is extremely useful here… but that’s a topic for an entire separate article. The topic is complex enough to deserve a couple of articles. And that’s what we will do shortly. 😉 Does this mean you shouldn’t use these techniques without Strategic DDD Patterns? Of course not! The current approach can be good enough for simpler projects. Unfortunately (or fortunately 😉), not all projects are simple. package training // ... type Training struct { uuid string userUUID string userName string time time.Time notes string proposedNewTime time.Time moveProposedBy UserType canceled bool } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/training.go Full source All fields are private to provide encapsulation. This is critical to meet the “always keep a valid state in the memory” rule from the article about DDD Lite. Thanks to validation in the constructor and encapsulated fields, we are sure that Training is always valid. Now, someone who has no knowledge about the project cannot use it in the wrong way. The same rule applies to any methods provided by Training. package training func NewTraining(uuid string, userUUID string, userName string, trainingTime time.Time) (*Training, error) { if uuid == "" { return nil, errors.New("empty training uuid") } if userUUID == "" { return nil, errors.New("empty userUUID") } if userName == "" { return nil, errors.New("empty userName") } if trainingTime.IsZero() { return nil, errors.New("zero training time") } return &Training{ uuid: uuid, userUUID: userUUID, userName: userName, time: trainingTime, }, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/training.go Full source Approve reschedule in the domain layer As described in DDD Lite introduction, we build our domain with methods oriented on behaviours. Not on data. Let’s model ApproveReschedule on our domain entity. package training // ...s func (t *Training) IsRescheduleProposed() bool { return !t.moveProposedBy.IsZero() && !t.proposedNewTime.IsZero() } var ErrNoRescheduleRequested = errors.New("no training reschedule was requested yet") func (t *Training) ApproveReschedule(userType UserType) error { if !t.IsRescheduleProposed() { return errors.WithStack(ErrNoRescheduleRequested) } if t.moveProposedBy == userType { return errors.Errorf( "trying to approve reschedule by the same user type which proposed reschedule (%s)", userType.String(), ) } t.time = t.proposedNewTime t.proposedNewTime = time.Time{} t.moveProposedBy = UserType{} return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/reschedule.go Full source If you haven’t had the chance to read: DDD Lite introduction, Introducing Clean Architecture, Introducing basic CQRS, Repository pattern. I highly recommend checking them out. They will help you understand this article better. They explain the decisions and techniques we combine in this article. Orchestrate with command Now the application layer can be responsible only for the orchestration of the flow. There is no domain logic there. We hide the entire business complexity in the domain layer. This was exactly our goal. For getting and saving a training, we use the Repository pattern. package command // ... func (h ApproveTrainingRescheduleHandler) Handle(ctx context.Context, cmd ApproveTrainingReschedule) (err error) { defer func() { logs.LogCommandExecution("ApproveTrainingReschedule", cmd, err) }() return h.repo.UpdateTraining( ctx, cmd.TrainingUUID, cmd.User, func(ctx context.Context, tr *training.Training) (*training.Training, error) { originalTrainingTime := tr.Time() if err := tr.ApproveReschedule(cmd.User.Type()); err != nil { return nil, err } err := h.trainerService.MoveTraining(ctx, tr.Time(), originalTrainingTime) if err != nil { return nil, err } return tr, nil }, ) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/approve_training_reschedule.go Full source Refactoring of training cancelation Let’s now take a look at CancelTraining from TrainingService. The domain logic is simple: you can cancel a training up to 24 hours before its date. If it’s less than 24 hours before the training and you want to cancel it anyway: if you are the trainer, the attendee will get their training “back” plus one extra session (nobody likes to change plans on the same day!) if you are the attendee, you will lose this training This is how the current implementation looks like: - func (c TrainingService) CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error { - return c.repo.CancelTraining(ctx, trainingUUID, func(training Training) error { - if user.Role != "trainer" && training.UserUUID != user.UUID { - return errors.Errorf("user '%s' is trying to cancel training of user '%s'", user.UUID, training.UserUUID) - } - - var trainingBalanceDelta int - if training.CanBeCancelled() { - // just give training back - trainingBalanceDelta = 1 - } else { - if user.Role == "trainer" { - // 1 for cancelled training +1 fine for cancelling by trainer less than 24h before training - trainingBalanceDelta = 2 - } else { - // fine for cancelling less than 24h before training - trainingBalanceDelta = 0 - } - } - - if trainingBalanceDelta != 0 { - err := c.userService.UpdateTrainingBalance(ctx, training.UserUUID, trainingBalanceDelta) - if err != nil { - return errors.Wrap(err, "unable to change trainings balance") - } - } - - err := c.trainerService.CancelTraining(ctx, training.Time) - if err != nil { - return errors.Wrap(err, "unable to cancel training") - } - - return nil - }) - } You can see some kind of “algorithm” for calculating the training balance delta during cancellation. That’s not a good sign in the application layer. Logic like this one should live in our domain layer. If you start to see some if’s related to logic in your application layer, you should think about how to move it to the domain layer. It will be easier to test and re-use in other places. It may depend on the project, but often domain logic is fairly stable after initial development and can live unchanged for a long time. It can survive moving between services, framework changes, library changes, and API changes. Thanks to this separation, we can make all these changes in a much safer and faster way. Let’s decompose the CancelTraining method to multiple, separated pieces. That will allow us to test and change them independently. First, we need to handle cancellation logic and marking Training as canceled. package training func (t Training) CanBeCanceledForFree() bool { return t.time.Sub(time.Now()) >= time.Hour*24 } var ErrTrainingAlreadyCanceled = errors.New("training is already canceled") func (t *Training) Cancel() error { if t.IsCanceled() { return ErrTrainingAlreadyCanceled } t.canceled = true return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/cancel.go Full source Nothing really complicated here. That’s good! The second part that needs to move is the “algorithm” for calculating the trainings balance after cancellation. In theory, we could put it in the Cancel() method, but in my opinion it would break the Single Responsibility Principle and CQS. And I like small functions. But where should we put it? Some object? A domain service? In some languages, like the one that starts with J and ends with ava, that would make sense. But in Go, it’s good enough to just create a simple function. package training // CancelBalanceDelta return trainings balance delta that should be adjusted after training cancelation. func CancelBalanceDelta(tr Training, cancelingUserType UserType) int { if tr.CanBeCanceledForFree() { // just give training back return 1 } switch cancelingUserType { case Trainer: // 1 for cancelled training +1 "fine" for cancelling by trainer less than 24h before training return 2 case Attendee: // "fine" for cancelling less than 24h before training return 0 default: panic(fmt.Sprintf("not supported user type %s", cancelingUserType)) } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/cancel_balance.go Full source The code is now straightforward. I can imagine that I could sit with any non-technical person and go through this code to explain how it works. What about tests? It may be a bit controversial, but in my opinion tests are redundant here. Test code would replicate the implementation of the function. Any change in the calculation algorithm would require copying the logic to the tests. I wouldn’t write a test here, but if you’ll sleep better at night – why not! Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Moving CancelTraining to command Our domain is ready, so let’s now use it. We will do it in the same way as previously: getting the entity from the repository, orchestration of domain stuff, calling external trainer service to cancel the training (this service is the point of truth of “trainer’s calendar”), returning entity to be saved in the database. package command // ... func (h CancelTrainingHandler) Handle(ctx context.Context, cmd CancelTraining) (err error) { defer func() { logs.LogCommandExecution("CancelTrainingHandler", cmd, err) }() return h.repo.UpdateTraining( ctx, cmd.TrainingUUID, cmd.User, func(ctx context.Context, tr *training.Training) (*training.Training, error) { if err := tr.Cancel(); err != nil { return nil, err } if balanceDelta := training.CancelBalanceDelta(*tr, cmd.User.Type()); balanceDelta != 0 { err := h.userService.UpdateTrainingBalance(ctx, tr.UserUUID(), balanceDelta) if err != nil { return nil, errors.Wrap(err, "unable to change trainings balance") } } if err := h.trainerService.CancelTraining(ctx, tr.Time()); err != nil { return nil, errors.Wrap(err, "unable to cancel training") } return tr, nil }, ) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/cancel_training.go Full source Repository refactoring The initial implementation of the repository was pretty tricky because of the custom method for every use case. - type trainingRepository interface { - FindTrainingsForUser(ctx context.Context, user auth.User) ([]Training, error) - AllTrainings(ctx context.Context) ([]Training, error) - CreateTraining(ctx context.Context, training Training, createFn func() error) error - CancelTraining(ctx context.Context, trainingUUID string, deleteFn func(Training) error) error - RescheduleTraining(ctx context.Context, trainingUUID string, newTime time.Time, updateFn func(Training) (Training, error)) error - ApproveTrainingReschedule(ctx context.Context, trainingUUID string, updateFn func(Training) (Training, error)) error - RejectTrainingReschedule(ctx context.Context, trainingUUID string, updateFn func(Training) (Training, error)) error - } Thanks to introducing the training.Training entity, we can have a much simpler version: one method for adding a new training and one for updating. package training // ... type Repository interface { AddTraining(ctx context.Context, tr *Training) error GetTraining(ctx context.Context, trainingUUID string, user User) (*Training, error) UpdateTraining( ctx context.Context, trainingUUID string, user User, updateFn func(ctx context.Context, tr *Training) (*Training, error), ) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/domain/training/repository.go Full source As in the previous article, we implemented our repository using Firestore. We will also use Firestore in the current implementation. Keep in mind that this is an implementation detail – you can use any database you want. In the previous article, we showed example implementations using different databases. package adapters // ... func (r TrainingsFirestoreRepository) UpdateTraining( ctx context.Context, trainingUUID string, user training.User, updateFn func(ctx context.Context, tr *training.Training) (*training.Training, error), ) error { trainingsCollection := r.trainingsCollection() return r.firestoreClient.RunTransaction(ctx, func(ctx context.Context, tx *firestore.Transaction) error { documentRef := trainingsCollection.Doc(trainingUUID) firestoreTraining, err := tx.Get(documentRef) if err != nil { return errors.Wrap(err, "unable to get actual docs") } tr, err := r.unmarshalTraining(firestoreTraining) if err != nil { return err } if err := training.CanUserSeeTraining(user, *tr); err != nil { return err } updatedTraining, err := updateFn(ctx, tr) if err != nil { return err } return tx.Set(documentRef, r.marshalTraining(updatedTraining)) }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/trainings_firestore_repository.go Full source Connecting everything How do we use our code now? What about our ports layer? Thanks to the refactoring that Miłosz did in refactoring to Clean Architecture article, our ports layer is decoupled from other layers. That’s why, after this refactoring, it requires almost no significant changes. We just call the application command instead of the application service. Clean/Hexagonal Architecture layers. package ports // ... type HttpServer struct { app app.Application } // ... func (h HttpServer) CancelTraining(w http.ResponseWriter, r *http.Request) { trainingUUID := r.Context().Value("trainingUUID").(string) user, err := newDomainUserFromAuthUser(r.Context()) if err != nil { httperr.RespondWithSlugError(err, w, r) return } err = h.app.Commands.CancelTraining.Handle(r.Context(), command.CancelTraining{ TrainingUUID: trainingUUID, User: user, }) if err != nil { httperr.RespondWithSlugError(err, w, r) return } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/ports/http.go Full source How to approach such refactoring in a real project? It may not be obvious how to do such a refactoring in a real project. It’s hard to do a code review and agree as a team on the refactoring direction. In my experience, the best approach is Pair or Mob programming. Even if at the beginning you may feel it’s a waste of time, the knowledge sharing and instant review will save a lot of time in the future. Thanks to great knowledge sharing, you can work much faster after the initial project or refactoring phase. In this case, don’t consider the time “lost” for Mob/Pair programming. Consider the time you may lose by not doing it. It will also help you finish the refactoring much faster because you won’t need to wait for decisions. You can agree on them immediately. Mob and pair programming also work perfectly when implementing complex, greenfield projects. Knowledge sharing is an especially important investment in that case. I’ve seen multiple times how this approach allowed teams to move very fast on the project in the long term. When you are doing refactoring, it’s also critical to agree on reasonable timeboxes. And keep them. You can quickly lose your stakeholders’ trust when you spend an entire month on refactoring and the improvement is not visible. It’s also critical to integrate and deploy your refactoring as fast as you can. Ideally, on a daily basis (if you can do it for non-refactoring work, I’m sure you can do it for refactoring as well!). If your changes stay unmerged and undeployed for a longer time, you increase the chance of breaking functionality. It will also block any work in the refactored service or make changes harder to merge (it’s not always possible to stop all other development around). But how do you know if the project is complex enough to use mob programming? Unfortunately, there is no magic formula for that. But there are questions you should ask yourself: do we understand the domain? do we know how to implement that? will it end up with a monstrous pull request that nobody will be able to review? can we risk worse knowledge sharing while not doing mob/pair programming? Summary And we come to an end. 😄 The entire diff for the refactoring is available on our Wild Workouts GitHub (watch out, it’s huge!). If you haven’t had the chance to read previous articles yet, you know what to do! Even if some of the approaches used are simplified, you should already be able to use them in your project and see value from them. I hope that after this article, you also see how all the introduced patterns work nicely together. If not yet, don’t worry. It took me three years to connect all the dots. But it was worth the time spent. After I understood how everything is connected, I started to look at new projects in a totally different way. It allowed me and my teams to work more efficiently in the long term. It’s also important to mention that, like all techniques, this combination is not a silver bullet. If you are creating a project that is not complex and won’t be touched any time soon after one month of development, it’s probably enough to put everything in one main package. 😉 Just keep in mind when that one month of development becomes one year! We will also continue these topics in the next articles. We will shortly drift to Strategic DDD Patterns, which should also help you gain a higher-level perspective on your projects. Did this article help you understand how to connect DDD, Clean Architecture, and CQRS? Is something still unclear? Please let us know in the comments! We’re happy to discuss all your questions!

You probably know at least one service that: has one big, unmaintainable model that is hard to understand and change, limits parallel work on new features, or can’t scale optimally. But bad things often come in threes. It’s not uncommon to see services with all these problems. What idea comes to mind first for solving these issues? Let’s split it into more microservices! Unfortunately, without proper research and planning, the situation after blindly refactoring may actually be worse than before: Business logic and flow may become even harder to understand: complex logic is often easier to understand if it’s in one place. Distributed transactions: things are sometimes together for a reason. A big transaction in one database is much faster and less complex than a distributed transaction across multiple services. Adding new changes may require extra coordination if one of the services is owned by another team. Microservices are useful, but they will not solve all your issues... To be clear: I’m not an enemy of microservices. I’m just against blindly applying microservices in a way that introduces unnecessary complexity and mess instead of making our lives easier. Another approach is using CQRS (Command Query Responsibility Segregation) with previously described Clean Architecture and DDD Lite. It can solve the mentioned problems in a much simpler way. Isn’t CQRS a complex technique? Isn’t CQRS one of those C#/Java/über enterprise patterns that are hard to implement and make a big mess in the code? Many books, presentations, and articles describe CQRS as a very complicated pattern. But that’s not the case. In practice, CQRS is a simple pattern that doesn’t require a lot of investment. It can be easily extended with more complex techniques like event-driven architecture, event-sourcing, or polyglot persistence. But these aren’t always needed. Even without any extra patterns, CQRS can offer better decoupling and code structure that is easier to understand. When to not use CQRS in Go? How to get all benefits from CQRS? You can learn all that in today’s article. 😉 As always, I will do this by refactoring the Wild Workouts application. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ How to implement basic CQRS in Go CQRS (Command Query Responsibility Segregation) was initially described by Greg Young. It has one simple assumption: instead of having one big model for reads and writes, you should have two separate models. One for writes and one for reads. It also introduces concepts of command and query, and leads to splitting application services into two separate types: command and query handlers. Standard, non-CQRS architecture CQRS architecture Command vs Query In the simplest terms: a Query should not modify anything, just return data. A command is the opposite: it should make changes in the system but not return any data. This lets us cache queries more efficiently and reduces the complexity of commands. This may sound like a serious constraint, but in practice, it’s not. Most operations are either reads or writes. Very rarely both. Of course, for a query, we don’t consider side effects like logs or metrics as modifying anything. For commands, returning an error is perfectly normal. Note As with most rules, it’s okay to break them… as long as you understand why they were introduced and what tradeoffs you’re making. In practice, you rarely need to break these rules. I’ll share examples at the end of the article. What does the most basic implementation look like in practice? In the previous article, Miłosz introduced an application service that executes application use cases. Let’s start by cutting this service into separate command and query handlers. ApproveTrainingReschedule command Previously, the training reschedule was approved from the application service TrainingService. - func (c TrainingService) ApproveTrainingReschedule(ctx context.Context, user auth.User, trainingUUID string) error { - return c.repo.ApproveTrainingReschedule(ctx, trainingUUID, func(training Training) (Training, error) { - if training.ProposedTime == nil { - return Training{}, errors.New("training has no proposed time") - } - if training.MoveProposedBy == nil { - return Training{}, errors.New("training has no MoveProposedBy") - } - if *training.MoveProposedBy == "trainer" && training.UserUUID != user.UUID { - return Training{}, errors.Errorf("user '%s' cannot approve reschedule of user '%s'", user.UUID, training.UserUUID) - } - if *training.MoveProposedBy == user.Role { - return Training{}, errors.New("reschedule cannot be accepted by requesting person") - } - - training.Time = *training.ProposedTime - training.ProposedTime = nil - - return training, nil - }) - } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/commit/8d9274811559399461aa9f6bf3829316b8ddfb63#diff-ddf06fa26668dd91e829c7bfbd68feaeL127 Full source There were some magic validations there. These are now done in the domain layer. I also found that we forgot to call the external trainer service to move the training. Oops. 😉 Let’s refactor it to the CQRS approach. Note Because CQRS works best with applications following Domain-Driven Design, I also refactored existing models to DDD Lite during the CQRS refactoring. DDD Lite is described in more detail in the previous article. We start implementing a command with the command structure definition. This structure provides all data needed to execute the command. If a command has only one field, you can skip the structure and just pass it as a parameter. It’s a good idea to use domain-defined types in the command, like training.User in this case. We don’t need to do any casting later, and we have type safety assured. It can save us a lot of issues with string parameters passed in wrong order. package command // ... type ApproveTrainingReschedule struct { TrainingUUID string User training.User } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/approve_training_reschedule.go Full source The second part is a command handler that executes the command. package command // ... type ApproveTrainingRescheduleHandler struct { repo training.Repository userService UserService trainerService TrainerService } // ... func (h ApproveTrainingRescheduleHandler) Handle(ctx context.Context, cmd ApproveTrainingReschedule) (err error) { defer func() { logs.LogCommandExecution("ApproveTrainingReschedule", cmd, err) }() return h.repo.UpdateTraining( ctx, cmd.TrainingUUID, cmd.User, func(ctx context.Context, tr *training.Training) (*training.Training, error) { originalTrainingTime := tr.Time() if err := tr.ApproveReschedule(cmd.User.Type()); err != nil { return nil, err } err := h.trainerService.MoveTraining(ctx, tr.Time(), originalTrainingTime) if err != nil { return nil, err } return tr, nil }, ) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/approve_training_reschedule.go Full source The flow is now much easier to understand. You can clearly see that we approve a reschedule of a persisted *training.Training, and if it succeeds, we call the external trainer service. Thanks to techniques described in the DDD Lite article, the command handler doesn’t need to know when it can perform this operation. The domain layer handles all of that. This clear flow is even more visible in complex commands. Fortunately, the current implementation is straightforward. That’s good. Our goal is to create simple software, not complicated software. If CQRS is the standard way of building applications in your team, it also speeds up onboarding teammates unfamiliar with a service. You just need a list of available commands and queries and a quick look at how their execution works. No more jumping through random places in code. This is how it looks in one of my team’s most complex services: You may ask: shouldn’t it be split into multiple services? In practice, that would be a terrible idea. Many operations here need to be transactionally consistent. Splitting into separate services would involve several distributed transactions (Sagas). This would make the flow much more complex, harder to maintain, and harder to debug. It’s not a good tradeoff. It’s also worth mentioning that none of these operations are very complex. Complexity scales horizontally here. We will cover the important topic of splitting microservices in-depth soon. Did I mention that we messed it up in Wild Workouts on purpose? 😉 Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! But let’s go back to our command. It’s time to use it in our HTTP port. It’s available in HttpServer via the injected Application structure, which contains all of our command and query handlers. package app import ( "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/query" ) type Application struct { Commands Commands Queries Queries } type Commands struct { ApproveTrainingReschedule command.ApproveTrainingRescheduleHandler CancelTraining command.CancelTrainingHandler // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/app.go Full source type HttpServer struct { app app.Application } // ... func (h HttpServer) ApproveRescheduleTraining(w http.ResponseWriter, r *http.Request) { trainingUUID := chi.URLParam(r, "trainingUUID") user, err := newDomainUserFromAuthUser(r.Context()) if err != nil { httperr.RespondWithSlugError(err, w, r) return } err = h.app.Commands.ApproveTrainingReschedule.Handle(r.Context(), command.ApproveTrainingReschedule{ User: user, TrainingUUID: trainingUUID, }) if err != nil { httperr.RespondWithSlugError(err, w, r) return } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/ports/http.go Full source The command handler can be called this way from any port: HTTP, gRPC, or CLI. It’s also useful for executing migrations and loading fixtures (we already do this in Wild Workouts). RequestTrainingReschedule command Some command handlers can be very simple. func (h RequestTrainingRescheduleHandler) Handle(ctx context.Context, cmd RequestTrainingReschedule) (err error) { defer func() { logs.LogCommandExecution("RequestTrainingReschedule", cmd, err) }() return h.repo.UpdateTraining( ctx, cmd.TrainingUUID, cmd.User, func(ctx context.Context, tr *training.Training) (*training.Training, error) { if err := tr.UpdateNotes(cmd.NewNotes); err != nil { return nil, err } tr.ProposeReschedule(cmd.NewTime, cmd.User.Type()) return tr, nil }, ) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/command/request_training_reschedule.go Full source It may be tempting to skip this layer for such simple cases to save some boilerplate. That’s true, but remember that writing code is always much cheaper than maintenance. Adding this simple type takes 3 minutes. People who read and extend this code later will appreciate that effort. AvailableHoursHandler query Queries in the application layer are usually boring. In the most common case, we need to write a read model interface (AvailableHoursReadModel) that defines how we query the data. Commands and queries are also a great place for cross-cutting concerns, like logging and instrumentation. By putting them here, we ensure that performance is measured consistently whether called from the HTTP or gRPC port. package query // ... type AvailableHoursHandler struct { readModel AvailableHoursReadModel } type AvailableHoursReadModel interface { AvailableHours(ctx context.Context, from time.Time, to time.Time) ([]Date, error) } // ... type AvailableHours struct { From time.Time To time.Time } func (h AvailableHoursHandler) Handle(ctx context.Context, query AvailableHours) (d []Date, err error) { start := time.Now() defer func() { logrus. WithError(err). WithField("duration", time.Since(start)). Debug("AvailableHoursHandler executed") }() if query.From.After(query.To) { return nil, errors.NewIncorrectInputError("date-from-after-date-to", "Date from after date to") } return h.readModel.AvailableHours(ctx, query.From, query.To) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/app/query/available_hours.go Full source We also need to define data types returned by the query. In our case, it’s query.Date. Note To understand why we don’t use structures generated from OpenAPI, you should check our articles on DRY and Clean Architecture. package query import ( "time" ) type Date struct { Date time.Time HasFreeHours bool Hours []Hour } type Hour struct { Available bool HasTrainingScheduled bool Hour time.Time } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/app/query/types.go Full source Our query model is more complex than the domain hour.Hour type. This is a common scenario. Often, it’s driven by the website’s UI, and it’s more efficient to generate optimal responses on the backend side. As the application grows, differences between domain and query models may increase. Thanks to this separation and decoupling, we can make changes to each independently. This is critical for maintaining fast development in the long term. package hour type Hour struct { hour time.Time availability Availability } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/hour.go Full source But where does AvailableHoursReadModel get the data? For the application layer, this is fully transparent and not relevant. This allows us to add performance optimizations in the future while touching just one part of the application. Note If you are not familiar with the concept of _ports and adapters_, I highly recommend reading our article about Clean Architecture in Go. In practice, the current implementation gets data from our write models database. You can find the AllTrainings read model implementation and tests for DatesFirestoreRepository in the adapters layer. Data for our queries is currently queried from the same database where write models are stored. If you’ve read about CQRS before, you may have seen recommendations to use a separate database built from events for queries. This can be a good idea, but only in very specific cases. I’ll describe this in the Future optimizations section. In our case, getting data from the write models database is sufficient. HourAvailabilityHandler query We don’t need to add a read model interface for every query. It’s also fine to use the domain repository and pick the data we need. import ( "context" "time" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour" ) type HourAvailabilityHandler struct { hourRepo hour.Repository } func (h HourAvailabilityHandler) Handle(ctx context.Context, time time.Time) (bool, error) { hour, err := h.hourRepo.GetHour(ctx, time) if err != nil { return false, err } return hour.IsAvailable(), nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/app/query/hour_availability.go Full source Naming Naming is one of the most challenging and essential parts of software development. In Introduction to DDD Lite article, I described a rule that says you should stick to language that is as close as possible to how non-technical people (often called “business”) talk. This also applies to command and query names. You should avoid names like “Create training” or “Delete training”. This is not how business and users understand your domain. You should instead use “Schedule training” and “Cancel training”. All commands and queries of the trainings service We will cover this topic more deeply in an article about Ubiquitous Language. Until then, talk to your business people and listen to how they describe operations. Think twice about whether any of your command names really need to start with “Create/Delete/Update”. Future optimizations Basic CQRS gives advantages like better code organization, decoupling, and simpler models. There is also one even more important advantage: the ability to extend CQRS with more powerful and complex patterns. Async commands Some commands are slow by nature. They may be making external calls or doing heavy computation. In these cases, we can introduce an Asynchronous Command Bus, which executes the command in the background. Using asynchronous commands has additional infrastructure requirements, like having a queue or pub/sub. Fortunately, the Watermill library can help you handle this in Go. You can find more details in the Watermill CQRS documentation. (By the way, we are the authors of Watermill as well. 😉 Feel free to contact us if something’s not clear there!) A separate database for queries Our current implementation uses the same database for reads (queries) and writes (commands). If we needed to provide more complex queries or have really fast reads, we could use the polyglot persistence technique. The idea is to duplicate queried data in a more optimal format in another database. For example, we could use Elasticsearch to index data that can be searched and filtered more easily. In this case, data synchronization can be done via events. One of the most important implications of this approach is eventual consistency. You should ask yourself if it’s an acceptable tradeoff in your system. If you’re not sure, you can start without polyglot persistence and migrate later. It’s good to defer key decisions like this one. An example implementation is described in the Watermill CQRS documentation as well. Maybe with time, we will introduce it also in Wild Workouts, who knows? CQRS with polyglot persistence Event-Sourcing If you work in a domain with strict audit requirements, you should definitely check out the event sourcing technique. For example, I’m currently working in the financial domain, and event sourcing is our default persistence choice. It provides out-of-the-box auditing and helps with reverting the effects of bugs. CQRS is often described together with event sourcing. The reason is that by design, event-sourced systems don’t store the model in a format ready for reads (queries), but just a list of events used by writes (commands). In other words, it’s harder to provide API responses. Thanks to the separation of command and query models, this isn’t a big problem. Our read models for queries live independently by design. There are many more advantages of event sourcing that are visible in financial systems. But let’s leave that for another article. 😉 Until then, you can check out the ebook by Greg Young: Versioning in an Event Sourced System. The same Greg Young who described CQRS. When to not use CQRS? CQRS is not a silver bullet that fits everywhere. A good example is authentication. You provide a login and password, and in return, you get confirmation of success and maybe a token. If your application is a simple CRUD that receives and returns the same data, it’s also not the best fit for CQRS. That’s why the users microservice in Wild Workouts doesn’t use Clean Architecture and CQRS. In simple, data-oriented services, these patterns usually don’t make sense. On the other hand, you should keep an eye on such services. If you notice the logic growing and development becoming painful, maybe it’s time for some refactoring? Returning created entity via API with CQRS I know some people struggle with using CQRS for REST APIs that return the created entity as the response to a POST request. Isn’t that against CQRS? Not really! You can solve it in two ways: Call the command in the HTTP port and, after it succeeds, call the query to get the data to return. Instead of returning the created entity, return a 204 HTTP code with the content-location header set to the created resource URL. The second approach is better in my opinion because it doesn’t require always querying for the created entity (even if the client doesn’t need the data). With the second approach, the client only follows the link if needed. That call can also be cached. The only question is: how do you get the created entity’s ID? A common practice is to provide the UUID of the entity to be created in the command. This approach has the advantage of still working as expected if the command handler is asynchronous. If you don’t want to work with UUIDs, as a last resort you can return the ID from the handler. It won’t be the end of the world. 😉 cmd := command.ScheduleTraining{ TrainingUUID: uuid.New().String(), UserUUID: user.UUID, UserName: user.DisplayName, TrainingTime: postTraining.Time, Notes: postTraining.Notes, } err = h.app.Commands.ScheduleTraining.Handle(r.Context(), cmd) if err != nil { httperr.RespondWithSlugError(err, w, r) return } w.Header().Set("content-location", "/trainings/" + cmd.TrainingUUID) w.WriteHeader(http.StatusNoContent) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/ports/http.go Full source You can now put CQRS in your resume! We did it: we have a basic CQRS implementation in Wild Workouts. You should also have an idea of how to extend the application in the future. While preparing the code for this article, I also refactored the trainer service towards DDD. I will cover this in the next article. The entire diff of that refactoring is already available on our GitHub repository. Having every command handler as a separate type also helps with testing because it’s easier to build dependencies for them. This part is covered by Miłosz in Microservices Test Architecture. Are you using CQRS with any extensions? Do you have any project where you don’t know how to apply these patterns? Feel free to share and ask in the comments!

The authors of Accelerate dedicate an entire chapter to software architecture and how it affects development performance. One recurring theme is designing applications to be “loosely coupled”. The goal is for your architecture to support the ability of teams to get their work done—from design through to deployment—without requiring high-bandwidth communication between teams. Accelerate Note If you haven’t read Accelerate yet, I highly recommend it. The book presents scientific evidence on methods leading to high performance in development teams. The approach I describe is not only based on our experiences but also mentioned throughout the book. While coupling seems mostly related to microservices across multiple teams, we find loosely coupled architecture just as useful for work within a single team. Maintaining architecture standards enables parallel work and helps onboard new team members. You’ve probably heard of the “low coupling, high cohesion” concept, but it’s rarely obvious how to achieve it. The good news: it’s the main benefit of Clean Architecture. The pattern is not only an excellent way to start a project but also helpful when refactoring a poorly designed application. I focus on the latter in this post. I show the refactoring of a real application, so it should be clear how to apply similar changes in your projects. We’ve also noticed other benefits of this approach: a standard structure, so it’s easy to find your way in the project, faster development in the long term, mocking dependencies becomes trivial in unit tests, easy switching from prototypes to proper solutions (e.g., changing in-memory storage to an SQL database). Clean Architecture I had a hard time coming up with this post’s title because the pattern comes in many flavors. There’s Clean Architecture, Onion Architecture, Hexagonal Architecture, and Ports and Adapters. Over the past few years, we’ve tried to use these patterns in Go in an idiomatic way. This involved trying out approaches, failing, changing them, and trying again. We arrived at a mix of the ideas above, sometimes not strictly following the original patterns, but we found it works well in Go. I’ll demonstrate our approach by refactoring Wild Workouts, our example application. I want to point out that the idea is not new at all. A big part of it is abstracting away implementation details, a standard practice in technology, especially software. Another name for it is separation of concerns. The concept is so old that it exists on several levels: structures, namespaces, modules, packages, and even (micro)services. All are meant to keep related things within a boundary. Sometimes, it feels like common sense: When you optimize an SQL query, you don’t want to risk changing the display format. When you change an HTTP response format, you don’t want to alter the database schema. Our approach to Clean Architecture combines two ideas: separating Ports and Adapters, and limiting how code structures refer to each other. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Before We Start Before introducing Clean Architecture in Wild Workouts, I refactored the project a bit. The changes come from patterns we shared in previous posts. The first one is using separate models for database entities and HTTP responses. I’ve introduced changes in the users service in my post on the DRY principle. I applied the same pattern now in trainer and trainings as well. See the full commit on GitHub. The second change follows the Repository Pattern that Robert introduced in the previous article. My refactoring moved database-related code in trainings to a separate structure. Separating Ports and Adapters Ports and Adapters can be called different names, like interfaces and infrastructure. The idea is to explicitly separate these two categories from the rest of your application code. We take the code in these groups and place it in different packages. We refer to them as “layers”. The layers we usually use are adapters, ports, application, and domain. Note Read this if you know Hexagonal Architecture You might be confused about the ports and adapters. We accidentally picked the same names for different things (even if they fit well). Our ports are Hexagonal Architecture’s Primary Adapters. Our adapters are Hexagonal Architecture’s Secondary Adapters. The idea stays the same. We find the original primary/secondary naming hard to grasp, so feel free to use what works for you. You could use gateways, entry points, interfaces, infrastructure, and so on. Just make sure it’s consistent and your team knows what goes where. What about the original ports? Thanks to Go’s implicit interfaces, we see no value in keeping a dedicated layer for them. We keep interfaces close to where they’re used (see below). An adapter is how your application talks to the external world. You have to adapt your internal structures to what the external API expects. Think SQL queries, HTTP or gRPC clients, file readers and writers, Pub/Sub message publishers. A port is an input to your application, and the only way the external world can reach it. It could be an HTTP or gRPC server, a CLI command, or a Pub/Sub message subscriber. The application logic is a thin layer that “glues together” other layers. It’s also known as “use cases”. If you read this code and can’t tell what database it uses or what URL it calls, it’s a good sign. Sometimes it’s very short, and that’s fine. Think about it as an orchestrator. If you also follow Domain-Driven Design, you can introduce a domain layer that holds just the business logic. Note If the idea of separating layers is still unclear, look at your smartphone. It uses similar concepts. You can control your smartphone using physical buttons, the touchscreen, or a voice assistant. Whether you press the “volume up” button, swipe the volume bar up, or say “Siri, volume up,” the effect is the same. There are several entry points (ports) to the “change volume” logic. When you play music, you hear it from the speaker. If you plug in headphones, the audio automatically switches to them. Your music app doesn’t care. It’s not talking to the hardware directly but using one of the adapters the OS provides. Can you imagine creating a mobile app that has to know which headphones model is connected to the smartphone? Including SQL queries directly inside application logic is similar: it exposes implementation details. Let’s start refactoring by introducing the layers in the trainings service. The project looks like this so far: trainings/ ├── firestore.go ├── go.mod ├── go.sum ├── http.go ├── main.go ├── openapi_api.gen.go └── openapi_types.gen.go This part of refactoring is simple: Create ports, adapters, and app directories. Move each file to the proper directory. trainings/ ├── adapters │ └── firestore.go ├── app ├── go.mod ├── go.sum ├── main.go └── ports ├── http.go ├── openapi_api.gen.go └── openapi_types.gen.go I introduced similar packages in the trainer service. We won’t make any changes to the users service this time. There’s no application logic there, and overall it’s tiny. As with every technique, apply Clean Architecture where it makes sense. Note If the project grows in size, you may find it helpful to add another level of subdirectories. For example, adapters/hour/mysql_repository.go or ports/http/hour_handler.go. You probably noticed there are no files in the app package. We now have to extract the application logic from HTTP handlers. The Application Layer Let’s see where our application logic lives. Take a look at the CancelTraining method in the trainings service. func (h HttpServer) CancelTraining(w http.ResponseWriter, r *http.Request) { trainingUUID := r.Context().Value("trainingUUID").(string) user, err := auth.UserFromCtx(r.Context()) if err != nil { httperr.Unauthorised("no-user-found", err, w, r) return } err = h.db.CancelTraining(r.Context(), user, trainingUUID) if err != nil { httperr.InternalError("cannot-update-training", err, w, r) return } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/http.go Full source This method is the entry point to the application. There’s not much logic there, so let’s go deeper into the db.CancelTraining method. Inside the Firestore transaction, there’s a lot of code that doesn’t belong in database handling. What’s worse, the application logic inside this method uses the database model (TrainingModel) for decision-making: if training.canBeCancelled() { // ... } else { // ... } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/firestore.go Full source Mixing business rules (like when a training can be canceled) with the database model slows down development because the code becomes hard to understand and reason about. It’s also difficult to test such logic. To fix this, we add an intermediate Training type in the app layer: type Training struct { UUID string UserUUID string User string Time time.Time Notes string ProposedTime *time.Time MoveProposedBy *string } func (t Training) CanBeCancelled() bool { return t.Time.Sub(time.Now()) > time.Hour*24 } func (t Training) MoveRequiresAccept() bool { return !t.CanBeCancelled() } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training.go Full source It should now be clear on first read when a training can be canceled. We can’t tell how the training is stored in the database or what JSON format the HTTP API uses. That’s a good sign. We can now update the database layer methods to return this generic application type instead of the database-specific structure (TrainingModel). The mapping is trivial because the structs have the same fields (but from now on, they can evolve independently from each other). t := TrainingModel{} if err := doc.DataTo(&t); err != nil { return nil, err } trainings = append(trainings, app.Training(t)) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters/trainings_firestore_repository.go Full source The Application Service We then create a TrainingsService struct in the app package that will serve as the entry point to the trainings application logic. type TrainingService struct { } func (c TrainingService) CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error { } So how do we call the database now? Let’s try to replicate what was used so far in the HTTP handler. type TrainingService struct { db adapters.DB } func (c TrainingService) CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error { return c.db.CancelTraining(ctx, user, trainingUUID) } This code won’t compile, though. import cycle not allowed package github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings imports github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters imports github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app imports github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/adapters We need to decide how the layers should refer to each other. The Dependency Inversion Principle A clear separation between ports, adapters, and application logic is useful by itself. Clean Architecture takes it further with Dependency Inversion. The rule states that outer layers (implementation details) can refer to inner layers (abstractions), but not the other way around. The inner layers should instead depend on interfaces. The Domain knows nothing about other layers whatsoever. It contains pure business logic. The Application can import domain but knows nothing about outer layers. It has no idea whether it’s being called by an HTTP request, a Pub/Sub handler, or a CLI command. Ports can import inner layers. Ports are the entry points to the application, so they often execute application services or commands. However, they can’t directly access Adapters. Adapters can import inner layers. Usually, they will operate on types found in Application and Domain, for example, retrieving them from the database. Again, this is not a new idea. The Dependency Inversion Principle is the “D” in SOLID. Do you think it applies only to OOP? It turns out Go interfaces are a perfect match. The principle addresses how packages should refer to each other. The best approach is rarely obvious, especially in Go, where import cycles are forbidden. Perhaps that’s why some developers claim it’s best to avoid “nesting” and keep all code in one package. But packages exist for a reason: separation of concerns. Going back to our example, how should we refer to the database layer? Because Go interfaces don’t need to be explicitly implemented, we can define them next to the code that needs them. So the application service says: “I need a way to cancel a training with a given UUID. I don’t care how you do it, but I trust you to do it right if you implement this interface.” type trainingRepository interface { CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error } type TrainingService struct { trainingRepository trainingRepository } func (c TrainingService) CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error { return c.trainingRepository.CancelTraining(ctx, user, trainingUUID) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service.go Full source The database method calls gRPC clients of the trainer and users services. That’s not the proper place, so we introduce two new interfaces for the service to use. type userService interface { UpdateTrainingBalance(ctx context.Context, userID string, amountChange int) error } type trainerService interface { ScheduleTraining(ctx context.Context, trainingTime time.Time) error CancelTraining(ctx context.Context, trainingTime time.Time) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service.go Full source Note Note that “user” and “trainer” in this context are not microservices, but application (business) concepts. It just happens that in this project, they live in the scope of microservices with the same names. We move implementations of these interfaces to adapters as UsersGrpc and TrainerGrpc. As a bonus, the timestamp conversion now happens there as well, invisible to the application service. Extracting the Application Logic The code compiles, but our application service doesn’t do much yet. Now it’s time to extract the logic and put it in the proper place. Finally, we can use the update function pattern from the Repositories post to extract the application logic out of the repository. func (c TrainingService) CancelTraining(ctx context.Context, user auth.User, trainingUUID string) error { return c.repo.CancelTraining(ctx, trainingUUID, func(training Training) error { if user.Role != "trainer" && training.UserUUID != user.UUID { return errors.Errorf("user '%s' is trying to cancel training of user '%s'", user.UUID, training.UserUUID) } var trainingBalanceDelta int if training.CanBeCancelled() { // just give training back trainingBalanceDelta = 1 } else { if user.Role == "trainer" { // 1 for cancelled training +1 fine for cancelling by trainer less than 24h before training trainingBalanceDelta = 2 } else { // fine for cancelling less than 24h before training trainingBalanceDelta = 0 } } if trainingBalanceDelta != 0 { err := c.userService.UpdateTrainingBalance(ctx, training.UserUUID, trainingBalanceDelta) if err != nil { return errors.Wrap(err, "unable to change trainings balance") } } err := c.trainerService.CancelTraining(ctx, training.Time) if err != nil { return errors.Wrap(err, "unable to cancel training") } return nil }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service.go Full source The amount of logic suggests we might want to introduce a domain layer in the future. For now, let’s keep it as is. I described the process for just a single CancelTraining method. Refer to the full diff to see how I refactored all other methods. Dependency Injection How do we tell the service which adapter to use? First, we define a simple constructor for the service. func NewTrainingsService( repo trainingRepository, trainerService trainerService, userService userService, ) TrainingService { if repo == nil { panic("missing trainingRepository") } if trainerService == nil { panic("missing trainerService") } if userService == nil { panic("missing userService") } return TrainingService{ repo: repo, trainerService: trainerService, userService: userService, } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service.go Full source Then, in main.go we inject the adapter. trainingsRepository := adapters.NewTrainingsFirestoreRepository(client) trainerGrpc := adapters.NewTrainerGrpc(trainerClient) usersGrpc := adapters.NewUsersGrpc(usersClient) trainingsService := app.NewTrainingsService(trainingsRepository, trainerGrpc, usersGrpc) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/main.go Full source Using the main function is the simplest way to inject dependencies. We’ll explore the wire library as the project becomes more complex in future posts. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Adding tests Initially, the project had all layers mixed, and it wasn’t possible to mock dependencies. The only way to test it was with integration tests, requiring a proper database and all services running. While it’s fine to cover some scenarios with such tests, they tend to be slower and not as enjoyable to work with as unit tests. After introducing these changes, I was able to cover CancelTraining with a unit test suite. I used the standard Go approach of table-driven tests to make all cases easy to read and understand. { Name: "return_training_balance_when_trainer_cancels", UserRole: "trainer", Training: app.Training{ UserUUID: "trainer-id", Time: time.Now().Add(48 * time.Hour), }, ShouldUpdateBalance: true, ExpectedBalanceChange: 1, }, { Name: "extra_training_balance_when_trainer_cancels_before_24h", UserRole: "trainer", Training: app.Training{ UserUUID: "trainer-id", Time: time.Now().Add(12 * time.Hour), }, ShouldUpdateBalance: true, ExpectedBalanceChange: 2, }, Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service_test.go Full source I didn’t use any libraries for mocking. You can use them if you like, but your interfaces should usually be small enough to write dedicated mocks by hand. type trainerServiceMock struct { trainingsCancelled []time.Time } func (t *trainerServiceMock) CancelTraining(ctx context.Context, trainingTime time.Time) error { t.trainingsCancelled = append(t.trainingsCancelled, trainingTime) return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/app/training_service_test.go Full source Did you notice the unusually high number of unimplemented methods in repositoryMock? That’s because we use a single training service for all methods, so we need to implement the full interface even when testing just one method. We’ll improve it in our next post on CQRS. What about the boilerplate? You might be wondering if we introduced too much boilerplate. The project did grow in lines of code, but that alone doesn’t do any harm. It’s an investment in loose coupling that will pay off as the project grows. Keeping everything in one package may seem easier at first, but having boundaries helps when working in a team. If all your projects have a similar structure, onboarding new team members is straightforward. Consider how much harder it would be with all layers mixed (Mattermost’s app package is an example of this approach). Handling application errors One extra thing I added is port-agnostic errors with slugs. They allow the application layer to return generic errors that both HTTP and gRPC handlers can handle. if from.After(to) { return nil, errors.NewIncorrectInputError("date-from-after-date-to", "Date from after date to") } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/app/hour_service.go Full source The error above translates to a 400 Bad Request HTTP response in ports. It includes a slug that can be translated on the frontend and shown to the user. It’s yet another pattern to avoid leaking implementation details into application logic. What else? I encourage you to read through the full commit to see how I refactored other parts of Wild Workouts. You might be wondering how to enforce correct usage of layers. Is it yet another thing to remember in code reviews? Luckily, you can check the rules with static analysis. Use Robert’s go-cleanarch linter locally or include it in your CI pipeline. With layers separated, we’re ready to introduce more advanced patterns. Next time, Robert will show how to improve the project by applying CQRS. If you’d like to read more on Clean Architecture, see Why using Microservices or Monolith can be just a detail?.

Did you ever hear about a project where changes were tested on customers you don’t like or countries that aren’t profitable? Or even worse: did you work on such a project? It’s not enough to say that it’s unfair and unprofessional. It’s also hard to develop anything new because you’re afraid to make any change in your codebase. In the 2019 HackerRank Developer Skills Report, Professional growth & learning was marked as the most important factor when looking for a new job. Do you think you can learn anything and grow when you test your application this way? It all leads to frustration and burnout. To develop your application easily and with confidence, you need a set of tests on multiple levels. In this article, I will cover practical examples of implementing high-quality database integration tests. I will also cover basic Go testing techniques, like test tables, assert functions, parallel execution, and black-box testing. What does it actually mean for test quality to be high? 4 principles of high-quality tests I prepared 4 rules that we need to pass to say that our integration test quality is high. 1. Fast Good tests need to be fast. There is no compromise here. Everybody hates long-running tests. Think about your teammates’ time and mental health when they’re waiting for test results, both in CI and locally. It’s terrifying. When you wait for a long time, you’ll likely start doing something else in the meantime. After the CI passes (hopefully), you’ll need to switch back to this task. Context switching is one of the biggest productivity killers. It’s exhausting for our brains. We are not robots. I know that some companies still have tests that run for 24 hours. We don’t want to follow this approach. 😉 You should be able to run your tests locally in less than 1 minute, ideally in less than 10s. I know that sometimes this requires a time investment. It’s an investment with an excellent ROI (Return on Investment)! You can quickly check your changes, and deployment times are much shorter too. From my experience, it’s always worth finding quick wins that reduce test execution time the most. The Pareto principle (80/20 rule) works here perfectly! 2. Testing enough scenarios on all levels I hope you already know that 100% test coverage is not the best idea (unless it’s a simple/critical library). It’s always a good idea to ask yourself the question “how easily can it break?”. It’s even more worth asking this question if the test you’re implementing starts to look exactly like the function you’re testing. At the end of the day, we’re not writing tests because tests are nice: they should save our ass! From my experience, coverage like 70-80% is a pretty good result in Go. It’s also not the best idea to cover everything with component or end-to-end tests. First, you won’t be able to do that because of the inability to simulate some error scenarios, like rollbacks on the repository. Second, it will break the first rule: these tests will be slow. Tests on several layers should also overlap, so we will know that integration is done correctly. You may think that solution for that is simple: the test pyramid! And that’s true…sometimes. Especially in applications that handle a lot of operations based on writes. But what about applications that aggregate data from multiple services and expose it via API? They have no complex logic for saving the data. Most of the code is probably related to database operations. In this case, we should use a reversed test pyramid (it actually looks more like a Christmas tree). When a big part of our application is connected to infrastructure (for example, a database), it’s hard to cover a lot of functionality with unit tests. 3. Tests need to be robust and deterministic Do you know that feeling when you’re doing some urgent fix, tests are passing locally, you push changes to the repository and… after 20 minutes they fail in the CI? It’s incredibly frustrating. It also discourages us from adding new tests and decreases our trust in them. You should fix that issue as fast as you can. The Broken windows theory is really valid here. 4. You should be able to execute most of the tests locally Tests that you run locally should give you enough confidence that the feature you developed or refactored is still working. E2E tests should just double-check if everything is integrated correctly. You’ll also have much more confidence when contracts between services are robust because of using gRPC, protobuf, or OpenAPI. This is a good reason to cover as much as we can at lower levels (starting with the lowest): unit, integration, and component tests. Only then E2E. Implementation We have some common theoretical ground. But nobody pays us for being masters of programming theory. Let’s go to some practical examples that you can implement in your project. Let’s start with the repository pattern that I described in the previous article. You don’t need to read the rest of the articles in the series, but it’s a good idea to check at least the previous one. It will make it much clearer how our repository implementation works. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ The way we interact with our database is defined by the hour.Repository interface. It assumes that our repository implementation is simple. All complex logic is handled by the domain part of our application. It should just save the data without any validations, etc. One of the significant advantages of this approach is the simplification of the repository and test implementation. In the previous article, I prepared three different database implementations: MySQL, Firebase, and in-memory. We will test all of them. They are fully compatible, so we can have just one test suite. package hour type Repository interface { GetOrCreateHour(ctx context.Context, hourTime time.Time) (*Hour, error) UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, error), ) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/repository.go Full source Because of multiple repository implementations, in our tests we iterate through a list of them. Note It’s actually a pretty similar pattern to how we implemented tests in Watermill. All Pub/Sub implementations are passing the same test suite. All tests that we write will be black-box tests. In other words, we will only cover public functions with tests. To ensure that, all our test packages have the _test suffix. This forces us to use only the public interface of the package. It will pay back in the future with much more stable tests that are not affected by any internal changes. If you can’t write good black-box tests, you should consider if your public APIs are well designed. All our repository tests are executed in parallel. Thanks to that, they take less than 200ms. After adding multiple test cases, this time should not increase significantly. package main_test // ... func TestRepository(t *testing.T) { rand.Seed(time.Now().UTC().UnixNano()) repositories := createRepositories(t) for i := range repositories { // When you are looping over the slice and later using iterated value in goroutine (here because of t.Parallel()), // you need to always create variable scoped in loop body! // More info here: https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables r := repositories[i] t.Run(r.Name, func(t *testing.T) { // It's always a good idea to build all non-unit tests to be able to work in parallel. // Thanks to that, your tests will be always fast and you will not be afraid to add more tests because of slowdown. t.Parallel() t.Run("testUpdateHour", func(t *testing.T) { t.Parallel() testUpdateHour(t, r.Repository) }) t.Run("testUpdateHour_parallel", func(t *testing.T) { t.Parallel() testUpdateHour_parallel(t, r.Repository) }) t.Run("testHourRepository_update_existing", func(t *testing.T) { t.Parallel() testHourRepository_update_existing(t, r.Repository) }) t.Run("testUpdateHour_rollback", func(t *testing.T) { t.Parallel() testUpdateHour_rollback(t, r.Repository) }) }) } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! When we have multiple tests where we pass the same input and check the same output, it’s a good idea to use a technique known as test table. The idea is simple: define a slice of inputs and expected outputs, then iterate over it to execute tests. func testUpdateHour(t *testing.T, repository hour.Repository) { t.Helper() ctx := context.Background() testCases := []struct { Name string CreateHour func(*testing.T) *hour.Hour }{ { Name: "available_hour", CreateHour: func(t *testing.T) *hour.Hour { return newValidAvailableHour(t) }, }, { Name: "not_available_hour", CreateHour: func(t *testing.T) *hour.Hour { h := newValidAvailableHour(t) require.NoError(t, h.MakeNotAvailable()) return h }, }, { Name: "hour_with_training", CreateHour: func(t *testing.T) *hour.Hour { h := newValidAvailableHour(t) require.NoError(t, h.ScheduleTraining()) return h }, }, } for _, tc := range testCases { t.Run(tc.Name, func(t *testing.T) { newHour := tc.CreateHour(t) err := repository.UpdateHour(ctx, newHour.Time(), func(_ *hour.Hour) (*hour.Hour, error) { // UpdateHour provides us existing/new *hour.Hour, // but we are ignoring this hour and persisting result of `CreateHour` // we can assert this hour later in assertHourInRepository return newHour, nil }) require.NoError(t, err) assertHourInRepository(ctx, t, repository, newHour) }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source You can see that we used the popular github.com/stretchr/testify library. It significantly reduces boilerplate in tests by providing multiple helpers for asserts. Note require.NoError() When assert.NoError assert fails, test execution is not interrupted. It’s worth mentioning that asserts from the require package stop execution of the test when they fail. Because of that, it’s often a good idea to use require for checking errors. In many cases, if some operation fails, it doesn’t make sense to check anything later. When we assert multiple values, assert is a better choice because you will receive more context. If we have more specific data to assert, it’s always a good idea to add some helpers. It removes a lot of duplication and improves test readability! func assertHourInRepository(ctx context.Context, t *testing.T, repo hour.Repository, hour *hour.Hour) { require.NotNil(t, hour) hourFromRepo, err := repo.GetOrCreateHour(ctx, hour.Time()) require.NoError(t, err) assert.Equal(t, hour, hourFromRepo) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source Testing transactions Mistakes taught me that I should not trust myself when implementing complex code. Sometimes we don’t understand the documentation or just introduce some stupid mistake. You can gain confidence in two ways: TDD - let’s start with a test that will check if the transaction is working properly. Let’s start with the implementation and add tests later. func testUpdateHour_rollback(t *testing.T, repository hour.Repository) { t.Helper() ctx := context.Background() hourTime := newValidHourTime() err := repository.UpdateHour(ctx, hourTime, func(h *hour.Hour) (*hour.Hour, error) { require.NoError(t, h.MakeAvailable()) return h, nil }) err = repository.UpdateHour(ctx, hourTime, func(h *hour.Hour) (*hour.Hour, error) { assert.True(t, h.IsAvailable()) require.NoError(t, h.MakeNotAvailable()) return h, errors.New("something went wrong") }) require.Error(t, err) persistedHour, err := repository.GetOrCreateHour(ctx, hourTime) require.NoError(t, err) assert.True(t, persistedHour.IsAvailable(), "availability change was persisted, not rolled back") } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source When I’m not using TDD, I try to be paranoid about whether the test implementation is valid. To be more confident, I use a technique that I call test sabotage. The method is simple: break the implementation that we’re testing and see if anything fails. func (m MySQLHourRepository) finishTransaction(err error, tx *sqlx.Tx) error { - if err != nil { - if rollbackErr := tx.Rollback(); rollbackErr != nil { - return multierr.Combine(err, rollbackErr) - } - - return err - } else { - if commitErr := tx.Commit(); commitErr != nil { - return errors.Wrap(err, "failed to commit tx") - } - - return nil + if commitErr := tx.Commit(); commitErr != nil { + return errors.Wrap(err, "failed to commit tx") } + + return nil } If your tests are passing after a change like that, I have bad news… Testing database race conditions Our applications don’t work in a void. It can always happen that multiple clients try to do the same operation, and only one can win! In our case, the typical scenario is when two clients try to schedule a training at the same time. We can have only one training scheduled in one hour. This constraint is achieved by optimistic locking (described in the previous article) and domain constraints (described two articles ago). Let’s verify if it’s possible to schedule one hour more than once. The idea is simple: create 20 goroutines, release them all at once, and try to schedule training. We expect exactly one worker to succeed. func testUpdateHour_parallel(t *testing.T, repository hour.Repository) { // ... workersCount := 20 workersDone := sync.WaitGroup{} workersDone.Add(workersCount) // closing startWorkers will unblock all workers at once, // thanks to that it will be more likely to have race condition startWorkers := make(chan struct{}) // if training was successfully scheduled, number of the worker is sent to this channel trainingsScheduled := make(chan int, workersCount) // we are trying to do race condition, in practice only one worker should be able to finish transaction for worker := 0; worker < workersCount; worker++ { workerNum := worker go func() { defer workersDone.Done() <-startWorkers schedulingTraining := false err := repository.UpdateHour(ctx, hourTime, func(h *hour.Hour) (*hour.Hour, error) { // training is already scheduled, nothing to do there if h.HasTrainingScheduled() { return h, nil } // training is not scheduled yet, so let's try to do that if err := h.ScheduleTraining(); err != nil { return nil, err } schedulingTraining = true return h, nil }) if schedulingTraining && err == nil { // training is only scheduled if UpdateHour didn't return an error trainingsScheduled <- workerNum } }() } close(startWorkers) // we are waiting, when all workers did the job workersDone.Wait() close(trainingsScheduled) var workersScheduledTraining []int for workerNum := range trainingsScheduled { workersScheduledTraining = append(workersScheduledTraining, workerNum) } assert.Len(t, workersScheduledTraining, 1, "only one worker should schedule training") } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source This is also a good example of use cases that are easier to test at the integration test level, not at acceptance or E2E level. Tests like this as E2E would be really heavy, and you would need more workers to ensure they execute transactions simultaneously. Making tests fast If your tests can’t be executed in parallel, they will be slow. Even on the best machine. Is putting t.Parallel() enough? Well, we need to ensure that our tests are independent. In our case, if two tests tried to edit the same hour, they could fail randomly. This is a highly undesirable situation. To achieve that, I created the newValidHourTime() function that provides a random hour unique to the current test run. In most applications, generating a unique UUID for your entities may be enough. In some situations, it may be less obvious but still not impossible. I encourage you to spend some time finding the solution. Please treat it as an investment in your and your teammates’ mental health 😉. // usedHours is storing hours used during the test, // to ensure that within one test run we are not using the same hour // (it should be not a problem between test runs) var usedHours = sync.Map{} func newValidHourTime() time.Time { for { minTime := time.Now().AddDate(0, 0, 1) minTimestamp := minTime.Unix() maxTimestamp := minTime.AddDate(0, 0, testHourFactory.Config().MaxWeeksInTheFutureToSet*7).Unix() t := time.Unix(rand.Int63n(maxTimestamp-minTimestamp)+minTimestamp, 0).Truncate(time.Hour).Local() _, alreadyUsed := usedHours.LoadOrStore(t.Unix(), true) if !alreadyUsed { return t } } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_repository_test.go Full source Another good thing about making our tests independent is that there’s no need for data cleanup. In my experience, doing data cleanup is always messy because: when it doesn’t work correctly, it creates hard-to-debug issues in tests, it makes tests slower, it adds overhead to development (you need to remember to update the cleanup function), it may make running tests in parallel harder. It may also happen that we’re not able to run tests in parallel. Two common examples are: pagination: if you iterate over pages, other tests can insert something in-between and move “items” in the pages. global counters: like with pagination, other tests may affect the counter in an unexpected way. In that case, it’s worth keeping these tests as short as we can. Please, don’t use sleep in tests! The last tip about what makes tests flaky and slow: putting the sleep function in them. Please, don’t! It’s much better to synchronize your tests with channels or sync.WaitGroup{}. They are faster and more stable that way. If you really need to wait for something, it’s better to use assert.Eventually instead of a sleep. Note Eventually asserts that given condition will be met in waitFor time, periodically checking target function each tick. assert.Eventually( t, func() bool { return true }, // condition time.Second, // waitFor 10*time.Millisecond, // tick ) Source: godoc.org/github.com/stretchr/testify/assert Running Now that our tests are implemented, it’s time to run them! Before that, we need to start our container with Firebase and MySQL using docker-compose up. I prepared a make test command that runs tests in a consistent way (for example, with the -race flag). It can also be used in CI. $ make test ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/auth [] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/client [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/genproto/trainer [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/genproto/users [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/logs [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/server [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/server/httperr [no test files] ok github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer 0.172s ok github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour 0.031s ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings [no test files] ? github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/users [no test files] Running one test and passing custom params If you would like to pass some extra params, to have a verbose output (-v) or execute exact test (-run), you can pass it after make test --. $ make test -- -v -run ^TestRepository/memory/testUpdateHour$ --- PASS: TestRepository (0.00s) --- PASS: TestRepository/memory (0.00s) --- PASS: TestRepository/memory/testUpdateHour (0.00s) --- PASS: TestRepository/memory/testUpdateHour/available_hour (0.00s) --- PASS: TestRepository/memory/testUpdateHour/not_available_hour (0.00s) --- PASS: TestRepository/memory/testUpdateHour/hour_with_training (0.00s) PASS If you are interested in how it is implemented, I’d recommend you check my Makefile magic :man_mage:. Debugging Sometimes our tests fail in an unclear way. In that case, it’s useful to be able to easily check what data is in our database. For SQL databases, my first choice for that is mycli for MySQL and pgcli for PostgreSQL. I’ve added a make mycli command to the Makefile, so you don’t need to pass credentials all the time. $ make mycli mysql user@localhost:db> SELECT * from `hours`; +---------------------+--------------------+ | hour | availability | |---------------------+--------------------| | 2020-08-31 15:00:00 | available | | 2020-09-13 19:00:00 | training_scheduled | | 2022-07-19 19:00:00 | training_scheduled | | 2023-03-19 14:00:00 | available | | 2023-08-05 03:00:00 | training_scheduled | | 2024-01-17 07:00:00 | not_available | | 2024-02-07 15:00:00 | available | | 2024-05-07 18:00:00 | training_scheduled | | 2024-05-22 09:00:00 | available | | 2025-03-04 15:00:00 | available | | 2025-04-15 08:00:00 | training_scheduled | | 2026-05-22 09:00:00 | training_scheduled | | 2028-01-24 18:00:00 | not_available | | 2028-07-09 00:00:00 | not_available | | 2029-09-23 15:00:00 | training_scheduled | +---------------------+--------------------+ 15 rows in set Time: 0.025s For Firestore, the emulator is exposing the UI at localhost:4000/firestore. First step for having a well-tested application The biggest gap we currently have is a lack of tests at the component and E2E level. Also, a big part of the application is not tested at all. We will fix that in the next articles. We will also cover some topics that we skipped this time. But before that, we have one topic that we need to cover first: Clean/Hexagonal architecture! This approach will help us organize our application and make future refactoring and features easier to implement. Just to remind, the entire source code of Wild Workouts is available on GitHub. You can run it locally and deploy to Google Cloud with one command. Did you like this article and haven’t had a chance to read the previous ones? There are 14 more articles to check! And that’s all for today. See you soon! 🙂

I’ve seen a lot of complicated code in my life. Pretty often, the reason for that complexity was application logic coupled with database logic. Keeping the logic of your application together with your database logic makes your application much more complex, harder to test, and harder to maintain. There is already a proven and simple pattern that solves these issues. This pattern allows you to separate your application logic from database logic. It allows you to make your code simpler and easier to extend with new functionalities. As a bonus, you can defer the important decision of choosing a database solution and schema. Another good side effect of this approach is out-of-the-box immunity to database vendor lock-in. The pattern that I have in mind is the repository. When I think back to the applications I worked with, I remember that it was tough to understand how they worked. I was always afraid to make any changes: you never knew what unexpected, bad side effects it could have. It’s hard to understand the application logic when it’s mixed with the database implementation. It’s also a source of duplication. One rescue here may be building end-to-end tests. But that hides the problem instead of solving it. Having a lot of E2E tests is slow, flaky, and hard to maintain. Sometimes they even prevent us from creating new functionality rather than help. In this article, I will teach you how to apply this pattern in Go in a pragmatic, elegant, and straightforward way. I will also deeply cover a topic that is often skipped - clean transactions handling. To prove that I prepared 3 implementations: Firestore, MySQL, and simple in-memory. Without a long introduction, let’s jump to the practical examples! Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Repository interface The idea of using the repository pattern is: Let’s abstract our database implementation by defining the interaction with it through an interface. You need to be able to use this interface for any database implementation: that means it should be free of any implementation details of any specific database. Let’s start with refactoring the trainer service. Currently, the service allows us to get information about hour availability via HTTP API and via gRPC. We can also change the hour’s availability via HTTP API and gRPC. In the previous article, we refactored Hour to use the DDD Lite approach. Thanks to that, we don’t need to think about enforcing rules for when Hour can be updated. Our domain layer ensures that we can’t do anything “stupid”. We also don’t need to think about validation. We can just use the type and execute the necessary operations. We need to be able to get the current state of Hour from the database and save it. Also, if two people try to schedule a training simultaneously, only one person should be able to schedule training for that hour. Let’s reflect our needs in the interface: package hour type Repository interface { GetOrCreateHour(ctx context.Context, hourTime time.Time) (*Hour, error) UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, error), ) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/repository.go Full source We will use GetOrCreateHour to get the data, and UpdateHour to save the data. We define the interface in the same package as the Hour type. Thanks to that, we can avoid duplication when using this interface in many modules (from my experience, this may often be the case). It’s also a similar pattern to io.Writer, where the io package defines the interface and all implementations are decoupled in separate packages. How to implement that interface? Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Reading the data Most database drivers can use ctx context.Context for cancellation, tracing, etc. It’s not specific to any database (it’s a common Go concept), so you shouldn’t be afraid that you’ll spoil the domain. 😉 In most cases, we query data by using UUID or ID rather than time.Time. In our case, it’s okay: the hour is unique by design. I can imagine a situation where we would want to support multiple trainers. In that case, this assumption would not be valid. Changing to UUID/ID would still be simple. But for now, YAGNI! For clarity, this is how an interface based on UUID might look: GetOrCreateHour(ctx context.Context, hourUUID string) (*Hour, error) Note You can find an example of a repository based on UUID in Combining DDD, CQRS, and Clean Architecture article. How is the interface used in the application? import ( // ... "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour" // ... ) type GrpcServer struct { hourRepository hour.Repository } // ... func (g GrpcServer) IsHourAvailable(ctx context.Context, request *trainer.IsHourAvailableRequest) (*trainer.IsHourAvailableResponse, error) { trainingTime, err := protoTimestampToTime(request.Time) if err != nil { return nil, status.Error(codes.InvalidArgument, "unable to parse time") } h, err := g.hourRepository.GetOrCreateHour(ctx, trainingTime) if err != nil { return nil, status.Error(codes.Internal, err.Error()) } return &trainer.IsHourAvailableResponse{IsAvailable: h.IsAvailable()}, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/grpc.go Full source No rocket science! We get hour.Hour and check if it’s available. Can you guess what database we use? No, and that’s the point! As I mentioned, we can avoid vendor lock-in and easily swap the database. If you can swap the database, it’s a sign that you implemented the repository pattern correctly. In practice, changing the database is rare. 😉 When you’re using a solution that is not self-hosted (like Firestore), it’s more important to mitigate the risk and avoid vendor lock-in. A helpful side effect is that we can defer the decision of which database implementation to use. I call this approach Domain First. I described it in depth in the previous article. Deferring the decision about the database for later can save time at the beginning of the project. With more information and context, we can also make a better decision. When we use the Domain-First approach, the first and simplest repository implementation may be an in-memory implementation. Example In-memory implementation Our memory uses a simple map under the hood. getOrCreateHour has 5 lines (without a comment and one newline)! 😉 type MemoryHourRepository struct { hours map[time.Time]hour.Hour lock *sync.RWMutex hourFactory hour.Factory } func NewMemoryHourRepository(hourFactory hour.Factory) *MemoryHourRepository { if hourFactory.IsZero() { panic("missing hourFactory") } return &MemoryHourRepository{ hours: map[time.Time]hour.Hour{}, lock: &sync.RWMutex{}, hourFactory: hourFactory, } } func (m MemoryHourRepository) GetOrCreateHour(_ context.Context, hourTime time.Time) (*hour.Hour, error) { m.lock.RLock() defer m.lock.RUnlock() return m.getOrCreateHour(hourTime) } func (m MemoryHourRepository) getOrCreateHour(hourTime time.Time) (*hour.Hour, error) { currentHour, ok := m.hours[hourTime] if !ok { return m.hourFactory.NewNotAvailableHour(hourTime) } // we don't store hours as pointers, but as values // thanks to that, we are sure that nobody can modify Hour without using UpdateHour return &currentHour, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_memory_repository.go Full source Unfortunately, the memory implementation has some downsides. The biggest one is that it doesn’t keep the data after a service restart. 😉 It can be enough for a functional pre-alpha version. To make our application production-ready, we need something a bit more persistent. Example MySQL implementation We already know what our model looks like and how it behaves. Based on that, let’s define our SQL schema. CREATE TABLE `hours` ( hour TIMESTAMP NOT NULL, availability ENUM ('available', 'not_available', 'training_scheduled') NOT NULL, PRIMARY KEY (hour) ); Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/sql/schema.sql Full source When I work with SQL databases, my default choices are: sqlx: for simpler data models, it provides useful functions that help with using structs to unmarshal query results. When the schema is more complex because of relations and multiple models, it’s time for… SQLBoiler: excellent for more complex models with many fields and relations, it’s based on code generation. Thanks to that, it’s very fast, and you don’t need to be afraid that you passed an invalid interface{} instead of another interface{}. 😉 Generated code is based on the SQL schema, so you can avoid a lot of duplication. Note Update (2026): SQLBoiler is now in maintenance mode. It still receives updates, and we continue using it in existing projects, but our default choice for new projects is sqlc. Like SQLBoiler, sqlc generates type-safe Go code from SQL, but it’s under more active development and has a growing ecosystem. Check out our list of recommended Go libraries for more details. We currently have only one table. sqlx will be more than enough. 😉 Let’s reflect our DB model with a “transport type”. type mysqlHour struct { ID string `db:"id"` Hour time.Time `db:"hour"` Availability string `db:"availability"` } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source Note You may ask why not add the `db` attribute to `hour.Hour`. From my experience, it's better to entirely separate domain types from the database. It's easier to test, we don't duplicate validation, and it doesn't introduce a lot of boilerplate. In case of any change in the schema, we can do it just in our repository implementation, not in half of the project. Miłosz described a similar case in Things to know about DRY article. I also described that rule deeper in the previous article about DDD Lite. How can we use this struct? // sqlContextGetter is an interface provided both by transaction and standard db connection type sqlContextGetter interface { GetContext(ctx context.Context, dest interface{}, query string, args ...interface{}) error } func (m MySQLHourRepository) GetOrCreateHour(ctx context.Context, time time.Time) (*hour.Hour, error) { return m.getOrCreateHour(ctx, m.db, time, false) } func (m MySQLHourRepository) getOrCreateHour( ctx context.Context, db sqlContextGetter, hourTime time.Time, forUpdate bool, ) (*hour.Hour, error) { dbHour := mysqlHour{} query := "SELECT * FROM `hours` WHERE `hour` = ?" if forUpdate { query += " FOR UPDATE" } err := db.GetContext(ctx, &dbHour, query, hourTime.UTC()) if errors.Is(err, sql.ErrNoRows) { // in reality this date exists, even if it's not persisted return m.hourFactory.NewNotAvailableHour(hourTime) } else if err != nil { return nil, errors.Wrap(err, "unable to get hour from db") } availability, err := hour.NewAvailabilityFromString(dbHour.Availability) if err != nil { return nil, err } domainHour, err := m.hourFactory.UnmarshalHourFromDatabase(dbHour.Hour.Local(), availability) if err != nil { return nil, err } return domainHour, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source With the SQL implementation, it’s simple because we don’t need to maintain backward compatibility. In previous articles, we used Firestore as our primary database. Let’s prepare an implementation based on that, keeping backward compatibility. Firestore implementation Note Since writing this article, our recommendation has changed: just use PostgreSQL. It’s easier to operate and simpler to migrate between cloud providers if needed. See the State of this article in 2026 note in our first post for more context. When you want to refactor a legacy application, abstracting the database may be a good starting point. Sometimes, applications are built in a database-centric way. In our case, it’s an HTTP Response-centric approach 😉: our database models are based on Swagger-generated models. In other words, our data models are based on Swagger data models that are returned by the API. Does it stop us from abstracting the database? Of course not! It will just need some extra code to handle unmarshaling. With the Domain-First approach, our database model would be much better, like in the SQL implementation. But we are where we are. Let’s cut this old legacy step by step. I also have a feeling that CQRS will help us with that. 😉 Note In practice, a migration of the data may be simple, as long as no other services are integrated directly via the database. Unfortunately, it’s an optimistic assumption when we work with a legacy response/database-centric or CRUD service… func (f FirestoreHourRepository) GetOrCreateHour(ctx context.Context, time time.Time) (*hour.Hour, error) { date, err := f.getDateDTO( // getDateDTO should be used both for transactional and non transactional query, // the best way for that is to use closure func() (doc *firestore.DocumentSnapshot, err error) { return f.documentRef(time).Get(ctx) }, time, ) if err != nil { return nil, err } hourFromDb, err := f.domainHourFromDateDTO(date, time) if err != nil { return nil, err } return hourFromDb, err } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_firestore_repository.go Full source // for now we are keeping backward comparability, because of that it's a bit messy and overcomplicated // todo - we will clean it up later with CQRS :-) func (f FirestoreHourRepository) domainHourFromDateDTO(date Date, hourTime time.Time) (*hour.Hour, error) { firebaseHour, found := findHourInDateDTO(date, hourTime) if !found { // in reality this date exists, even if it's not persisted return f.hourFactory.NewNotAvailableHour(hourTime) } availability, err := mapAvailabilityFromDTO(firebaseHour) if err != nil { return nil, err } return f.hourFactory.UnmarshalHourFromDatabase(firebaseHour.Hour.Local(), availability) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_firestore_repository.go Full source Unfortunately, the Firebase interfaces for transactional and non-transactional queries are not fully compatible. To avoid duplication, I created getDateDTO, which can handle this difference by passing getDocumentFn. func (f FirestoreHourRepository) getDateDTO( getDocumentFn func() (doc *firestore.DocumentSnapshot, err error), dateTime time.Time, ) (Date, error) { doc, err := getDocumentFn() if status.Code(err) == codes.NotFound { // in reality this date exists, even if it's not persisted return NewEmptyDateDTO(dateTime), nil } if err != nil { return Date{}, err } date := Date{} if err := doc.DataTo(&date); err != nil { return Date{}, errors.Wrap(err, "unable to unmarshal Date from Firestore") } return date, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_firestore_repository.go Full source Even if some extra code is needed, it’s not bad. And at least it’s easy to test. Updating the data As I mentioned before, it’s critical to ensure that only one person can schedule a training in one hour. To handle that scenario, we need to use optimistic locking and transactions. Even if transactions is a pretty common term, let’s ensure we’re on the same page with Optimistic Locking. Optimistic concurrency control assumes that many transactions can frequently complete without interfering with each other. While running, transactions use data resources without acquiring locks on those resources. Before committing, each transaction verifies that no other transaction has modified the data it has read. If the check reveals conflicting modifications, the committing transaction rolls back and can be restarted. wikipedia.org Technically, transaction handling is not complicated. The biggest challenge I had was a bit different: how to manage transactions in a clean way that does not affect the rest of the application too much, is not dependent on the implementation, and is explicit and fast. I experimented with many ideas, like passing a transaction via context.Context, handling transactions at the HTTP/gRPC/message middleware level, etc. All approaches I tried had many major issues: they were a bit magical, not explicit, and slow in some cases. Currently, my favorite approach is based on a closure passed to the update function. type Repository interface { // ... UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, error), ) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/repository.go Full source The basic idea is that when we run UpdateHour, we need to provide an updateFn that can update the provided hour. So in practice, in one transaction we: get and provide all parameters for updateFn (h *Hour in our case) based on provided UUID or any other parameter (in our case hourTime time.Time) execute the closure (updateFn in our case) save return values (*Hour in our case, if needed we can return more) execute a rollback in case of an error returned from the closure How is it used in practice? func (g GrpcServer) MakeHourAvailable(ctx context.Context, request *trainer.UpdateHourRequest) (*trainer.EmptyResponse, error) { trainingTime, err := protoTimestampToTime(request.Time) if err != nil { return nil, status.Error(codes.InvalidArgument, "unable to parse time") } if err := g.hourRepository.UpdateHour(ctx, trainingTime, func(h *hour.Hour) (*hour.Hour, error) { if err := h.MakeAvailable(); err != nil { return nil, err } return h, nil }); err != nil { return nil, status.Error(codes.Internal, err.Error()) } return &trainer.EmptyResponse{}, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/grpc.go Full source As you can see, we get an Hour instance from some (unknown!) database. After that, we make this hour Available. If everything is fine, we save the hour by returning it. As part of the previous article, all validations were moved to the domain level, so here we’re sure that we aren’t doing anything “stupid”. It also simplified this code a lot. +func (g GrpcServer) MakeHourAvailable(ctx context.Context, request *trainer.UpdateHourRequest) (*trainer.EmptyResponse, error) { @ ... -func (g GrpcServer) UpdateHour(ctx context.Context, req *trainer.UpdateHourRequest) (*trainer.EmptyResponse, error) { - trainingTime, err := grpcTimestampToTime(req.Time) - if err != nil { - return nil, status.Error(codes.InvalidArgument, "unable to parse time") - } - - date, err := g.db.DateModel(ctx, trainingTime) - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("unable to get data model: %s", err)) - } - - hour, found := date.FindHourInDate(trainingTime) - if !found { - return nil, status.Error(codes.NotFound, fmt.Sprintf("%s hour not found in schedule", trainingTime)) - } - - if req.HasTrainingScheduled && !hour.Available { - return nil, status.Error(codes.FailedPrecondition, "hour is not available for training") - } - - if req.Available && req.HasTrainingScheduled { - return nil, status.Error(codes.FailedPrecondition, "cannot set hour as available when it have training scheduled") - } - if !req.Available && !req.HasTrainingScheduled { - return nil, status.Error(codes.FailedPrecondition, "cannot set hour as unavailable when it have no training scheduled") - } - hour.Available = req.Available - - if hour.HasTrainingScheduled && hour.HasTrainingScheduled == req.HasTrainingScheduled { - return nil, status.Error(codes.FailedPrecondition, fmt.Sprintf("hour HasTrainingScheduled is already %t", hour.HasTrainingScheduled)) - } - - hour.HasTrainingScheduled = req.HasTrainingScheduled - if err := g.db.SaveModel(ctx, date); err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("failed to save date: %s", err)) - } - - return &trainer.EmptyResponse{}, nil -} Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/commit/0249977c58a310d343ca2237c201b9ba016b148e#diff-5e57cb39050b6e252711befcf6fb0a89L20 Full source In our case, from updateFn we return only (*Hour, error): you can return more values if needed. You can return event sourcing events, read models, etc. We could also, in theory, use the same hour.*Hour that we provide to updateFn. I decided not to do that. Using the returned value gives us more flexibility (we can replace it with a different instance of hour.*Hour if we want). It’s also fine to have multiple UpdateHour-like functions created with extra data to save. Under the hood, the implementation should reuse the same code without a lot of duplication. type Repository interface { // ... UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, error), ) error UpdateHourWithMagic( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, *Magic, error), ) error } How to implement it now? In-memory transactions implementation The memory implementation is again the simplest one. 😉 We need to get the current value, execute the closure, and save the result. What’s essential is that in the map, we store a copy instead of a pointer. Thanks to that, we’re sure that without the “commit” (m.hours[hourTime] = *updatedHour) our values are not saved. We’ll double-check this in tests. func (m *MemoryHourRepository) UpdateHour( _ context.Context, hourTime time.Time, updateFn func(h *hour.Hour) (*hour.Hour, error), ) error { m.lock.Lock() defer m.lock.Unlock() currentHour, err := m.getOrCreateHour(hourTime) if err != nil { return err } updatedHour, err := updateFn(currentHour) if err != nil { return err } m.hours[hourTime] = *updatedHour return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_memory_repository.go Full source Firestore transactions implementation The Firestore implementation is a bit more complex, but again, it’s related to backward compatibility. Functions getDateDTO, domainHourFromDateDTO, updateHourInDataDTO could probably be skipped if our data model were better. Another reason not to use a Database-centric/Response-centric approach! func (f FirestoreHourRepository) UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *hour.Hour) (*hour.Hour, error), ) error { err := f.firestoreClient.RunTransaction(ctx, func(ctx context.Context, transaction *firestore.Transaction) error { dateDocRef := f.documentRef(hourTime) firebaseDate, err := f.getDateDTO( // getDateDTO should be used both for transactional and non transactional query, // the best way for that is to use closure func() (doc *firestore.DocumentSnapshot, err error) { return transaction.Get(dateDocRef) }, hourTime, ) if err != nil { return err } hourFromDB, err := f.domainHourFromDateDTO(firebaseDate, hourTime) if err != nil { return err } updatedHour, err := updateFn(hourFromDB) if err != nil { return errors.Wrap(err, "unable to update hour") } updateHourInDataDTO(updatedHour, &firebaseDate) return transaction.Set(dateDocRef, firebaseDate) }) return errors.Wrap(err, "firestore transaction failed") } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_firestore_repository.go Full source As you can see, we get *hour.Hour, call updateFn, and save the results inside RunTransaction. Despite some extra complexity, this implementation is still clear, easy to understand and test. MySQL transactions implementation Let’s compare it with the MySQL implementation, where we designed models in a better way. Even though the implementation approach is similar, the biggest difference is how transactions are handled. In the SQL driver, the transaction is represented by *db.Tx. We use this particular object to call all queries and do rollback and commit. To ensure that we don’t forget about closing the transaction, we do rollback and commit in the defer. func (m MySQLHourRepository) UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *hour.Hour) (*hour.Hour, error), ) (err error) { tx, err := m.db.Beginx() if err != nil { return errors.Wrap(err, "unable to start transaction") } // Defer is executed on function just before exit. // With defer, we are always sure that we will close our transaction properly. defer func() { // In `UpdateHour` we are using named return - `(err error)`. // Thanks to that that can check if function exits with error. // // Even if function exits without error, commit still can return error. // In that case we can override nil to err `err = m.finish...`. err = m.finishTransaction(err, tx) }() existingHour, err := m.getOrCreateHour(ctx, tx, hourTime, true) if err != nil { return err } updatedHour, err := updateFn(existingHour) if err != nil { return err } if err := m.upsertHour(tx, updatedHour); err != nil { return err } return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source In this case, we also get the hour by passing forUpdate == true to getOrCreateHour. This flag adds the FOR UPDATE statement to our query. The FOR UPDATE statement is critical because without it, we would allow parallel transactions to change the hour. SELECT ... FOR UPDATE For index records the search encounters, locks the rows and any associated index entries, the same as if you issued an UPDATE statement for those rows. Other transactions are blocked from updating those rows. dev.mysql.com func (m MySQLHourRepository) getOrCreateHour( ctx context.Context, db sqlContextGetter, hourTime time.Time, forUpdate bool, ) (*hour.Hour, error) { dbHour := mysqlHour{} query := "SELECT * FROM `hours` WHERE `hour` = ?" if forUpdate { query += " FOR UPDATE" } // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source I never sleep well when I don’t have an automated test for code like that. Let’s address it later. 😉 finishTransaction is executed when UpdateHour exits. When commit or rollback fails, we can also override the returned error. // finishTransaction rollbacks transaction if error is provided. // If err is nil transaction is committed. // // If the rollback fails, we are using multierr library to add error about rollback failure. // If the commit fails, commit error is returned. func (m MySQLHourRepository) finishTransaction(err error, tx *sqlx.Tx) error { if err != nil { if rollbackErr := tx.Rollback(); rollbackErr != nil { return multierr.Combine(err, rollbackErr) } return err } else { if commitErr := tx.Commit(); commitErr != nil { return errors.Wrap(err, "failed to commit tx") } return nil } } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source // upsertHour updates hour if hour already exists in the database. // If your doesn't exists, it's inserted. func (m MySQLHourRepository) upsertHour(tx *sqlx.Tx, hourToUpdate *hour.Hour) error { updatedDbHour := mysqlHour{ Hour: hourToUpdate.Time().UTC(), Availability: hourToUpdate.Availability().String(), } _, err := tx.NamedExec( `INSERT INTO hours (hour, availability) VALUES (:hour, :availability) ON DUPLICATE KEY UPDATE availability = :availability`, updatedDbHour, ) if err != nil { return errors.Wrap(err, "unable to upsert hour") } return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/hour_mysql_repository.go Full source Summary Even if the repository approach adds a bit more code, it’s totally worth making that investment. In practice, you may spend 5 minutes more to do that, and the investment should pay off shortly. In this article, we’re missing one essential part: tests. Now adding tests should be much easier, but it still may not be obvious how to do it properly. To not make a “monster” article, I will cover it in the next 1-2 weeks. 🙂 The entire diff of this refactoring, including tests, is available on GitHub. And just to remind – you can also run the application with one command and find the entire source code on GitHub! Another technique that works pretty well is Clean/Hexagonal architecture: Miłosz covers that in Introducing Clean Architecture. And to see how to combine repositories with transactions in Clean Architecture, see Database Transactions in Go with Layered Architecture. Do you think this is helpful in your application? Are you already using the repository pattern differently? Let us know in the comments!

When I started working in Go, the community did not look favorably on techniques like DDD (Domain-Driven Design) and Clean Architecture. I heard multiple times: “Don’t do Java in Golang!”, “I’ve seen that in Java, please don’t!”. At the time, I already had almost 10 years of experience in PHP and Python. I’d seen too many bad things there. I remember all those “Eight-thousanders” (methods with 8k+ lines of code 😉) and applications that nobody wanted to maintain. When I checked the git history of these ugly monsters, they all looked harmless in the beginning. But over time, small, innocent problems grew more significant and more serious. I’d also seen how DDD and Clean Architecture solved these issues. Maybe Golang is different? Maybe writing microservices in Golang will fix this issue? It was supposed to be so beautiful Now, after exchanging experience with many people and seeing a lot of codebases, my point of view is clearer than 3 years ago. Unfortunately, I’m far from thinking that just using Go and microservices will save us from all the issues I encountered earlier. I started having flashbacks from the old, bad times. It’s less visible because of relatively younger codebases. It’s less visible because of Go’s design. But I’m sure that over time, we’ll have more and more legacy Go applications that nobody wants to maintain. Fortunately, 3 years ago, despite the chilly reception, I didn’t give up. I decided to try using DDD and related techniques that had worked for me previously in Go. With Miłosz, we led teams for 3 years that were all successfully using DDD, Clean Architecture, and other not-popular-enough techniques in Go. They gave us the ability to develop our applications and products at constant velocity, regardless of the code’s age. It was obvious from the beginning that moving patterns 1:1 from other technologies would not work. Crucially, we did not abandon idiomatic Go code and microservices architecture: they fit together perfectly! Today I’d like to share with you the first, most straightforward technique: DDD Lite. State of DDD in Golang Before sitting down to write this article, I checked a couple of articles about DDD in Go on Google. I’ll be brutal here: they all miss the most critical points that make DDD work. If I imagined reading these articles without any DDD knowledge, I would not be encouraged to use them in my team. This superficial approach may also be why DDD is still not widely adopted in the Go community. In this series, we try to show all essential techniques in the most pragmatic way. Before describing any patterns, we start with a question: what does this give us? It’s an excellent way to challenge our current thinking. I’m sure we can change how the Go community receives these techniques. We believe they are the best way to implement complex business projects. I believe we will help establish Go as a great language for building not only infrastructure but also business software. You need to go slow, to go fast It may be tempting to implement your project in the simplest way. It’s even more tempting when you feel pressure from “the top”. We’re using microservices, though, right? If needed, we’ll just rewrite the service? I’ve heard that story many times, and it rarely had a happy ending. 😉 It’s true that taking shortcuts saves time. But only in the short term. Consider the example of tests of any kind. You can skip writing tests at the beginning of the project. You’ll obviously save some time, and management will be happy. The calculation seems simple: the project was delivered faster. But this shortcut is not worthwhile in the long term. As the project grows, your team will become afraid of making any changes. In the end, you’ll spend more total time than if you had implemented tests from the beginning. You will slow down in the long term because you sacrificed quality for a quick performance boost at the start. On the other hand, if a project is not critical and needs to ship fast, you can skip tests. It should be a pragmatic decision, not just “we know better, and we don’t create bugs”. The case is the same for DDD. When you use DDD, you need a bit more time in the beginning, but the long-term savings are enormous. However, not every project is complex enough to warrant advanced techniques like DDD. There is no quality vs. speed tradeoff in the long run. If you want to go fast in the long term, you need to maintain high quality. 'Is High Quality Software Worth the Cost?' from martinfowler.com That’s great, but do you have any evidence it works? If you asked me this question two years ago, I would have said: “Well, I feel that it works better!”. But just trusting my words may not be enough. 😉 Many tutorials show dumb ideas and claim they work without any evidence. Let’s not trust them blindly! Just a reminder: if someone has a couple thousand Twitter followers, that alone is not a reason to trust them! Fortunately, 2 years ago Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations was released. In brief, this book describes what factors affect development teams’ performance. But the reason this book became famous is that it’s not just a set of unvalidated ideas: it’s based on scientific research. I was most interested in the part showing what allows teams to be top performers. This book shows some obvious facts, like introducing DevOps, CI/CD, and loosely coupled architecture, which are all essential factors in high-performing teams. Note If things like DevOps and CI/CD are not obvious to you, you can start with these books: The Phoenix Project and The DevOps Handbook. What does Accelerate tell us about top-performing teams? We found that high performance is possible with all kinds of systems, provided that systems and the teams that build and maintain them—are loosely coupled. This key architectural property enables teams to easily test and deploy individual components or services even as the organization and the number of systems it operates grow—that is, it allows organizations to increase their productivity as they scale. So let’s use microservices, and we are done? I would not be writing this article if it was enough. 😉 Make large-scale changes to the design of their system without depending on other teams to make changes in their systems or creating significant work for other teams Complete their work without communicating and coordinating with people outside their team Deploy and release their product or service on demand, regardless of other services it depends upon Do most of their testing on demand, without requiring an integrated test environment Perform deployments during normal business hours with negligible downtime Unfortunately, in real life, many so-called service-oriented architectures don’t permit testing and deploying services independently of each other, and thus will not enable teams to achieve higher performance. […] employing the latest whizzy microservices architecture deployed on containers is no guarantee of higher performance if you ignore these characteristics. […] To achieve these characteristics, design systems are loosely coupled — that is, can be changed and validated independently of each other. Using microservices architecture and splitting services into small pieces is not enough. If done the wrong way, it adds extra complexity and slows teams down. DDD can help us here. I’m mentioning DDD multiple times. What is DDD, actually? What is DDD (Domain-Driven Design) Let’s start with the Wikipedia definition: Domain-driven design (DDD) is the concept that the structure and language of your code (class names, class methods, class variables) should match the business domain. For example, if your software processes loan applications, it might have classes such as LoanApplication and Customer, and methods such as AcceptOffer and Withdraw. Well, it’s not the perfect definition. 😅 It’s still missing some of the most important points. It’s also worth mentioning that DDD was introduced in 2003. That’s a pretty long time ago. Some distillation may help put DDD in the 2020 and Go context. Note If you are interested in some historical context on when DDD was created, you should check Tackling Complexity in the Heart of Software by the DDD creator - Eric Evans Eric Evans - the creator of DDD.Please print this and hang over the bed to receive +10 DDD blessing. My simple DDD definition is: Ensure that you solve a valid problem in the optimal way. Then implement the solution so that your business stakeholders understand it without any translation from technical language. How to achieve that? Coding is a war, to win you need a strategy! I like to say that “5 days of coding can save 15 minutes of planning”. Before starting to write any code, you should ensure that you’re solving a valid problem. This may sound obvious, but from my experience, it’s not as easy as it sounds. It’s often the case that the solution engineers create doesn’t actually solve the problem the business requested. A set of patterns that helps us in this area is called Strategic DDD patterns. From my experience, DDD Strategic Patterns are often skipped. The reason is simple: we’re all developers, and we like to write code rather than talk to the “business people”. 😉 Unfortunately, this approach of staying closed in a basement without talking to any business people has many downsides. Lack of trust from the business, lack of knowledge of how the system works (from both business and engineering sides), solving the wrong problems: these are just some of the most common issues. The good news is that in most cases, this is caused by a lack of proper techniques like Event Storming. They can benefit both sides. What’s also surprising is that talking to the business may be one of the most enjoyable parts of the work! That said, we’ll start with patterns that apply to the code. They can give us some advantages of DDD. They’ll also be useful for you sooner. Without Strategic patterns, I’d say you’ll only get 30% of the advantages that DDD can offer. We’ll return to Strategic patterns in the next articles. DDD Lite in Go After a pretty long introduction, it’s finally time to touch some code! In this article, we’ll cover some basics of Tactical Domain-Driven Design patterns in Go. Keep in mind that this is just the beginning. We’ll need a few more articles to cover the entire topic. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! One of the most crucial parts of Tactical DDD is reflecting the domain logic directly in the code. But that’s still a vague definition, and it’s not needed at this point. I also don’t want to start by describing what Value Objects, Entities, and Aggregates are. Let’s start with practical examples instead. Wild workouts I haven’t mentioned yet that we created an entire application called Wild Workouts especially for these articles. Interestingly, we introduced some subtle issues in this application to have something to refactor. If Wild Workouts looks like an application you’re working on, you’d better stay with us for a moment. 😉 Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Refactoring of trainer service The first (micro)service that we’ll refactor is trainer. We’ll leave other services untouched for now and return to them later. This service is responsible for keeping the trainer schedule and ensuring that we can have only one training scheduled in one hour. It also keeps the information about available hours (trainer’s schedule). The initial implementation was not the best. Even though there’s not a lot of logic, some parts of the code started to get messy. I also have a gut feeling, based on experience, that over time it will get worse. 😉 func (g GrpcServer) UpdateHour(ctx context.Context, req *trainer.UpdateHourRequest) (*trainer.EmptyResponse, error) { trainingTime, err := grpcTimestampToTime(req.Time) if err != nil { return nil, status.Error(codes.InvalidArgument, "unable to parse time") } date, err := g.db.DateModel(ctx, trainingTime) if err != nil { return nil, status.Error(codes.Internal, fmt.Sprintf("unable to get data model: %s", err)) } hour, found := date.FindHourInDate(trainingTime) if !found { return nil, status.Error(codes.NotFound, fmt.Sprintf("%s hour not found in schedule", trainingTime)) } if req.HasTrainingScheduled && !hour.Available { return nil, status.Error(codes.FailedPrecondition, "hour is not available for training") } if req.Available && req.HasTrainingScheduled { return nil, status.Error(codes.FailedPrecondition, "cannot set hour as available when it have training scheduled") } if !req.Available && !req.HasTrainingScheduled { return nil, status.Error(codes.FailedPrecondition, "cannot set hour as unavailable when it have no training scheduled") } hour.Available = req.Available if hour.HasTrainingScheduled && hour.HasTrainingScheduled == req.HasTrainingScheduled { return nil, status.Error(codes.FailedPrecondition, fmt.Sprintf("hour HasTrainingScheduled is already %t", hour.HasTrainingScheduled)) } hour.HasTrainingScheduled = req.HasTrainingScheduled Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/grpc.go Full source Even though it’s not the worst code ever, it reminds me of what I saw when I checked the git history of the code I worked on. I can imagine that after some time, new features will arrive and it will become much worse. It’s also hard to mock dependencies here, so there are also no unit tests. The First Rule - reflect your business logic literally While implementing your domain, you should stop thinking about structs as dummy data structures or “ORM-like” entities with a list of setters and getters. Instead, think about them as types with behavior. When you talk with your business stakeholders, they say “I’m scheduling training at 13:00”, rather than “I’m setting the attribute state to ’training scheduled’ for hour 13:00.”. They also don’t say: “you can’t set attribute status to ’training_scheduled’”. It is rather: “You can’t schedule training if the hour is not available”. How to put it directly in the code? func (h *Hour) ScheduleTraining() error { if !h.IsAvailable() { return ErrHourNotAvailable } h.availability = TrainingScheduled return nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/availability.go Full source One question that can help us with implementation is: “Will business stakeholders understand my code without any translation of technical terms?”. You can see in that snippet that even a non-technical person will be able to understand when you can schedule training. This approach’s cost is low, and it helps tackle complexity by making rules much easier to understand. Even though the change is not big, we got rid of this wall of ifs that would become much more complicated in the future. We’re also now able to easily add unit tests. What’s good: we don’t need to mock anything here. The tests also serve as documentation that helps us understand how Hour behaves. func TestHour_ScheduleTraining(t *testing.T) { h, err := hour.NewAvailableHour(validTrainingHour()) require.NoError(t, err) require.NoError(t, h.ScheduleTraining()) assert.True(t, h.HasTrainingScheduled()) assert.False(t, h.IsAvailable()) } func TestHour_ScheduleTraining_with_not_available(t *testing.T) { h := newNotAvailableHour(t) assert.Equal(t, hour.ErrHourNotAvailable, h.ScheduleTraining()) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/availability_test.go Full source Now, if anybody asks the question “When can I schedule training?”, you can quickly answer. In larger systems, the answer to this kind of question is even less obvious. Multiple times I spent hours trying to find all places where some objects were used in unexpected ways. The next rule will help us with that even more. Testing Helpers Note Testing helpers were introduced in the next article. They are not available in this article’s diff. 😉 It’s useful to have helpers in tests for creating our domain entities. For example: newExampleTrainingWithTime, newCanceledTraining, etc. This also makes our domain tests much more readable. Custom asserts, like assertTrainingsEquals, can also save a lot of duplication. The github.com/google/go-cmp library is extremely useful for comparing complex structs. It allows us to compare our domain types with private fields, skip some field validation, or implement custom validation functions. func assertTrainingsEquals(t *testing.T, tr1, tr2 *training.Training) { cmpOpts := []cmp.Option{ cmpRoundTimeOpt, cmp.AllowUnexported( training.UserType{}, time.Time{}, training.Training{}, ), } assert.True( t, cmp.Equal(tr1, tr2, cmpOpts...), cmp.Diff(tr1, tr2, cmpOpts...), ) } It’s also a good idea to provide a Must version of frequently used constructors, for example MustNewUser. Unlike normal constructors, these panic if parameters are not valid (for tests, that’s not a problem). func NewUser(userUUID string, userType UserType) (User, error) { if userUUID == "" { return User{}, errors.New("missing user UUID") } if userType.IsZero() { return User{}, errors.New("missing user type") } return User{userUUID: userUUID, userType: userType}, nil } func MustNewUser(userUUID string, userType UserType) User { u, err := NewUser(userUUID, userType) if err != nil { panic(err) } return u } The Second Rule: always keep a valid state in the memory I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended. The Rugged Manifesto The world would be better if everyone took this quote into account. I’m not without fault here either. 😉 From my observation, when you’re sure that the object you use is always valid, it helps avoid a lot of ifs and bugs. You’ll also feel much more confident knowing that you can’t do anything stupid with the current code. I have many flashbacks of being afraid to make changes because I wasn’t sure of the side effects. Developing new features is much slower without confidence that you’re using the code correctly! Our goal is to do validation in only one place (good DRY) and ensure that nobody can change the internal state of Hour. The only public API of the object should be methods describing behaviors. No dumb getters and setters! We also need to put our types in a separate package and make all attributes private. type Hour struct { hour time.Time availability Availability } // ... func NewAvailableHour(hour time.Time) (*Hour, error) { if err := validateTime(hour); err != nil { return nil, err } return &Hour{ hour: hour, availability: Available, }, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/hour.go Full source We should also ensure that we’re not breaking any rules inside our type. Bad example: h := hour.NewAvailableHour("13:00") if h.HasTrainingScheduled() { h.SetState(hour.Available) } else { return errors.New("unable to cancel training") } Good example: func (h *Hour) CancelTraining() error { if !h.HasTrainingScheduled() { return ErrNoTrainingScheduled } h.availability = Available return nil } // ... h := hour.NewAvailableHour("13:00") if err := h.CancelTraining(); err != nil { return err } The Third Rule - domain needs to be database agnostic There are multiple schools of thought here. Some say it’s fine to have the domain impacted by the database client. From our experience, keeping the domain strictly free of any database influence works best. The most important reasons are: domain types are not shaped by the database solution used: they should be shaped only by business rules we can store data in the database in a more optimal way because of Go’s design and lack of “magic” like annotations, ORMs or any database solutions have an even more significant impact Tip Domain-First approach If the project is complex enough, we can spend 2-4 weeks working on the domain layer with just an in-memory database implementation. In that case, we can explore the idea deeper and defer the decision about which database to choose. All our implementation is based on unit tests. We’ve tried this approach several times, and it always worked nicely. It’s also a good idea to set a timebox here to avoid spending too much time. Keep in mind that this approach requires a good relationship and a lot of trust from the business! If your relationship with business is far from good, Strategic DDD patterns will improve it. Been there, done that! To keep this article from getting too long, let’s just introduce the Repository interface and assume that it works. 😉 I’ll cover this topic more in-depth in the next article. type Repository interface { GetOrCreateHour(ctx context.Context, time time.Time) (*Hour, error) UpdateHour( ctx context.Context, hourTime time.Time, updateFn func(h *Hour) (*Hour, error), ) error } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/domain/hour/repository.go Full source Note You may ask why `UpdateHour` has `updateFn func(h *Hour) (*Hour, error)`: we'll use that for handling transactions nicely. You can learn more in the article about repositories. Using domain objects I did a small refactor of our gRPC endpoints to provide an API that is more “behavior-oriented” rather than CRUD. It better reflects the new characteristics of the domain. From my experience, it’s also much easier to maintain multiple small methods than one “god” method that allows us to update everything. --- a/api/protobuf/trainer.proto +++ b/api/protobuf/trainer.proto @@ -6,7 +6,9 @@ import "google/protobuf/timestamp.proto"; service TrainerService { rpc IsHourAvailable(IsHourAvailableRequest) returns (IsHourAvailableResponse) {} - rpc UpdateHour(UpdateHourRequest) returns (EmptyResponse) {} + rpc ScheduleTraining(UpdateHourRequest) returns (EmptyResponse) {} + rpc CancelTraining(UpdateHourRequest) returns (EmptyResponse) {} + rpc MakeHourAvailable(UpdateHourRequest) returns (EmptyResponse) {} } message IsHourAvailableRequest { @@ -19,9 +21,6 @@ message IsHourAvailableResponse { message UpdateHourRequest { google.protobuf.Timestamp time = 1; - - bool has_training_scheduled = 2; - bool available = 3; } message EmptyResponse {} Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/commit/0249977c58a310d343ca2237c201b9ba016b148e#diff-15fd9ad3f3992b0210090109b82c5594 Full source The implementation is now much simpler and easier to understand. We also have no logic here: just some orchestration. Our gRPC handler now has 18 lines and no domain logic! func (g GrpcServer) MakeHourAvailable(ctx context.Context, request *trainer.UpdateHourRequest) (*trainer.EmptyResponse, error) { trainingTime, err := protoTimestampToTime(request.Time) if err != nil { return nil, status.Error(codes.InvalidArgument, "unable to parse time") } if err := g.hourRepository.UpdateHour(ctx, trainingTime, func(h *hour.Hour) (*hour.Hour, error) { if err := h.MakeAvailable(); err != nil { return nil, err } return h, nil }); err != nil { return nil, status.Error(codes.Internal, err.Error()) } return &trainer.EmptyResponse{}, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/grpc.go Full source Tip No more Eight-thousanders As I remember from the old times, many Eight-thousanders were actually HTTP controllers with a lot of domain logic. By hiding complexity inside our domain types and keeping the rules I described, we can prevent uncontrolled growth here. That’s all for today I don’t want to make this article too long. Let’s go step by step! If you can’t wait, the entire working diff for the refactor is available on GitHub. In the next article, I’ll cover one part of the diff that isn’t explained here: repositories. Even though it’s still the beginning, some simplifications in our code are already visible. The current implementation of the model is also not perfect: that’s good! You’ll never implement the perfect model from the beginning. It’s better to be prepared to change this model easily than to waste time trying to make it perfect. After I added tests for the model and separated it from the rest of the application, I can change it without any fear. Can I already put that I know DDD to my CV? Not yet. I needed 3 years after I heard about DDD to connect all the dots (this was before I heard about Go). 😉 After that, I saw why all the techniques we’ll describe in the next articles are so important. But connecting the dots requires some patience and trust that it will work. It’s worth it! You won’t need 3 years like me, but we’ve currently planned about 10 articles on both strategic and tactical patterns. 😉 There are still a lot of new features and parts to refactor in Wild Workouts! I know that nowadays many people promise you can become an expert in some area after 10 minutes with an article or video. The world would be beautiful if that were possible, but in reality, it’s not that simple. Fortunately, a big part of the knowledge we share is universal and can be applied to multiple technologies, not only Go. You can treat these learnings as an investment in your career and mental health in the long term. 😉 There’s nothing better than solving the right problems without fighting unmaintainable code! What is your experience with DDD in Go? Was it good or bad? Was it different from how we’re doing it? Do you think DDD might be useful in your project? Please let us know in the comments! Do you have any colleagues who might be interested in this topic? Please share this article with them! Even if they don’t work in Go. 😉

If you’re here for the first time, this post is part of our Business Applications in Go series. Previously, we introduced Wild Workouts, our example application built with modern tools but containing some subtle anti-patterns. We added them on purpose to show common pitfalls and how to avoid them. In this post, we begin refactoring Wild Workouts. Previous articles will give you more context, but reading them isn’t necessary to understand this one. Background story Meet Susan, a software engineer bored at her current job working with legacy enterprise software. Susan started looking for a new gig and found the Wild Workouts startup, which uses serverless Go microservices. This seemed fresh and modern, so after a smooth hiring process, she started her first day at the company. There were just a few engineers on the team, so Susan’s onboarding was blazing fast. On her first day, she was assigned a task meant to make her familiar with the application. We need to store each user’s last IP address. This will enable new security features in the future, like extra confirmation when logging in from a new location. For now, we just want to keep it in the database. Susan explored the application for a while, trying to understand what was happening in each service and where to add a new field for the IP address. Finally, she discovered a User structure that she could extend. // User defines model for User. type User struct { Balance int `json:"balance"` DisplayName string `json:"displayName"` Role string `json:"role"` + LastIp string `json:"lastIp"` } It didn’t take Susan long to post her first pull request. Soon, Dave, a senior engineer, added a comment during code review. I don’t think we’re supposed to expose this field via the REST API. Susan was surprised since she was sure she’d updated the database model. Confused, she asked Dave if this was the correct place to add a new field. Dave explained that Firestore, the application’s database, stores documents marshaled from Go structures. So the User struct is compatible with both frontend responses and storage. “Thanks to this approach, you don’t need to duplicate the code. It’s enough to change the YAML definition once and regenerate the file”, he said enthusiastically. Grateful for the tip, Susan added one more change to hide the new field from the API response. diff --git a/internal/users/http.go b/internal/users/http.go index 9022e5d..cd8fbdc 100644 --- a/internal/users/http.go +++ b/internal/users/http.go @@ -27,5 +30,8 @@ func (h HttpServer) GetCurrentUser(w http.ResponseWriter, r *http.Request) { user.Role = authUser.Role user.DisplayName = authUser.DisplayName + // Don't expose the user's last IP externally + user.LastIp = nil + render.Respond(w, r, user) } One line was enough to fix this issue. That was quite a productive first day for Susan. Second thoughts Even though Susan’s solution was approved and merged, something bothered her on her commute home. Is it the right approach to keep the same struct for both API response and database model? Don’t we risk accidentally exposing users’ private details if we keep extending it as the application grows? What if we want to change only the API response without changing the database fields? What about splitting the two structures? That’s what Susan would do at her old job, but maybe these were enterprise patterns that should not be used in a Go microservice. Also, the team seemed very rigorous about the Don’t Repeat Yourself principle. Refactoring The next day, Susan explained her doubts to Dave and asked for his opinion. At first, he didn’t understand the concern and mentioned that maybe she needed to get used to the “Go way” of doing things. Susan pointed to another piece of code in Wild Workouts that used a similar ad hoc solution. She shared that, from her experience, such code can quickly get out of control. user, err := h.db.GetUser(r.Context(), authUser.UUID) if err != nil { httperr.InternalError("cannot-get-user", err, w, r) return } user.Role = authUser.Role user.DisplayName = authUser.DisplayName render.Respond(w, r, user) It seems the HTTP handler modifies the user in-place. Eventually, they agreed to discuss this again over a new PR. Susan prepared a refactoring proposal. diff --git a/internal/users/firestore.go b/internal/users/firestore.go index 7f3fca0..670bfaa 100644 --- a/internal/users/firestore.go +++ b/internal/users/firestore.go @@ -9,6 +9,13 @@ import ( "google.golang.org/grpc/status" ) +type UserModel struct { + Balance int + DisplayName string + Role string + LastIP string +} + diff --git a/internal/users/http.go b/internal/users/http.go index 9022e5d..372b5ca 100644 --- a/internal/users/http.go +++ b/internal/users/http.go @@ -1,6 +1,7 @@ - user.Role = authUser.Role - user.DisplayName = authUser.DisplayName - render.Respond(w, r, user) + userResponse := User{ + DisplayName: authUser.DisplayName, + Balance: user.Balance, + Role: authUser.Role, + } + + render.Respond(w, r, userResponse) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/commit/14d9e7badcf5a91811059d377cfa847ec7b4592f Full source This time, Susan didn’t touch the OpenAPI definition. After all, she wasn’t supposed to change the REST API. Instead, she manually created another structure, just like User, but exclusive to the database model. Then she extended it with a new field. The new solution is a bit longer in lines of code, but it removed coupling between the REST API and the database layer (all without introducing another microservice). The next time someone wants to add a field, they can do it by updating the proper structure. The Clash of Principles Dave’s biggest concern was that the second solution breaks the DRY principle and introduces boilerplate. On the other hand, Susan was afraid that the original approach violates the Single Responsibility Principle (the “S” in SOLID). Who’s right? It’s tough to come up with strict rules. Sometimes code duplication seems like boilerplate, but it’s one of the best tools to fight code coupling. It helps to ask yourself if the code using the common structure is likely to change together. If not, it’s safe to assume duplication is the right choice. Usually, DRY is better applied to behaviors, not data. For example, extracting common code to a separate function doesn’t have the downsides we’ve discussed. What’s the big deal? Is such a minor change even “architecture”? Susan introduced a small change that had consequences she didn’t anticipate. It was obvious to other engineers, but not to a newcomer. I guess you also know the feeling of being afraid to make changes in an unknown system because you can’t predict what it might trigger. If you make many wrong decisions, even small ones, they tend to compound. Eventually, developers start complaining that the application is hard to work with. The turning point is when someone mentions a “rewrite”, and suddenly you know you have a big problem. The alternative to good design is always bad design. There is no such thing as no design. Adam Judge It’s worth discussing architecture decisions before you run into issues. “No architecture” will just leave you with bad architecture. Can microservices save you? With all the benefits that microservices gave us, a dangerous idea also appeared, preached by some “how to build microservices” guides. It says that microservices will simplify your application. Because building large software projects is hard, some promise that you won’t need to worry about it if you split your application into tiny chunks. This sounds good on paper, but it misses the point of splitting software. How do you know where to put the boundaries? Will you separate a service based on each database entity? REST endpoints? Features? How do you ensure low coupling between services? The Distributed Monolith If you start with poorly separated services, you’re likely to end up with the same monolith you tried to avoid, plus network overhead and complex tooling to manage the mess (also known as a distributed monolith). You’ll just replace highly coupled modules with highly coupled services. And because everyone runs a Kubernetes cluster now, you may even think you’re following industry standards. Even if you can rewrite a single service in one afternoon, can you as quickly change how services communicate with each other? What if they’re owned by multiple teams, based on wrong boundaries? Consider how much simpler it is to refactor a single application. None of this negates other benefits of microservices, like independent deployments (crucial for Continuous Delivery) and easier horizontal scaling. As with all patterns, make sure you use the right tool for the job at the right time. Note We introduced similar issues on purpose in Wild Workouts. We will look into this in future posts and discuss other splitting techniques. You can find some of the ideas in Robert’s post: Why using Microservices or Monolith can be just a detail?. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Does this all apply to Go? Open-source Go projects are usually low-level applications and infrastructure tools. This was especially true in the early days, and it’s still hard to find good examples of Go applications that handle domain logic. By “domain logic,” I don’t mean financial applications or complex business software. If you develop any kind of web application, there’s a good chance you have some complex domain cases you need to model somehow. Following some DDD examples can make you feel like you’re no longer writing Go. I’m well aware that forcing OOP patterns straight from Java is not fun to work with. However, some language-agnostic ideas are worth considering. What’s the alternative? We spent the last few years exploring this topic. We love the simplicity of Go, but we’ve also had success with ideas from Domain-Driven Design and Clean Architecture. For some reason, developers didn’t stop talking about technical debt and legacy software with the arrival of microservices. Instead of looking for a silver bullet, we prefer to use business-oriented patterns together with microservices in a pragmatic way. We’re not done refactoring Wild Workouts yet. In one of the following posts, we’ll see how to introduce Clean Architecture to the project.

Welcome to the third and last article covering how to build “Too Modern Go application”. But don’t worry. This doesn’t mean we’re done showing you how to build applications that are easy to develop, maintain, and fun to work with in the long term. It’s actually just the beginning of a bigger series! We intentionally built the current version of the application to make it hard to maintain and develop in the future. In a subtle way. 😉 In the next article, we will start the refactoring process. We will show you how subtle issues and shortcuts can become a problem in the long term. Even adding a new feature now may not be as easy as developing greenfield features (even if the application is still pretty new). But before that, we have one important topic to cover: authentication. This is the part of the application that will not need refactoring. 😉 You shouldn’t reinvent the wheel (I hope that) you are not creating a programming language and a framework for every project you work on. Even if you do that, besides losing a lot of time, it’s not harmful. That’s not the case for authentication. Why? Let’s imagine that you are working in a company implementing a cryptocurrency exchange. At the beginning of the project, you decided to build your own authentication mechanism. The first reason is your boss, who doesn’t trust external authentication providers. The second reason is that you believe it should be simple. “You did it many times” – this is what strange voices in your head repeat every time. Have you seen the Mayday / Air Crash Investigation documentary TV series? In the Polish edition, every episode starts with something like: “Catastrophes are not a matter of coincidence, but a series of unfortunate events.” In programming, this statement is surprisingly true. In our hypothetical project, your company decided to introduce a service account that could move funds between any account. Even from a wallet that, in theory, should be offline. Of course, only temporarily. 😆 That doesn’t sound like a good idea, but it simplified customer support from the very beginning. With time, everyone forgot about this feature. Until one day, when hackers found a bug in the authentication mechanism that allowed them to hijack any account on our exchange. Including the service account. Our boss and coworkers were less happy than the hackers because they lost all the money. The problem wouldn’t be so big if we’d lost “only” the company’s money. In this case, we also lost all our customers’ money. 😉 This example may sound extreme and unlikely to happen. Is it a rare scenario? The OWASP Top Ten report lists Broken Authentication in second place! Broken Authentication. Application functions related to authentication and session management are often implemented incorrectly, allowing attackers to compromise passwords, keys, or session tokens, or to exploit other implementation flaws to assume other users’ identities temporarily or permanently. owasp.org/www-project-top-ten/ You may say: “My company is just selling toasters! Why should I care?” I’m pretty sure you still care about your company’s image. The reputational damage after an incident caused by hackers hijacking your customers’ accounts is always embarrassing. Do you still feel that you can implement perfect authentication? Even giants with hundreds of researchers and developers working just on authentication are unable to do that. In March 2020, a researcher found a Facebook vulnerability that could allow hijacking anyone’s Facebook account. Just a few months later, a bug in Apple Sign In allowed full account takeover of user accounts on third-party applications. If you’re still not convinced, consider saving your time. I’m sure your customers will be much happier to receive a long-awaited feature than fancy, custom authentication. 😉 Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Using Firebase authentication Many solutions can be used for implementing authentication. The Wild Workouts example application that we created for this article series is now hosted on Firebase hosting. We will use Firebase Authentication. There is one significant advantage of this solution: it works almost entirely out of the box, both on the backend and frontend. Using an external identity provider shouldn’t introduce significant vendor lock-in, as long as you design your integration to be easily switchable. We’ll cover how to do that in a future article. 😉 Note Authentication recommendation in 2026 As of 2026, we recommend using passwordless login combined with SSO (Google, Facebook, GitHub) for most applications. This approach offers the best balance of security and user experience: no passwords to remember or leak, and users sign in with accounts they already have. You can easily switch authentication providers when you use an Identity Provider (IDP) correctly. Design your system so the IDP handles only authentication, while your application manages authorization and user data separately. This keeps your core business logic decoupled from the IDP, making future migrations straightforward. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Note Deployment of the project is described in detail in the A complete Terraform setup of a serverless application on Google Cloud Run and Firebase article. In the first article you can find deployment tl;dr at the end. During the setup, don't forget about enabling Email/Password Sign-in provider in Authentication / Sign-in method tab in the [Firebase Console](https://console.firebase.google.com/project/_/authentication/providers)! Frontend The first thing that we need to do is the initialization of Firebase SDK. Next, in the form on the main page, we call the loginUser function. This function calls Auth.login, Auth.waitForAuthReady, and Auth.getJwtToken. The result is set to OpenAPI-generated clients by setApiClientsAuth. // ... export function loginUser(login, password) { return Auth.login(login, password) .then(function () { return Auth.waitForAuthReady() }) .then(function () { return Auth.getJwtToken(false) }) .then(token => { setApiClientsAuth(token) }) // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/src/repositories/user.js Full source Auth is a class with two implementations: Firebase and mock. We will go through the mock implementation later. Let’s focus on Firebase now. To log in, we need to call firebase.auth().signInWithEmailAndPassword. If everything is fine, auth().currentUser.getIdToken returns our JWT token. class FirebaseAuth { login(login, password) { return firebase.auth().signInWithEmailAndPassword(login, password) } waitForAuthReady() { return new Promise((resolve) => { firebase .auth() .onAuthStateChanged(function () { resolve() }); }) } getJwtToken(required) { return new Promise((resolve, reject) => { if (!firebase.auth().currentUser) { if (required) { reject('no user found') } else { resolve(null) } return } firebase.auth().currentUser.getIdToken(false) .then(function (idToken) { resolve(idToken) }) .catch(function (error) { reject(error) }); }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/src/repositories/auth.js Full source Note Firebase also provides out of the box support for logging in with most popular OAuth providers like Facebook, Gmail, or GitHub. Next, we need to set authentications['bearerAuth'].accessToken of OpenAPI-generated clients to the JWT token received from Auth.getJwtToken(false). Once we set this token on the OpenAPI clients, voila! All our requests are now authenticated. export function setApiClientsAuth(idToken) { usersClient.authentications['bearerAuth'].accessToken = idToken trainerClient.authentications['bearerAuth'].accessToken = idToken trainingsClient.authentications['bearerAuth'].accessToken = idToken } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/src/repositories/auth.js Full source Note If you are creating your own OpenAPI spec, it will not work without proper authentication definition. In Wild Workouts’ OpenAPI spec it’s already done. If you would like to know more, I recommend checking the Firebase Auth API reference. Backend The last part is using this authentication in our HTTP server. I created a simple HTTP middleware that will do this for us. This middleware does three things: Get token from HTTP header Verify token with Firebase auth client Save user data in credentials package auth import ( "context" "net/http" "strings" "firebase.google.com/go/auth" "github.com/pkg/errors" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/server/httperr" ) type FirebaseHttpMiddleware struct { AuthClient *auth.Client } func (a FirebaseHttpMiddleware) Middleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx := r.Context() bearerToken := a.tokenFromHeader(r) if bearerToken == "" { httperr.Unauthorised("empty-bearer-token", nil, w, r) return } token, err := a.AuthClient.VerifyIDToken(ctx, bearerToken) if err != nil { httperr.Unauthorised("unable-to-verify-jwt", err, w, r) return } // it's always a good idea to use custom type as context value (in this case ctxKey) // because nobody from the outside of the package will be able to override/read this value ctx = context.WithValue(ctx, userContextKey, User{ UUID: token.UID, Email: token.Claims["email"].(string), Role: token.Claims["role"].(string), DisplayName: token.Claims["name"].(string), }) r = r.WithContext(ctx) next.ServeHTTP(w, r) }) } type ctxKey int const ( userContextKey ctxKey = iota ) // ... func UserFromCtx(ctx context.Context) (User, error) { u, ok := ctx.Value(userContextKey).(User) if ok { return u, nil } return User{}, NoUserInContextError } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/auth/http.go Full source User data can now be accessed in every HTTP request by using auth.UserFromCtx function. func (h HttpServer) GetTrainings(w http.ResponseWriter, r *http.Request) { user, err := auth.UserFromCtx(r.Context()) if err != nil { httperr.Unauthorised("no-user-found", err, w, r) return } // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/http.go Full source We can also limit access to some resources based on the user role. func (h HttpServer) MakeHourAvailable(w http.ResponseWriter, r *http.Request) { user, err := auth.UserFromCtx(r.Context()) if err != nil { httperr.Unauthorised("no-user-found", err, w, r) return } if user.Role != "trainer" { httperr.Unauthorised("invalid-role", nil, w, r) return } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/http.go Full source Adding users In our case, we add users during the start of the users service. You can also add them from the Firebase UI. Unfortunately, via the UI you cannot set all required data, like claims: you need to do it via API. config := &firebase.Config{ProjectID: os.Getenv("GCP_PROJECT")} firebaseApp, err := firebase.NewApp(context.Background(), config, opts...) if err != nil { return nil, err } authClient, err := firebaseApp.Auth(context.Background()) if err != nil { return nil, err } // ... for _, user := range usersToCreate { userToCreate := (&auth.UserToCreate{}). Email(user.Email). Password("123456"). DisplayName(user.DisplayName) createdUser, err := authClient.CreateUser(context.Background(), userToCreate) // ... err = authClient.SetCustomUserClaims(context.Background(), createdUser.UID, map[string]interface{}{ "role": user.Role, }) // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/users/fixtures.go Full source Mock Authentication for local dev Note Update (2026): Since this article was written, Firebase has released the Authentication Emulator as part of the Local Emulator Suite. You can now run Firebase Authentication locally without the workarounds described below. There is high demand and a discussion going on about Firebase Authentication emulator support. Unfortunately, it doesn’t exist yet. The situation is similar to Firestore: I want to be able to run my application locally without any external dependencies. As long as there is no emulator, there is no other way than to implement a simple mock. There is nothing complicated in either implementation. The frontend’s getJwtToken is implemented by generating a JWT token with mock secret. The backend, instead of calling Firebase to verify the token, checks if the JWT was generated with mock secret. This gives us some confidence that our flow is implemented more or less correctly. But is there any way to test it with production Firebase locally? Firebase Authentication for local dev Mock authentication does not give us 100% confidence that the flow is working correctly. We should be able to test Firebase Authentication locally. To do that, you need to take some extra steps. It is not straightforward. If you are not changing anything in the authentication, you can probably skip this part. First of all, you need to generate a service account file into the repository. Note Remember to not share it with anyone! Don’t worry, it’s already in .gitignore. gcloud auth login gcloud iam service-accounts keys create service-account-file.json --project [YOUR PROJECT ID] --iam-account [YOUR PROJECT ID]@appspot.gserviceaccount.com Next, you need to uncomment all lines linking service account in docker-compose.yml volumes: - ./pkg:/pkg -# - ./service-account-file.json:$SERVICE_ACCOUNT_FILE + - ./service-account-file.json:$SERVICE_ACCOUNT_FILE working_dir: /pkg/trainer ports: - "127.0.0.1:3000:$PORT" context: ./dev/docker/app volumes: - ./pkg:/pkg -# - ./service-account-file.json:$SERVICE_ACCOUNT_FILE + - ./service-account-file.json:$SERVICE_ACCOUNT_FILE working_dir: /pkg/trainer @ ... do it for all services! Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/docker-compose.yml Full source After that, you should set GCP_PROJECT to your project id, uncomment SERIVCE_ACCOUNT_FILE and set MOCK_AUTH to false in .env -GCP_PROJECT=threedotslabs-cloudnative +GCP_PROJECT=YOUR-PROJECT-ID PORT=3000 -FIRESTORE_PROJECT_ID=threedotslabs-cloudnative +FIRESTORE_PROJECT_ID=YOUR-PROJECT-ID FIRESTORE_EMULATOR_HOST=firestore:8787 # env used by karhoo/firestore-emulator container -GCP_PROJECT_ID=threedotslabs-cloudnative +GCP_PROJECT_ID=YOUR-PROJECT-ID TRAINER_GRPC_ADDR=trainer-grpc:3000 USERS_GRPC_ADDR=users-grpc:3000 @ ... CORS_ALLOWED_ORIGINS=http://localhost:8080 -#SERVICE_ACCOUNT_FILE=/service-account-file.json -MOCK_AUTH=true +SERVICE_ACCOUNT_FILE=/service-account-file.json +MOCK_AUTH=false LOCAL_ENV=true Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/.env Full source -const MOCK_AUTH = process.env.NODE_ENV === 'development' +const MOCK_AUTH = false Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/src/repositories/auth.js Full source Now you need to go to Project Settings (you can find it after clicking the gear icon right to Project overview) and update your Firebase settings in web/public/__/firebase/init.json. - "apiKey": "", - "appId": "", - "authDomain": "", - "databaseURL": "", - "messagingSenderId": "", - "projectId": "", - "storageBucket": "" + "apiKey": "AIzaSyBW6TNG5Xdq1F7MleYml8CovOej5SbxXhw", + "authDomain": "wild-workouts-6.firebaseapp.com", + "databaseURL": "https://wild-workouts-6.firebaseio.com", + "projectId": "wild-workouts-6", + "storageBucket": "wild-workouts-6.appspot.com", + "messagingSenderId": "519519305315", + "appId": "1:519519305315:web:cde76e4847f0d95cc84256" } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/public/__/firebase/init.json Full source The last thing is stopping and starting again docker-compose to reload envs and links. And that’s all As you can see, the setup was straightforward. We were able to save a lot of time. The situation could be more complicated if we didn’t want to host our application on Firebase. Our idea from the beginning was to prepare a simple base for future articles. We didn’t exhaust the topic of authentication in this article. But for now, we don’t plan to explore authentication any deeper. Maybe you have some good or bad experiences with any authentication solution in Go? Please share in the comments, so all readers can profit from that! 👍 In this series, we would like to show you how to build applications that are easy to develop, maintain, and fun to work with in the long term. Maybe you’ve seen examples of Domain-Driven Design or Clean Architecture in Go. Most of them are not done in a pragmatic way that works within the language context. In following articles, we will show patterns that we have successfully used in teams we’ve led for a couple of years. We will also show when applying them makes sense and when it’s over-engineering or CV-Driven Development. I’m sure that it will make you rethink your point of view on these techniques. 😉 That’s all for today. The next article will be available in 1-2 weeks. Please join our newsletter if you don’t want to miss it! 😉

Welcome to the third article in the series on building business-oriented applications in Go! In this series, we show you how to build applications that are easy to develop, maintain, and fun to work with in the long term. In this article, I describe how to build robust internal communication between your services using gRPC. I also cover the extra configuration required to set up authentication and TLS for Cloud Run. Why gRPC? Let’s imagine a story that is true for many companies: Meet Dave. Dave works at a company that spent about two years building their product from scratch. During this time, they found thousands of customers who wanted to use their product. They started developing this application during the biggest microservices “boom.” It was an obvious choice to use that architecture. Now they have more than 50 microservices using HTTP calls to communicate with each other. Of course, Dave’s company didn’t do everything perfectly. The biggest pain is that all engineers are now afraid to change anything in the HTTP contracts. It’s easy to make incompatible changes or return invalid data. The entire application breaking because of this isn’t rare. “Didn’t we build microservices to avoid that?” is a question the scary voices in Dave’s head ask every day. Dave already proposed using OpenAPI to generate HTTP server responses and clients. But he quickly found that he could still return invalid data from the API. Whether this story sounds familiar or not, the solution for Dave’s company is simple and straightforward to implement. You can easily achieve robust contracts between your services by using gRPC. The way gRPC generates servers and clients is much stricter than OpenAPI. It’s far better than OpenAPI’s client and server, which just copy structures. Note gRPC doesn’t solve data quality problems. You can still send data that is not empty but doesn’t make sense. Make sure your data is valid using robust contracts, contract testing, and end-to-end tests. Another reason to consider gRPC is performance. Studies show gRPC can be up to 10x faster than REST. When your API handles millions of requests per second, this becomes a cost optimization opportunity. For applications like Wild Workouts, where traffic may be less than 10 requests/sec, it doesn’t matter. To avoid bias, I tried to find reasons not to use gRPC for internal communication. I couldn’t find any: The barrier to entry is low. Adding a gRPC server requires no extra infrastructure work: it works on top of HTTP/2. It works with many languages like Java, C/C++, Python, C#, JS, and more. In theory, you can even use gRPC for frontend communication (I haven’t tested that). It’s “Goish”: the compiler ensures you don’t return anything invalid. Sounds promising? Let’s verify this with the implementation in Wild Workouts! Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Generated server Currently, Wild Workouts doesn’t have many gRPC endpoints. We can update trainer hours availability and user training balance (credits). Let’s look at the Trainer gRPC service. To define our gRPC server, we need to create a trainer.proto file. syntax = "proto3"; package trainer; import "google/protobuf/timestamp.proto"; service TrainerService { rpc IsHourAvailable(IsHourAvailableRequest) returns (IsHourAvailableResponse) {} rpc UpdateHour(UpdateHourRequest) returns (EmptyResponse) {} } message IsHourAvailableRequest { google.protobuf.Timestamp time = 1; } message IsHourAvailableResponse { bool is_available = 1; } message UpdateHourRequest { google.protobuf.Timestamp time = 1; bool has_training_scheduled = 2; bool available = 3; } message EmptyResponse {} Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/api/protobuf/trainer.proto Full source The .proto definition is converted into Go code using the Protocol Buffer Compiler (protoc). .PHONY: proto proto: protoc --go_out=plugins=grpc:internal/common/genproto/trainer -I api/protobuf api/protobuf/trainer.proto protoc --go_out=plugins=grpc:internal/common/genproto/users -I api/protobuf api/protobuf/users.proto Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/Makefile Full source Note To generate Go code from .proto, you need to install protoc and the protoc Go Plugin. A list of supported types can be found in the Protocol Buffers Version 3 Language Specification. More complex built-in types like Timestamp can be found in the Well-Known Types list. Here’s an example of a generated model: type UpdateHourRequest struct { Time *timestamp.Timestamp `protobuf:"bytes,1,opt,name=time,proto3" json:"time,omitempty"` HasTrainingScheduled bool `protobuf:"varint,2,opt,name=has_training_scheduled,json=hasTrainingScheduled,proto3" json:"has_training_scheduled,omitempty"` Available bool `protobuf:"varint,3,opt,name=available,proto3" json:"available,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` // ... more proto garbage ;) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/genproto/trainer/trainer.pb.go Full source And the server: type TrainerServiceServer interface { IsHourAvailable(context.Context, *IsHourAvailableRequest) (*IsHourAvailableResponse, error) UpdateHour(context.Context, *UpdateHourRequest) (*EmptyResponse, error) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/genproto/trainer/trainer.pb.go Full source The difference between HTTP and gRPC is that with gRPC, we don’t need to worry about what to return or how to do it. If I were to compare the level of confidence between HTTP and gRPC, it would be like comparing Python and Go. gRPC is much stricter, and it’s impossible to return or receive invalid values: the compiler will let us know. Protobuf also has built-in support for field deprecation and backward compatibility. This helps in environments with many independent teams. Note Protobuf vs gRPC Protobuf (Protocol Buffers) is the Interface Definition Language used by default for defining the service interface and payload structure. Protobuf also serializes these models to binary format. You can find more details about gRPC and Protobuf on the gRPC Concepts page. Implementing the server works almost the same as HTTP generated by OpenAPI: we need to implement an interface (TrainerServiceServer in this case). type GrpcServer struct { db db } func (g GrpcServer) IsHourAvailable(ctx context.Context, req *trainer.IsHourAvailableRequest) (*trainer.IsHourAvailableResponse, error) { timeToCheck, err := grpcTimestampToTime(req.Time) if err != nil { return nil, status.Error(codes.InvalidArgument, "unable to parse time") } model, err := g.db.DateModel(ctx, timeToCheck) if err != nil { return nil, status.Error(codes.Internal, fmt.Sprintf("unable to get data model: %s", err)) } if hour, found := model.FindHourInDate(timeToCheck); found { return &trainer.IsHourAvailableResponse{IsAvailable: hour.Available && !hour.HasTrainingScheduled}, nil } return &trainer.IsHourAvailableResponse{IsAvailable: false}, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/grpc.go Full source As you can see, you cannot return anything other than IsHourAvailableResponse, and you can always be sure you’ll receive IsHourAvailableRequest. For errors, you can return one of the predefined error codes. These are more modern than HTTP status codes. Starting the gRPC server works the same as an HTTP server: server.RunGRPCServer(func(server *grpc.Server) { svc := GrpcServer{firebaseDB} trainer.RegisterTrainerServiceServer(server, svc) }) Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/main.go Full source Internal gRPC client After our server is running, it’s time to use it. First, we need to create a client instance. trainer.NewTrainerServiceClient is generated from .proto. type TrainerServiceClient interface { IsHourAvailable(ctx context.Context, in *IsHourAvailableRequest, opts ...grpc.CallOption) (*IsHourAvailableResponse, error) UpdateHour(ctx context.Context, in *UpdateHourRequest, opts ...grpc.CallOption) (*EmptyResponse, error) } type trainerServiceClient struct { cc grpc.ClientConnInterface } func NewTrainerServiceClient(cc grpc.ClientConnInterface) TrainerServiceClient { return &trainerServiceClient{cc} } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/genproto/trainer/trainer.pb.go Full source To make the generated client work, we need to pass a few extra options that handle the following: Authentication. TLS encryption. “Service discovery” (we use hardcoded names of services provided by Terraform via the TRAINER_GRPC_ADDR env variable). import ( // ... "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/genproto/trainer" // ... ) func NewTrainerClient() (client trainer.TrainerServiceClient, close func() error, err error) { grpcAddr := os.Getenv("TRAINER_GRPC_ADDR") if grpcAddr == "" { return nil, func() error { return nil }, errors.New("empty env TRAINER_GRPC_ADDR") } opts, err := grpcDialOpts(grpcAddr) if err != nil { return nil, func() error { return nil }, err } conn, err := grpc.Dial(grpcAddr, opts...) if err != nil { return nil, func() error { return nil }, err } return trainer.NewTrainerServiceClient(conn), conn.Close, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/client/grpc.go Full source After creating our client, we can call any of its methods. In this example, we call UpdateHour when creating a training. package main import ( // ... "github.com/pkg/errors" "github.com/golang/protobuf/ptypes" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/genproto/trainer" // ... ) type HttpServer struct { db db trainerClient trainer.TrainerServiceClient usersClient users.UsersServiceClient } // ... func (h HttpServer) CreateTraining(w http.ResponseWriter, r *http.Request) { // ... timestamp, err := ptypes.TimestampProto(postTraining.Time) if err != nil { return errors.Wrap(err, "unable to convert time to proto timestamp") } _, err = h.trainerClient.UpdateHour(ctx, &trainer.UpdateHourRequest{ Time: timestamp, HasTrainingScheduled: true, Available: false, }) if err != nil { return errors.Wrap(err, "unable to update trainer hour") } // ... } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/http.go Full source Cloud Run authentication & TLS Authentication of the client is handled by Cloud Run out of the box. You can also enable Authentication from Cloud Run UI. You need to also grant the roles/run.invoker role to service's service account. The simpler (and recommended by us) way is using Terraform. Miłosz described it in detail in the previous article. One thing that doesn’t work out of the box is sending authentication with the request. Did I mention that the standard gRPC transport is HTTP/2? For that reason, we can use good old JWT (JSON Web Tokens) for authentication. To make it work, we need to implement the google.golang.org/grpc/credentials.PerRPCCredentials interface. The implementation is based on the official guide from Google Cloud Documentation. type metadataServerToken struct { serviceURL string } func newMetadataServerToken(grpcAddr string) credentials.PerRPCCredentials { // based on https://cloud.google.com/run/docs/authenticating/service-to-service#go // service need to have https prefix without port serviceURL := "https://" + strings.Split(grpcAddr, ":")[0] return metadataServerToken{serviceURL} } // GetRequestMetadata is called on every request, so we are sure that token is always not expired func (t metadataServerToken) GetRequestMetadata(ctx context.Context, in ...string) (map[string]string, error) { // based on https://cloud.google.com/run/docs/authenticating/service-to-service#go tokenURL := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", t.serviceURL) idToken, err := metadata.Get(tokenURL) if err != nil { return nil, errors.Wrap(err, "cannot query id token for gRPC") } return map[string]string{ "authorization": "Bearer " + idToken, }, nil } func (metadataServerToken) RequireTransportSecurity() bool { return true } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/client/auth.go Full source The last step is passing it to the []grpc.DialOption list when creating all gRPC clients. It’s also a good idea to ensure our server’s certificate is valid with grpc.WithTransportCredentials. Authentication and TLS encryption are disabled in the local Docker environment. func grpcDialOpts(grpcAddr string) ([]grpc.DialOption, error) { if noTLS, _ := strconv.ParseBool(os.Getenv("GRPC_NO_TLS")); noTLS { return []grpc.DialOption{grpc.WithInsecure()}, nil } systemRoots, err := x509.SystemCertPool() if err != nil { return nil, errors.Wrap(err, "cannot load root CA cert") } creds := credentials.NewTLS(&tls.Config{ RootCAs: systemRoots, }) return []grpc.DialOption{ grpc.WithTransportCredentials(creds), grpc.WithPerRPCCredentials(newMetadataServerToken(grpcAddr)), }, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/client/grpc.go Full source Are all the problems of internal communication solved? A hammer is great for hammering nails but awful for cutting a tree. The same applies to gRPC or any other technique. gRPC works great for synchronous communication, but not every process is synchronous by nature. Applying synchronous communication everywhere will create a slow, unstable system. Currently, Wild Workouts doesn’t have any flow that should be asynchronous. We will cover this topic in more depth in the next articles by implementing new features. In the meantime, check out the Watermill library, which we also created. 😉 It helps with building asynchronous, event-driven applications the easy way. What’s next? Having robust contracts doesn’t mean we aren’t introducing unnecessary internal communication. In some cases, operations can be handled in one service in a simpler, more pragmatic way. Avoiding these issues isn’t simple. Fortunately, we know techniques that help. We’ll share them with you soon. 😉 Until then, we still have one article left about our “Too modern application.” It will cover Firebase HTTP authentication. After that, we’ll start the refactoring! As I mentioned in the first article, we intentionally introduced some issues in Wild Workouts. Are you curious about what’s wrong with Wild Workouts? Let us know in the comments! 😉 See you next week 👋

In the previous post, Robert introduced Wild Workouts, our example serverless application. Every week or two, we release new articles about this project, focusing on creating business-oriented applications in Go. In this post, I continue where Robert left off and describe the infrastructure setup. We picked Google Cloud Platform (GCP) as the provider of all infrastructure parts of the project. We use Cloud Run for running Go services, Firestore as the database, Cloud Build as CI/CD, and Firebase for web hosting and authentication. The project is based on Terraform. Infrastructure as Code If you’re not familiar with Terraform, it’s a tool for storing infrastructure configuration as text files. This technique is also known as “Infrastructure as Code”. It’s a broad topic, so I’ll mention just a few benefits: Storing infrastructure configuration in a repository. This gives you all the benefits of version control: commit history, a single source of truth, and code reviews. Consistency over time. Ideally, there are no manual changes to infrastructure other than those described in the configuration. Multiple environments. When you stop treating your servers as pets, it’s much easier to create identical environments, sometimes even on demand. Terraform 101 You can skip this part if you already know Terraform basics. In Terraform, you define a set of resources in .tf files using HCL syntax. A resource can be a database, network configuration, compute instance, or even a set of permissions. Available resource types depend on the provider you use (e.g., AWS, GCP, Digital Ocean). All resources are defined declaratively. You don’t write pseudocode that creates servers. Instead, you define the desired outcome. It’s up to Terraform to figure out how to configure what you specified. For example, it’s smart enough to create a resource only if it doesn’t exist. Resources can refer to each other using the full name. Besides resources, a project can also specify input variables to be filled by the user and output values that can be printed on the screen or stored in a file. There are also data sources that don’t create anything but read existing remote resources. You can find all the available resources in the specific provider documentation. Here’s an example for Google Cloud Platform. Two basic commands that you need to know are terraform plan and terraform apply. apply applies all resources defined in the current directory across all files with the .tf extension. The plan command is a “dry-run” mode that prints all changes that would be applied by apply. After applying changes, you will find a terraform.tfstate file in the same directory. This file holds a local “state” of your infrastructure. Google Cloud Project Our Terraform configuration creates a new GCP project. It’s completely separated from your other projects and is easy to clean up. Because some resources go beyond the free tier, you need a Billing Account. It can be an account with a linked credit card, but the $300 credit for new accounts also works. The basic part of the project definition looks like this: provider "google" { project = var.project region = var.region } data "google_billing_account" "account" { display_name = var.billing_account } resource "google_project" "project" { name = "Wild Workouts" project_id = var.project billing_account = data.google_billing_account.account.id } resource "google_project_iam_member" "owner" { role = "roles/owner" member = "user:${var.user}" depends_on = [google_project.project] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/project.tf Full source Let’s consider what each block does: Enables the GCP provider. Sets project name and chosen region from variables. These two fields are “inherited” by all resources unless overridden. Finds a billing account with the display name provided by the variable. Creates a new google project linked with the billing account. Note the reference to the account ID. Adds your user as the owner of the project. Note the string interpolation. Enabling APIs On a fresh GCP project, you can’t use most services right away. You first have to enable the API for each of them. You can enable an API by clicking a button in the GCP Console or do the same in Terraform: resource "google_project_service" "compute" { service = "[compute.googleapis.com](http://compute.googleapis.com/)" depends_on = [google_project.project] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/project.tf Full source We enable the following APIs: Cloud Source Repositories Cloud Build Container Registry Compute Cloud Run Firebase Firestore A note on dependencies If you’re wondering about the depends_on line, it sets an explicit dependency between the service and the project. Terraform detects dependencies between resources if they refer to each other. In the first snippet, you can see this with the billing account that’s referenced by the project: data "google_billing_account" "account" { display_name = var.billing_account } resource "google_project" "project" { # ... billing_account = data.google_billing_account.account.id } In the google_project_service resource, we don’t use project anywhere because it’s already set in the provider block. Instead, we use depends_on to specify an explicit dependency. This line tells Terraform to wait before creating the resource until the project is correctly created. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Cloud Run In Cloud Run, a service is a set of Docker containers with a common endpoint, exposing a single port (HTTP or gRPC). Each service is automatically scaled, depending on the incoming traffic. You can choose the maximum number of containers and how many requests each container can handle. It’s also possible to connect services with Google Cloud Pub/Sub. Our project doesn’t use it yet, but we will introduce it in future versions. You are charged only for the computing resources you use, so when a request is being processed or when the container starts. Wild Workouts consists of 3 services: trainer, trainings, and users. We decided to serve public API with HTTP and internal API with gRPC. Because you can’t expose two separate ports from a single service, we have to expose two containers per service (except trainings, which doesn’t have an internal API at the moment). We deploy 5 services in total, each with a similar configuration. Following the DRY principle, the common Terraform configuration is encapsulated in the service module. Note A Module in Terraform is a separate set of files in a subdirectory. Think of it as a container for a group of resources. It can have its own input variables and output values. Any module can call other modules by using the module block and passing a path to the directory in the source field. A single module can be called multiple times. Resources defined in the main working directory are considered to be in a root module. The module holds a definition of a single generic Cloud Run service and is used multiple times in cloud-run.tf. It exposes several variables, e.g., name and type of the server (HTTP or gRPC). A service exposing gRPC is the simpler one: module cloud_run_trainer_grpc { source = "./service" project = var.project location = var.region dependency = null_resource.init_docker_images name = "trainer" protocol = "grpc" } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/cloud-run.tf Full source The protocol is passed to the SERVER_TO_RUN environment variable, which decides what server is run by the service. Compare this with an HTTP service definition. We use the same module, but with additional environment variables. We need to add them because public services contact internal services via the gRPC API and need to know their addresses. module cloud_run_trainings_http { source = "./service" project = var.project location = var.region dependency = null_resource.init_docker_images name = "trainings" protocol = "http" auth = false envs = [ { name = "TRAINER_GRPC_ADDR" value = module.cloud_run_trainer_grpc.endpoint }, { name = "USERS_GRPC_ADDR" value = module.cloud_run_users_grpc.endpoint } ] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/cloud-run.tf Full source The reference: module.cloud_run_trainer_grpc.endpoint points to endpoint output defined in the service module: output endpoint { value = "${trimprefix(google_cloud_run_service.service.status[0].url, "https://")}:443" } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/service/outputs.tf Full source Using environment variables is an easy way to make services aware of each other. With more complex connections and more services, it would probably be best to implement some kind of service discovery. We may cover this in a future post. If you’re curious about the dependency variable, see In-depth below for details. Cloud Run Permissions Cloud Run services have authentication enabled by default. Setting the auth = false variable in the service module adds additional IAM policy for the service, making it accessible for the public. We do this just for the HTTP APIs. data "google_iam_policy" "noauth" { binding { role = "roles/run.invoker" members = [ "allUsers", ] } } resource "google_cloud_run_service_iam_policy" "noauth_policy" { count = var.auth ? 0 : 1 location = google_cloud_run_service.service.location service = google_cloud_run_service.service.name policy_data = data.google_iam_policy.noauth.policy_data } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/service/service.tf Full source Note the following line: count = var.auth ? 0 : 1 This line is Terraform’s way of making an if statement. count defines how many copies of a resource Terraform should create. It skips resources with a count of 0. Firestore We picked Firestore as the database for Wild Workouts. See our first post for reasons behind this. Note Since writing this article, our recommendation has changed: just use PostgreSQL. It’s easier to operate and simpler to migrate between cloud providers if needed. See the State of this article in 2026 note in our first post for more context. Firestore works in two modes: Native or Datastore. You have to decide early on, as the choice is permanent after your first write to the database. You can pick the mode in the GCP Console GUI, but we wanted to make the setup fully automated. In Terraform, there’s a google_project_service resource available, but it enables Datastore mode. Sadly, we can’t use it since we want to use Native mode. A workaround is to use the gcloud command to enable it (note the alpha version). gcloud alpha firestore databases create To run this command, we use the null resource. It’s a special kind of resource that lets you run custom provisioners locally or on remote servers. A provisioner is a command or other software that makes changes to the system. We use local-exec provisioner, which is simply executing a bash command on the local system. In our case, it’s one of the targets defined in Makefile. resource "null_resource" "enable_firestore" { provisioner "local-exec" { command = "make firestore" } depends_on = [google_firebase_project_location.default] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/firestore.tf Full source Firestore requires creating all composite indexes upfront. It’s also available as a Terraform resource. resource "google_firestore_index" "trainings_user_time" { collection = "trainings" fields { field_path = "UserUuid" order = "ASCENDING" } fields { field_path = "Time" order = "ASCENDING" } fields { field_path = "__name__" order = "ASCENDING" } depends_on = [null_resource.enable_firestore] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/firestore.tf Full source Note the explicit depends_on that points to the null_resource creating database. Firebase Firebase provides us with frontend application hosting and authentication. Currently, Terraform supports only part of Firebase API, and some of it is still in beta. That’s why we need to enable the google-beta provider: provider "google-beta" { project = var.project region = var.region credentials = base64decode(google_service_account_key.firebase_key.private_key) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/firebase.tf Full source Then we define the project, project’s location and the web application. resource "google_firebase_project" "default" { provider = google-beta depends_on = [ google_project_service.firebase, google_project_iam_member.service_account_firebase_admin, ] } resource "google_firebase_project_location" "default" { provider = google-beta location_id = var.firebase_location depends_on = [ google_firebase_project.default, ] } resource "google_firebase_web_app" "wild_workouts" { provider = google-beta display_name = "Wild Workouts" depends_on = [google_firebase_project.default] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/firebase.tf Full source Authentication management still lacks a Terraform API, so you have to enable it manually in the Firebase Console. Firebase authentication is the only thing we found no way to automate. Firebase Routing Firebase also handles public routing to services. Thanks to this the frontend application can call API with /api/trainer instead of https://trainer-http-smned2eqeq-ew.a.run.app. These routes are defined in web/firebase.json. "rewrites": [ { "source": "/api/trainer{,/**}", "run": { "serviceId": "trainer-http", "region": "europe-west1" } }, // ... ] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/web/firebase.json Full source Cloud Build Cloud Build is our Continuous Delivery pipeline. It has to be enabled for a repository, so we define a trigger in Terraform and the repository itself. resource "google_sourcerepo_repository" "wild_workouts" { name = var.repository_name depends_on = [ google_project_service.source_repo, ] } resource "google_cloudbuild_trigger" "trigger" { trigger_template { branch_name = "master" repo_name = google_sourcerepo_repository.wild-workouts.name } filename = "cloudbuild.yaml" } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/repo.tf Full source The build configuration has to be defined in the cloudbuild.yaml file committed to the repository. We defined a couple of steps in the pipeline: Linting (go vet): this step could be extended in the future with running tests and all kinds of static checks and linters Building docker images Deploying docker images to Cloud Run Deploying web app to Firebase hosting We keep several services in one repository, and their building and deployment are almost the same. To reduce repetition, there are a few helper bash scripts in the scripts directory. Note Should I use a monorepo? We decided to keep all services in one repository. We did this mainly because Wild Workouts is an example project, and it’s much easier to set up everything this way. We can also easily share common code (e.g., setting up gRPC and HTTP servers). From our experience, using a single repository is a great starting point for most projects. It’s essential, though, that all services are entirely isolated from each other. If needed, we could easily split them into separate repositories. We might show this in future articles in this series. The biggest disadvantage of the current setup is that all services and the frontend application are deployed at the same time. That’s usually not what you want when working with microservices with a bigger team. But it’s probably acceptable when you’re just starting out and creating an MVP. As usual in CI/CD tools, the Build configuration is defined in YAML. Let’s see the whole configuration for just one service, to reduce some noise. steps: - id: trainer-lint name: golang entrypoint: ./scripts/lint.sh args: [trainer] - id: trainer-docker name: gcr.io/cloud-builders/docker entrypoint: ./scripts/build-docker.sh args: ["trainer", "$PROJECT_ID"] waitFor: [trainer-lint] - id: trainer-http-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainer, http, "$PROJECT_ID"] waitFor: [trainer-docker] - id: trainer-grpc-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainer, grpc, "$PROJECT_ID"] waitFor: [trainer-docker] options: env: - 'GO111MODULE=on' machineType: 'N1_HIGHCPU_8' images: - 'gcr.io/$PROJECT_ID/trainer' Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source A single step definition is pretty short: id is a unique identifier of a step. It can be used in waitFor array to specify dependencies between steps (steps are running in parallel by default). name is the name of a docker image that will be run for this step. entrypoint works like in Docker images, so it’s a command that executes when the container starts. We use bash scripts to keep the YAML definition short, but this can be any bash command. args will be passed to entrypoint as arguments. In options, we override the machine type to make our builds run faster. There’s also an environment variable to force the use of Go modules. The images list defines Docker images that should be pushed to Container Registry. In the example above, the Docker image is built in the trainer-docker step. Deploys Deploys to Cloud Run use the gcloud run deploy command. Cloud Build builds the Docker image in the previous step, so we can use the latest image from the registry. - id: trainer-http-deploy name: gcr.io/cloud-builders/gcloud entrypoint: ./scripts/deploy.sh args: [trainer, http, "$PROJECT_ID"] waitFor: [trainer-docker] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source gcloud run deploy "$service-$server_to_run" \ --image "gcr.io/$project_id/$service" \ --region europe-west1 \ --platform managed Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/scripts/deploy.sh Full source The frontend is deployed with the firebase command. There’s no need to use a helper script here, as there’s just one frontend application. - name: gcr.io/$PROJECT_ID/firebase args: ['deploy', '--project=$PROJECT_ID'] dir: web waitFor: [web-build] Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/cloudbuild.yaml Full source This step uses the gcr.io/$PROJECT_ID/firebase Docker image. It doesn’t exist by default, so we use another null_resource to build it from cloud-builders-community: resource "null_resource" "firebase_builder" { provisioner "local-exec" { command = "make firebase_builder" } depends_on = [google_project_service.container_registry] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/repo.tf Full source Cloud Build Permissions All required permissions are defined in iam.tf. Cloud Build needs permissions for Cloud Run (to deploy backend services) and Firebase (to deploy frontend application). First, we define account names as local variables to make the file more readable. If you’re wondering where these names come from, you can find them in the Cloud Build documentation. locals { cloud_build_member = "serviceAccount:${google_project.project.number}@cloudbuild.gserviceaccount.com" compute_account = "projects/${var.project}/serviceAccounts/${google_project.project.number}-compute@developer.gserviceaccount.com" } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/iam.tf Full source We then define all permissions, as described in the documentation. For example, here’s Cloud Build member with Cloud Run Admin role: resource "google_project_iam_member" "cloud_run_admin" { role = "roles/run.admin" member = local.cloud_build_member depends_on = [google_project_service.cloud_build] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/iam.tf Full source Dockerfiles We have several Dockerfiles defined in the project in the docker directory. app - Go service image for local development. It uses reflex for hot code recompilation. See our post about local Go environment to learn how it works. app-prod - production image for Go services. It builds the Go binary for a given service and runs it. web - frontend application image for local development. Setup Usually, you would execute a Terraform project with terraform apply. This command applies all resources in the current directory. Our example is a bit more complex because it sets up the whole GCP project along with dependencies. The Makefile orchestrates the setup. Requirements: git, gcloud, and terraform installed on your system. Docker A GCP account with a Billing Account. Authentication There are two credentials that you need for the setup. Start with logging in with gcloud: gcloud auth login Then you need to obtain Application Default Credentials: gcloud auth application-default login This stores your credentials in a well-known place, so Terraform can use it. Note Security warning! Authenticating this way is the easiest approach and fine for local development, but you probably don’t want to use it in a production setup. Instead, consider creating a service account with only the necessary permissions (depending on what your Terraform configuration does). You can then download the JSON key and store its path in the GOOGLE_CLOUD_KEYFILE_JSON environment variable. Picking a Region During the setup, you need to pick Cloud Run region and Firebase location. These two are not the same thing (see the lists on linked documentation). We picked europe-west1 as the default region for Cloud Run. It’s hardcoded on the repository in two files: scripts/deploy.sh - for deploying the Cloud Run services web/firebase.json - for routing Cloud Run public APIs If you’re fine with the defaults, you don’t need to change anything. Just pick Firebase location that’s close, i.e., europe-west. If you’d like to use a different region, you need to update the files mentioned above. It’s important to commit your changes on master before you run the setup. It’s not enough to change them locally! Run it! Clone the project repository and enter the terraform directory from the command line. A single make command should be enough to start. See the included README for more details. Note Billing warning! This project goes beyond the Google Cloud Platform free tier. You need to have a billing account to use it. You can use the $300 free credit given to new accounts or create a new account in the Billing section. For pricing, see Cloud Run Pricing. The best way to estimate your project cost is to use the official calculator. Wild Workouts should cost you up to $1 per month if you use it just occasionally. We recommend deleting the project after you’re done playing with it. If you want to run it continuously, you should know that most of the cost comes from Cloud Build and depends on the number of builds (triggered by new commits on master). You can downgrade the machineType in cloudbuild.yaml to reduce some costs at the expense of longer build times. It’s also a good idea to set up a Budget Alert in the Billing Account settings. You will then receive a notification if something triggers unexpected charges. At the end of the setup, a new Git remote called google will be added to your local repository. The master branch will be pushed to this remote, triggering your first Cloud Build. If you’d like to make any changes to the project, you need to push to the correct origin, i.e., git push google. You can also update the origin remote with git remote set-url. Note If you keep your code on GitHub, GitLab, or another platform, you can set up a mirror instead of hosting the repository on Cloud Source Repositories. In-depth Env variables All available variables are defined in the vars.tf file. If a variable has no default value and you don’t provide it, Terraform won’t let you apply anything. There are several ways to supply variables for terraform apply. You can pass them individually with -var flag or all at once with -var-file flag. Terraform also looks for them in TF_VAR_name environment variables. We’ve picked the last option for this project. We need to use the environment variables anyway in the Makefile and bash scripts. This way, there’s just one source of truth. Make runs set-envs.sh at the beginning of the setup. It’s a helper script that asks the user for all the required variables. These are then saved to the .env file. You can edit this file manually as well. Note Terraform does not automatically read it. It’s sourced just before running apply: load_envs:=source ./.env # ... apply: ${load_envs} && terraform apply Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/Makefile Full source Building docker images Because we’re setting up the project from scratch, we run into a dilemma: we need Docker images in the project’s Container Registry to create Cloud Run instances, but we can’t create the images without a project. You could set the image name in Terraform definition to an example image and then let Cloud Build overwrite it. But then each terraform apply would try to change them back to the original value. Our solution is to build the images based on the “hello” image first and then deploy Cloud Run instances. Then Cloud Build builds and deploys proper images, but the name stays the same. resource "null_resource" "init_docker_images" { provisioner "local-exec" { command = "make docker_images" } depends_on = [google_project_service.container_registry] } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/docker-images.tf Full source Note the depends_on that points at Container Registry API. It ensures Terraform won’t start building images until there’s a registry where it can push them. Destroying You can delete the entire project with make destroy. If you take a look in the Makefile, there’s something unusual before terraform destroy: terraform state rm "google_project_iam_member.owner" terraform state rm "google_project_service.container_registry" terraform state rm "google_project_service.cloud_run" Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/terraform/Makefile Full source These commands are a workaround for Terraform’s lack of a “skip destroy” feature. They remove some resources from Terraform’s local state file. Terraform won’t destroy these resources because, as far as it’s concerned, they don’t exist at this point. We don’t want to destroy the owner IAM policy, because if something goes wrong during the destroy, you will lock yourself out of the project and won’t be able to access it. The other two lines relate to enabled APIs: there’s a possible race condition where some resources would still use these APIs during destroy. By removing them from the state, we avoid this issue. A note on magic Our goal was to make it as easy as possible to set up the whole project. Because of that, we had to cut some corners where Terraform is missing features or an API is not yet available. There’s also some Makefile and bash magic involved that’s not always easy to understand. I want to make this clear because you will likely encounter similar dilemmas in your projects. You will need to choose between fully automated solutions glued together with alpha APIs or plain Terraform with some manual steps documented in the project. Both approaches have their place. For example, this project is straightforward to set up multiple times with the same configuration. If you’d like to create an exact copy of the production environment, you can have a completely separate project working within minutes. If you keep just one or two environments, you won’t need to recreate them every week. It’s then probably fine to stick to the available APIs and document the rest. Is serverless the way to go? This project is our first approach to “serverless” deployment, and at first, we had some concerns. Is this all just hype, or is serverless the future? Wild Workouts is a fairly small project that might not show everything Cloud Run has to offer. Overall, we found it quite simple to set up, and it nicely hides all complexities. It’s also more natural to work with than Cloud Functions. After the initial setup, there shouldn’t be much infrastructure maintenance needed. You don’t have to worry about keeping up a Docker registry or a Kubernetes cluster and can instead focus on creating the application. On the other hand, it’s also quite limited in features, as only a few protocols are supported. It looks like a great fit for services with a REST API. The pricing model also seems reasonable, as you pay only for the resources you use. What about vendor lock-in? The entire Terraform configuration is now tied to GCP, and there’s no way around it. If we wanted to migrate the project to AWS, a new configuration would be needed. However, there are some universal concepts. Services running in Docker images can be deployed on any platform, whether it’s a Kubernetes cluster or docker-compose. Most platforms also offer some kind of registry for images. Is writing a new Terraform configuration all that’s required to migrate out of Google? Not really, as the application itself is coupled to Firebase and Authentication offered by GCP. We will show a better approach to this problem in future posts in this series. What’s next? This post just scratches the surface of Terraform. There are many advanced topics you should take a look at if you consider using it. For example, the remote state is necessary when working in a team. I hope this was a solid introduction. If you have any questions, please let us know in the comments below. We might even follow up with full posts to some of them, as the topic is broad enough. Next week, we’re going to cover gRPC communication. 👋

Welcome to the first article from the series covering how to build business-oriented applications in Go! In this series, we want to show you how to build applications that are easy to develop, maintain, and fun to work with in the long term. This series doesn’t focus too heavily on infrastructure and implementation details. But we need some foundation to build on later. In this article, we start by covering some basic tools from Google Cloud that can help us to do that. Note This is not just another article with random code snippets. This post is part of a bigger series where we show how to build Go applications that are easy to develop, maintain, and fun to work with in the long term. We are doing it by sharing proven techniques based on many experiments we did with teams we lead and scientific research. You can learn these patterns by building with us a fully functional example Go web application – Wild Workouts. We did one thing differently – we included some subtle issues to the initial Wild Workouts implementation. Have we lost our minds to do that? Not yet. 😉 These issues are common for many Go projects. In the long term, these small issues become critical and stop adding new features. It’s one of the essential skills of a senior or lead developer; you always need to keep long-term implications in mind. We will fix them by refactoring Wild Workouts. In that way, you will quickly understand the techniques we share. Do you know that feeling after reading an article about some technique and trying implement it only to be blocked by some issues skipped in the guide? Cutting these details makes articles shorter and increases page views, but this is not our goal. Our goal is to create content that provides enough know-how to apply presented techniques. If you did not read previous articles from the series yet, we highly recommend doing that. We believe that in some areas, there are no shortcuts. If you want to build complex applications in a fast and efficient way, you need to spend some time learning that. If it was simple, we wouldn’t have large amounts of scary legacy code. Here’s the full list of 14 articles released so far. The full source code of Wild Workouts is available on GitHub. Don’t forget to leave a star for our project! ⭐ Note State of this article in 2026 It’s interesting to see how things have changed since we wrote this article in 2020. There were many changes on the frontend: Vue.js and Bootstrap are now obsolete in favor of React and Tailwind CSS. But as a contrast, not much has changed on the backend side! This matters for long-term maintainability: you don’t want to rewrite your backend every six years because a shiny new framework appeared. With Go and solid architectural patterns, this stability is achievable. The only big difference is that we no longer recommend Firebase/Firestore by default. Our advice now: just use PostgreSQL. It’s easier to operate and simpler to migrate between cloud providers if needed (or even host it on a VM). We still recommend abstracting your database behind interfaces as described in The Repository Pattern article, which makes changing such decisions straightforward. That’s about it. The backend content here hasn’t become outdated in six years, and it won’t in the next six or more. That’s the power of learning timeless patterns over chasing the newest tools. Your knowledge compounds over time instead of requiring you to constantly relearn everything. Why serverless? Running a Kubernetes cluster requires a lot of support from “DevOps teams”. Let’s skip the fact that DevOps is not a job title for now. DevOps is a culture and mindset. It is not a job title! Slide from The gordian knot - Alberto Brandolini Small applications that could easily run on one virtual machine now get deployed on highly complex Kubernetes clusters. All these clusters require significant maintenance. On the other hand, moving applications to containers has given us significant flexibility in building and deploying them. It enabled rapid deployments of hundreds of microservices with considerable autonomy. But the cost is high. Wouldn’t it be great if a fully managed solution existed? 🤔 Maybe your company already uses a managed Kubernetes cluster. If so, you probably know that even a managed cluster still requires substantial “DevOps” support. Maybe serverless? Well, splitting a big application into multiple, independent Lambdas (Cloud Functions) is a great way to create an unmaintainable cataclysm. You should probably rethink your architecture if it can be used for summoning demons. But wait, is it the only way to build serverless applications? No! Google Cloud Run The idea behind Google Cloud Run is simple: you just need to provide a Docker container, and Google Cloud runs it. Inside this container, you can run an application written in any language that can expose a port with your HTTP or gRPC API. You’re not limited to synchronous processing: you can also process Pub/Sub messages inside this container. And that’s all that you need from the infrastructure side. Google Cloud does all the magic. Based on the traffic, the container will automatically scale up and down. Sounds like a perfect solution? In practice, it is not so simple. Many articles show how to use Google Cloud Run, but they usually present small building blocks you can use to build an application. It’s hard to combine all these pieces from multiple places into a fully working project (been there, done that). Most of these articles ignore the problem of vendor lock-in. The deployment method should be just an implementation detail. I covered this topic in the Why using Microservices or Monolith can be just a detail? article in 2018. But most importantly, using the newest and shiniest technologies doesn’t mean your application won’t become hated legacy in the next 3 months. Serverless solves only infrastructure challenges. It doesn’t stop you from building an application that is hard to maintain. I even have the impression that it’s the opposite: these fancy applications sooner or later become the hardest to maintain. For this series of articles, we created a fully functional, real-life application. You can deploy this application to Google Cloud with one command using Terraform. You can run a local copy with a single docker-compose command. There is also one thing we’re doing differently than others. We included some subtle issues that, from our observations, are common in Go projects. In the long term, these small issues become critical and prevent us from adding new features. Have we lost our minds? Not yet. 😉 This approach will help you understand which issues you can solve and which techniques help. It’s also a test for the practices we use. If something isn’t a problem, why use any technique to solve it? Plan In the next few articles, we cover all topics related to running the application on Google Cloud. In this part, we didn’t add any issues or bad practices. 😉 The first articles may be a bit basic if you already have some experience in Go. We want to ensure that if you’re just starting with Go, you’ll be able to follow the more complex topics that come next. Next, we refactor parts of the application that handle business logic. This part will be much more complex. Don't miss new posts.Join over 20k subscribers of our newsletter and get a free e-book! Go With The Domain Three Dots Labs Failed to subscribe, please try again later. Subscribe Loading... 🔒 We do not send spam. You can unsubscribe at any time! Running the project locally The ability to run a project locally is critical for efficient development. It’s frustrating when you can’t check your changes quickly and easily. It’s much harder to achieve it for projects built from hundreds of microservices. Fortunately, our project has only 5 services. 😉 In Wild Workouts, we created a Docker Compose setup with live code reloading for both frontend and backend. For the frontend, we use a container with the vue-cli-service serve tool. For the backend, the situation is a bit more complex. In all containers, we run the reflex tool. reflex listens for code changes and triggers recompilation of the service. If you’re interested in the details, you can find them in our Go Docker dev environment with Go Modules and live code reloading blog post. Requirements The only requirements needed to run the project are Docker and Docker Compose. Running git clone https://github.com/ThreeDotsLabs/wild-workouts-go-ddd-example.git && cd wild-workouts-go-ddd-example And run Docker Compose: docker-compose up After downloading all JavaScript and Go dependencies, you should see a message with the frontend address: web_1 | $ vue-cli-service serve web_1 | INFO Starting development server... web_1 | DONE Compiled successfully in 6315ms11:18:26 AM web_1 | web_1 | web_1 | App running at: web_1 | - Local: http://localhost:8080/ web_1 | web_1 | It seems you are running Vue CLI inside a container. web_1 | Access the dev server via http://localhost:<your container's external mapped port>/ web_1 | web_1 | Note that the development build is not optimized. web_1 | To create a production build, run yarn build. Congratulations! Your local version of the Wild Workouts application is available at http://localhost:8080/. There is also a public version available at https://threedotslabs-wildworkouts.web.app. What Wild Workouts can do? How often have you seen tutorials without any real-life functionality? How often have patterns from these tutorials failed in real projects? Probably too often. 😉 Real life is not as simple as tutorials suggest. To avoid this problem, we created Wild Workouts as a fully functional project. It’s much harder to take shortcuts and skip complexity when an application needs to be complete. This makes all articles longer, but there are no shortcuts here. If you don’t spend enough time at the beginning, you’ll lose much more time later during implementation. Or worse: you’ll be fixing problems in a rush with the application already running in production. tl;dr Wild Workouts is an application for personal gym trainers and attendees. Trainers can set a schedule for when they are available for training. Attendees can schedule training for available dates. Other functionalities are: management of “credits” (how many trainings the attendee can schedule) cancellation if a training is canceled less than 24 hours before it begins, the attendee will not receive their credits back training reschedule if someone wants to reschedule a training less than 24 hours before it begins, the other participant (trainer or attendee) must approve it calendar view Sounds simple. What can go wrong? 🤔 Frontend If you are not interested in the frontend part, you can go straight to the Backend section. I’m primarily a backend engineer, and to be honest, I’m not the greatest JavaScript or frontend specialist. 😉 But we can’t have an end-to-end application without a frontend! In this series, we focus on the backend part. I’ll give a high-level overview of the technologies I used on the frontend. This will be nothing new if you have any basic frontend knowledge. For more details, I recommend checking the source code in the web/ directory. If you want to know more about how the frontend is built, please let us know in the comments! OpenAPI (Swagger) client Nobody likes keeping API contracts up to date manually. It’s tedious and counterproductive to keep multiple JSON files in sync. OpenAPI solves this problem by generating a JavaScript HTTP client and Go HTTP server from the provided specification. We dive into the details in the Backend part. Bootstrap You probably already know Bootstrap, the greatest friend of every backend engineer, like me. 😉 Wrestling with HTML and CSS is the part of frontend development I dislike the most. Bootstrap provided almost all the building blocks I needed for the application’s HTML. Vue.js After evaluating several popular frontend frameworks, I decided to use Vue.js. I really enjoyed its simplicity. I started my journey as a full-stack developer in the pre-jQuery era. Frontend tooling has made huge progress… but I’ll stay with the backend for now. 😉 Backend The backend of Wild Workouts is built from 3 services. trainer – provides public HTTP and internal gRPC endpoints for managing trainer schedule trainings – provides public HTTP for managing attendee trainings users – provides public HTTP endpoints and internal gRPC endpoints , manages credits and user data If a service exposes 2 types of APIs, each of them is exposed in a separate process. Public HTTP API Most operations performed by applications are triggered by the public HTTP API. I’ve heard Go newcomers ask many times what framework they should use to create an HTTP service. I always advise against using any kind of HTTP framework in Go. A simple router, like chi, is more than enough. chi provides only the lightweight glue to define what URLs and methods our API supports. Under the hood, it uses the Go standard library http package, so all related tools like middlewares are 100% compatible. It may feel strange not to use a framework if you’re coming from a language where Spring, Symfony, Django, or Express are the obvious choices. It felt strange to me too. Using any framework in Go adds unnecessary complexity and couples your project with that framework. KISS. 😉 All the services run the HTTP server in the same way. It makes sense not to copy this code three times. func RunHTTPServer(createHandler func(router chi.Router) http.Handler) { apiRouter := chi.NewRouter() setMiddlewares(apiRouter) rootRouter := chi.NewRouter() // we are mounting all APIs under /api path rootRouter.Mount("/api", createHandler(apiRouter)) logrus.Info("Starting HTTP server") http.ListenAndServe(":"+os.Getenv("PORT"), rootRouter) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/server/http.go Full source chi provides a set of useful built-in HTTP middlewares, but we’re not limited to them. All middlewares compatible with the Go standard library will work. Note Long story short: middlewares allow us to do anything before and after a request is executed (with access to the http.Request). Using HTTP middlewares gives us significant flexibility in building our custom HTTP server. We build our server from multiple decoupled components that you can customize for your needs. func setMiddlewares(router *chi.Mux) { router.Use(middleware.RequestID) router.Use(middleware.RealIP) router.Use(logs.NewStructuredLogger(logrus.StandardLogger())) router.Use(middleware.Recoverer) addCorsMiddleware(router) addAuthMiddleware(router) router.Use( middleware.SetHeader("X-Content-Type-Options", "nosniff"), middleware.SetHeader("X-Frame-Options", "deny"), ) router.Use(middleware.NoCache) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/common/server/http.go Full source We have our framework almost ready now. :) It’s time to use it. We can call server.RunHTTPServer in the trainings service. package main // ... func main() { // ... server.RunHTTPServer(func(router chi.Router) http.Handler { return HandlerFromMux(HttpServer{firebaseDB, trainerClient, usersClient}, router) }) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/main.go Full source createHandler needs to return http.Handler. In our case, it is HandlerFromMux generated by oapi-codegen. It provides us all the paths and query parameters from the OpenAPI specs. // HandlerFromMux creates http.Handler with routing matching OpenAPI spec based on the provided mux. func HandlerFromMux(si ServerInterface, r chi.Router) http.Handler { r.Group(func(r chi.Router) { r.Use(GetTrainingsCtx) r.Get("/trainings", si.GetTrainings) }) r.Group(func(r chi.Router) { r.Use(CreateTrainingCtx) r.Post("/trainings", si.CreateTraining) }) // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/openapi_api.gen.go Full source # ... paths: /trainings: get: operationId: getTrainings responses: '200': description: todo content: application/json: schema: $ref: '#/components/schemas/Trainings' default: description: unexpected error content: application/json: schema: $ref: '#/components/schemas/Error' # ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/api/openapi/trainings.yml Full source If you make changes to the OpenAPI spec, you need to regenerate the Go server and JavaScript clients. Run: make openapi Part of the generated code is ServerInterface. It contains all methods the API must support. You implement server functionality by implementing this interface. type ServerInterface interface { // (GET /trainings) GetTrainings(w http.ResponseWriter, r *http.Request) // (POST /trainings) CreateTraining(w http.ResponseWriter, r *http.Request) // ... } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/openapi_api.gen.go Full source This is an example of how trainings.HttpServer is implemented: package main import ( "net/http" "github.com/go-chi/render" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/auth" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/genproto/trainer" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/genproto/users" "github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/pkg/internal/server/httperr" ) type HttpServer struct { db db trainerClient trainer.TrainerServiceClient usersClient users.UsersServiceClient } func (h HttpServer) GetTrainings(w http.ResponseWriter, r *http.Request) { user, err := auth.UserFromCtx(r.Context()) if err != nil { httperr.Unauthorised("no-user-found", err, w, r) return } trainings, err := h.db.GetTrainings(r.Context(), user) if err != nil { httperr.InternalError("cannot-get-trainings", err, w, r) return } trainingsResp := Trainings{trainings} render.Respond(w, r, trainingsResp) } // ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/http.go Full source But HTTP paths are not the only thing generated from the OpenAPI spec. More importantly, it also provides the models for responses and requests. Models are, in most cases, much more complex than API paths and methods. Generating them saves time and frustration during API contract changes. # ... schemas: Training: type: object required: [uuid, user, userUuid, notes, time, canBeCancelled, moveRequiresAccept] properties: uuid: type: string format: uuid user: type: string example: Mariusz Pudzianowski userUuid: type: string format: uuid notes: type: string example: "let's do leg day!" time: type: string format: date-time canBeCancelled: type: boolean moveRequiresAccept: type: boolean proposedTime: type: string format: date-time moveProposedBy: type: string Trainings: type: object required: [trainings] properties: trainings: type: array items: $ref: '#/components/schemas/Training' # ... Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/api/openapi/trainings.yml Full source // Training defines model for Training. type Training struct { CanBeCancelled bool `json:"canBeCancelled"` MoveProposedBy *string `json:"moveProposedBy,omitempty"` MoveRequiresAccept bool `json:"moveRequiresAccept"` Notes string `json:"notes"` ProposedTime *time.Time `json:"proposedTime,omitempty"` Time time.Time `json:"time"` User string `json:"user"` UserUuid string `json:"userUuid"` Uuid string `json:"uuid"` } // Trainings defines model for Trainings. type Trainings struct { Trainings []Training `json:"trainings"` } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/openapi_types.gen.go Full source Cloud Firestore database All right, we have the HTTP API. But even the best API is useless without data and the ability to persist it. If we want to build the application in the most modern, scalable, and truly serverless way, Firestore is a natural choice. We get all of that out of the box. What’s the cost? For the Europe multi-region option, you pay: $0.06 per 100,000 documents reads $0.18 per 100,000 documents writes $0.02 per 100,000 documents deletes $0.18/GiB of stored data/month Sounds pretty cheap? For comparison, let’s take the cheapest Cloud SQL MySQL db-f1-micro instance with a shared Virtual CPU and 3 GB of storage as a reference: it costs $15.33/month. The cheapest instance with high availability and with 1 non-shared Virtual CPU costs $128.21/month. Even better, in the free plan, you can store up to 1 GiB of data with 20k document writes per day. Firestore is a NoSQL database, so we shouldn’t expect to build relational models the SQL way. Instead, we have a system of hierarchical collections. In our case, the data model is simple, so we have only one level of collections. Unlike many NoSQL databases, Firestore offers ACID transactions on any operation. This also works when updating multiple documents. Firestore limitations An important limitation is 1 update per second per document. You can still update many independent documents in parallel. This is an important factor to consider when designing your database. In some cases, you should consider batching operations, different document designs, or using a different database. If data changes frequently, a key-value database might be a good choice. In my experience, the 1 update per second per document limitation isn’t a serious problem. In most cases when I used Firestore, we were updating many independent documents. This also applies when using Firestore for event sourcing: you only use append operations. In Wild Workouts, we shouldn’t have a problem with this limit. 😉 Note I have also observed that Firestore needs some time to warm up. In other words, if you want to insert 10 million documents within one minute after setting up a new project, it may not work. I assume this is related to some internal mechanism that handles scalability. Fortunately, in the real world, traffic spikes from 0 to 10 million writes per minute are uncommon. Running Firestore locally Unfortunately, the Firestore emulator is not perfect. 😉 I found some situations where the emulator wasn’t 100% compatible with the real version. I also encountered situations where updating and reading the same document in a transaction caused a deadlock. From my perspective, this functionality is sufficient for local development. The alternative is to have a separate Google Cloud project for local development. I prefer having a local environment that is truly local and doesn’t depend on any external services. It’s also easier to set up and can be used later in continuous integration. Since the end of May, the Firestore emulator provides a UI. It is added to the Docker Compose and is available at http://localhost:4000/. At the time of writing, sub-collections aren’t displayed properly in the emulator UI. Don’t worry, for Wild Workouts this isn’t a problem. 😉 Using Firestore Apart from the Firestore implementation, the code works the same way locally and in production. When using the emulator locally, we need to run our application with the FIRESTORE_EMULATOR_HOST environment variable set to the emulator hostname (in our case firestore:8787). This is set in the .env file. In production, Google Cloud handles everything under the hood, and no extra configuration is needed. firebaseClient, err := firestore.NewClient(ctx, os.Getenv("GCP_PROJECT")) if err != nil { panic(err) } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainings/main.go Full source Here’s an example of how I used the Firestore client to query the trainer schedule. You can see how I used the queries functionality to get only dates within the queried date interval. package main import ( // ... "cloud.google.com/go/firestore" // ... ) // ... type db struct { firestoreClient *firestore.Client } func (d db) TrainerHoursCollection() *firestore.CollectionRef { return d.firestoreClient.Collection("trainer-hours") } // ... func (d db) QueryDates(params *GetTrainerAvailableHoursParams, ctx context.Context) ([]Date, error) { iter := d. TrainerHoursCollection(). Where("Date.Time", ">=", params.DateFrom). Where("Date.Time", "<=", params.DateTo). Documents(ctx) var dates []Date for { doc, err := iter.Next() if err == iterator.Done { break } if err != nil { return nil, err } date := Date{} if err := doc.DataTo(&date); err != nil { return nil, err } date = setDefaultAvailability(date) dates = append(dates, date) } return dates, nil } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/firestore.go Full source No extra data mapping is needed. The Firestore library can marshal any struct with public fields or map[string]interface, as long as there’s nothing unusual inside. 😉 You can find the full specification of how conversion works in the cloud.google.com/go/firestore GoDoc. type Date struct { Date openapi_types.Date `json:"date"` HasFreeHours bool `json:"hasFreeHours"` Hours []Hour `json:"hours"` } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/openapi_types.gen.go Full source date := Date{} if err := doc.DataTo(&date); err != nil { return nil, err } Full source: github.com/ThreeDotsLabs/wild-workouts-go-ddd-example/internal/trainer/firestore.go Full source Production deployment tl;dr You can deploy your own version of Wild Workouts with one command: > cd terraform/ > make Fill all required parameters: project [current: wild-workouts project]: # <----- put your Wild Workouts Google Cloud project name here (it will be created) user [current: email@gmail.com]: # <----- put your Google (Gmail, G-suite etc.) e-mail here billing_account [current: My billing account]: # <----- your billing account name, can be found here https://console.cloud.google.com/billing region [current: europe-west1]: firebase_location [current: europe-west]: # it may take a couple of minutes... The setup is almost done! Now you need to enable Email/Password provider in the Firebase console. To do this, visit https://console.firebase.google.com/u/0/project/[your-project]/authentication/providers You can also downgrade the subscription plan to Spark (it's set to Blaze by default). The Spark plan is completely free and has all features needed for running this project. Congratulations! Your project should be available at: https://[your-project].web.app If it's not, check if the build finished successfully: https://console.cloud.google.com/cloud-build/builds?project=[your-project] If you need help, feel free to contact us at https://threedots.tech We describe the deployment in detail in the next articles. What’s next? That’s all for today. In the next article, I will cover internal gRPC communication between services. After that, I will cover HTTP Firebase authentication. The code with gRPC communication and authentication is already on our GitHub. Feel free to read, run, and experiment with it. Deployment and infrastructure In parallel, Miłosz is working on articles describing the deployment and infrastructure part. They will cover in detail: Terraform Cloud Run CI/CD Firebase hosting What is wrong with this application?! After finishing all articles describing the current application, we start the part related to refactoring and adding new features to Wild Workouts. We don’t want to use many fancy techniques just to make our CVs look better. Our goal is to solve issues present in the application using Domain-Driven Design, Clean Architecture, CQRS, Event Storming, and Event Modeling. We do this through refactoring. It should then be clear what issues are solved and how the implementation becomes cleaner. Maybe during this process we’ll even discover that some of these techniques aren’t useful in Go. Who knows? 😉 Do you see any of the issues we included in this application? Please post your guess in the comments. :) See you next week.

If you compare MySQL or PostgreSQL with Kafka or RabbitMQ, at first, it seems they are entirely different software. And usually, that’s true, as you would use them for quite different tasks. What they have in common is processing streams of data, and they specialize in specific ways of doing it. While Kafka and RabbitMQ are popular examples of Pub/Subs (also known as message queues or stream processing platforms), I’d like to share some patterns for using SQL databases as Pub/Subs as well. But shouldn’t we pick the right tool for the job? That’s right. However, SQL databases are readily available if you don’t have access to the dedicated messaging infrastructure. They also have some unique features that turn out useful while working with message-driven applications. I’m going to show three examples of such cases with solutions based on the Watermill library for Go, although the patterns stay the same regardless of the language. But first, let’s ask the fundamental question. What is a Pub/Sub? If you generalize it enough, you can define any Pub/Sub client with just two methods (as the name suggests): one for publishing a new message to a topic, and the other for subscribing for messages from a topic. Publish(topic, message) Subscribe(topic) stream<message> So how do you implement this interface for an SQL database? The simplest way is to use INSERT statements for publishing and SELECT for consuming messages. More advanced subscribing patterns can be achieved with mechanisms like MySQL’s binlog or PostgreSQL’s LISTEN and NOTIFY commands. But for most use cases, the simple approach should be good enough. Using Watermill The two-method pseudocode above is also the base idea behind Watermill’s Publisher and Subscriber interfaces. It turns out you can hide quite a lot of complexity behind this abstraction. To implement any Pub/Sub, all you have to do is implement these two methods. One of the supported Pub/Subs is the SQL Pub/Sub. All examples below include a short snippet of Watermill’s handler, which is a function that receives a message and can return new messages to be published. func(msg *message.Message) ([]*message.Message, error) See the Getting started guide for more details. Pattern 1: Persistent Event Log Message queues offer various features depending on the design. That’s why no one fits all use cases. One of such features is persistent storage. For example, Apache Kafka comes out of the box with support for storing messages on topics as long as you need it. Others, like Google Cloud Pub/Sub*,* treat messages as ephemeral and delete them after they are consumed or even after several days without being acknowledged. You don’t always need to store messages, but it comes in useful when working with domain events. It allows processing events from the past again, sort of like “going back in time” (a similar approach to Event Sourcing, although much simplified). If you’d like to keep your current messaging software but need persistent storage for events, an SQL database might be the right choice. Implementation The Watermill handler for this pattern is the simplest one. It works just like a transparent proxy, moving messages from one system to the other. The Router component does all the work in the background. All you have to do is define the subscriber, publisher, and corresponding topics. router.AddHandler( "googlecloud-to-mysql", // unique handler name googleCloudTopic, // subscriber's topic googleCloudSubscriber, // subscriber mysqlTable, // publisher's topic mysqlPublisher, // publisher func(msg *message.Message) ([]*message.Message, error) { return []*message.Message{msg}, nil }, ) See full source: persistent-event-log example. Pattern 2: Transactional Events Consider user signup, a classic use case in every web application. The requirements are straightforward. You need to store user details in an SQL table and publish UserSignedUp event, so other services in your system consume it and take some action (e.g., send a newsletter, update statistics report, fill personal details in a CRM). The issue is, you can’t execute SQL statements and publish events both at the same time. You have two options: Save the user first, then publish the event — if publishing fails, the SQL table is up to date, but the rest of the system is inconsistent. The user can log in, but doesn’t receive a newsletter, the monthly statistics report is incorrect, and your sales team doesn’t know whom to call. Other teams that depend on the event will never receive it. Publish the event first, then save the user — if saving the user fails, the system is consistent, but the SQL table is outdated. The user can’t log in but receives a newsletter. Your reports show more users than there are using your product, and your sales team nags the frustrated user with calls. Let’s hope your “unsubscribe” button still works. You can come up with complicated solutions for this problem, like two-phase commits or periodically synchronizing all systems. There’s also a more straightforward pattern using the SQL database. The idea is to insert the event into a separate table instead of publishing it directly. The critical part is doing it within the same transaction as saving the user. It guarantees both or none are stored. In the background, a separate process is subscribed to the table with events and publishes all incoming rows on some message queue. Note One thing to keep in mind is that eventual consistency must be acceptable in your system. It’s possible that the background process stops working for some reason and publishes events sometime in the future. But if you’re already working with events, I hope you did consider that. Implementation When using Watermill, the handler function is very similar to the previous example. After all, it’s the same kind of event proxy, but this time the database is the entry point. router.AddHandler( "mysql-to-kafka", mysqlTable, mysqlSubscriber, kafkaTopic, kafkaPublisher, func(msg *message.Message) ([]*message.Message, error) { return []*message.Message{msg}, nil }, ) See full source: transactional-events example. Pattern 3: Synchronizing Databases Synchronizing two databases isn’t a new problem. That’s how replication already worked for decades. It operates on low-level mechanisms (like the binary log), but the idea is very close to a Pub/Sub: one database is publishing changes, and the other subscribes to them. A bit harder task is moving data between two different database engines, each with distinct table schema. If replication is using an interface similar to a Pub/Sub, can we use the same principles for migration as well? Let’s consider migrating a MySQL table to PostgreSQL. It would look roughly like this: The application inserts a new row into the MySQL table. Synchronization service receives this as an update (because it subscribed to that table). The service translates the row into the destination format (only if schemas differ, so this step is optional). The service inserts a new row into the PostgreSQL table (publish). Note The hard part about implementing this handler is making sure we keep proper order of events, don’t miss any rows, and don’t duplicate them. The first two are solved for you by Watermill out of the box, but you need to take care of de-duplicating the events yourself. A simple approach is to make sure your event handlers are idempotent. Implementation If table schemas in both databases are the same, the handler can be as short, as in the first two patterns. In this example, we deal with distinct tables, so we need to implement the translation part as well. First of all, we need to rename the user column to username. Second, merge first_name and last_name columns into full_name. You can see that the gob library is used for encoding and decoding the payloads, but you can use anything you like, as long as it serializes to a slice of bytes. Note By default, the SQL subscriber consumes all existing records in the table and listens for any new incoming rows. So you can start with migrating the current data and then keep the service running to keep both databases up to date. The synchronization will work as long as your tables are append-only - see Limitations below for details. type mysqlUser struct { ID int User string FirstName string LastName string CreatedAt time.Time } type postgresUser struct { ID int Username string FullName string CreatedAt time.Time } router.AddHandler( "mysql-to-postgres", mysqlTable, mysqlSubscriber, postgresTable, postgresPublisher, func(msg *message.Message) ([]*message.Message, error) { // Decode the row coming from the MySQL table decoder := gob.NewDecoder(bytes.NewBuffer(msg.Payload)) originUser := mysqlUser{} err := decoder.Decode(&originUser) if err != nil { return nil, err } // Translate from the MySQL schema to the PostgreSQL schema newUser := postgresUser{ ID: originUser.ID, Username: originUser.User, FullName: fmt.Sprintf("%s %s", originUser.FirstName, originUser.LastName), CreatedAt: originUser.CreatedAt, } // Encode the row to be saved in the PstgreSQL table var payload bytes.Buffer encoder := gob.NewEncoder(&payload) err = encoder.Encode(newUser) if err != nil { return nil, err } newMessage := message.NewMessage(watermill.NewULID(), payload.Bytes()) return []*message.Message{newMessage}, nil }, ) This example is complex enough to introduce a new Schema Adapter. Refer to the full source to see the implementation: synchronizing-databases example. Limitations While you can implement all patterns above using Watermill’s SQL Pub/Sub, it has some limitations. Currently, only the INSERT statement triggers new messages sent to the subscriber. The good news is there’s planned support for MySQL binlog and PostgreSQL LISTEN/NOTIFY subscribers, and they allow listening for updates and deletes as well. The SQL Pub/Sub is also a low performer compared to other Pub/Subs. The current MySQL subscriber can process about 150 messages per second. That’s almost 13 million messages daily, so there’s a chance that’s more than enough for your needs. Update: There’s an interesting discussion on Hacker News on using databases as Pub/Subs and related topics.

54 days of work, 12,909 lines of code, 47 Monsters and 42 KFC Twisters later finally it is Watermill v0.3.0! To keep it short, let’s go through the changes. One important thing: at the end of this post there is a 3 question survey. Please take a moment to fill it out, it will help us make Watermill even better. CQRS component One of the most important parts of the v0.3.0 release is built-in CQRS support. The implementation itself is pretty simple and based on the existing message.Router and message.PubSub examples. We built CQRS in a way that allows you to use this component with your pre-existing (even not Golang-based) infrastructure. All you need to do is implement your own Marshaler. If you are interested in how to use CQRS, please see watermill.io documentation. Thanks to @adymitruk for a bit of consulting and inspirations! Prometheus metrics component Second, not less important feature is the metrics component. Our goal was to allow you to add metrics to your Watermill application in one minute. All you need to do, is add a few lines: prometheusRegistry, closeMetricsServer := metrics.CreateRegistryAndServeHTTP(":8081") defer closeMetricsServer() metrics.NewPrometheusMetricsBuilder(prometheusRegistry, "", "").AddPrometheusRouterMetrics(router) and load our dashboard in your Grafana instance. If you are not using Prometheus or Grafana in your system, you may use this component as the reference implementation. It is not using any internals, but only publicly available handler middlewares and pub/sub decorators. For the detailed documentation, please check watermill.io website. All kudos for the implementation goes to @maclav3! AMQP (RabbitMQ) Pub/Sub implementation We are happy to announce that AMQP Pub/Sub has joined the list of supported Pub/Sub implementations. You can now use it in addition to Kafka, Google Cloud Pub/Sub, and NATS. The title may be a bit confusing, because RabbitMQ can work both as a Pub/Sub (fanout) and a queue - and we support both ways. There are available pre-created configs which allow you to create a Pub/Sub or a queue. The implementation is based on github.com/streadway/amqp, but offers higher-level API and reconnect support. You can use it as a standalone Pub/Sub, but I kindly recommend to use message.Router. As always, I encourage you to check getting started with Watermill and watermill.io documentation. Please keep in mind, that there are some glossary incompatibilities between Watermill and AMQP (for example, topic has a different meaning). They are all explained in the documentation. HTTP Publisher Can HTTP work as a Pub/Sub? Of course! At least when it implements Watermill’s message.PubSub interface. :) But why? For example, you can create deadly easy Kafka-based webhooks. See more details on this publisher in the documentation. Thanks @maclav3 for the implementation! Added context.Context to Subscribe Subscribe now accepts a context as the first parameter. When the context is canceled, the subscription will be closed. This context is also added to the received message. GoChannel Pub/Sub update added Persistent option, which stores all produced message in memory and sends them to all new subscribers. added BlockPublishUntilSubscriberAck, which will block Publish until the message is Acked by all consumers. UUID and ULID functions We were using satori/go.uuid in many places. Unfortunately, this library is no longer maintained. We replaced all UUID functions in the project with just added watermill.NewUUID(), watermill.NewULID() and watermill.NewShortUUID(). You may use our functions or your own implementation - UUID is still a string. Misc Travis CI replaced with GitLab CI. Thanks @m1_10sz! Rewritten and optimized pub/sub tests. Updated and added new examples and docs. Made some breaking changes. Added SubscribeInitialize method for some subscribers, which require initialization before publish (for example: create AMQP exchange, queue, etc.). If you are using Subscribe before Publish, it is not required! Better stack traces for middleware.Recoverer. Added subscriber.BulkReadWithDeduplication. Added Publishers and Subscribers decorators support to the router (used in the metrics component, for example). Added watermill.LogFields.Copy method. Added watermill.LoggerAdapter.With method to the interface. Survey In the end, we would like to ask for your feedback. We created a short 1-minute survey. We will be grateful for your answers, as it will help us to keep improving Watermill. Support If you didn’t find an answer to your question in the documentation, you can hit me on Twitter: @roblaszczak. Also, check out the #watermill channel on gophers.slack.com (invite here).

This time I’d like to touch on a few more advanced topics related to GitLab CI. The common theme here is implementing custom features within your pipeline. Again, most of the tips are specific to GitLab, but some could be easily applied in other CI systems as well. Running integration tests Checking your code with unit tests is usually easy to plug into any CI system. It usually is as simple as running one command built in your language’s toolset. In these tests, you will most likely use various mocks and stubs to hide implementation details and focus on testing particular logic. For example, you can use in-memory database as storage or write stub HTTP clients that always return some prepared payloads. However, eventually, you will need to run integration tests to cover more unusual test cases. I don’t want to start a discussion on all possible test types here, so let me say this: by integration tests I mean tests that use some sort of external resource. This can be a real database server, an HTTP service, attachable storage, and so on. GitLab makes it trivial to run attached resources as Docker containers, linked with the container running your scripts. You can use the services keyword to define these dependencies. They will be accessible by their image name or your chosen name if you specify it in the alias field. This is a simple example of using an attached MySQL container: integration_tests: stage: tests services: - name: mysql:8 alias: db script: - ./run_tests.sh db:3306 In that case, you have to connect to the db hostname in your test scripts. Using an alias is usually a good idea, as you can switch the images without changing the test code. For example, you could change mysql image with mariadb and the script would still run correctly. Note Waiting for containers Because attached containers can take a while to warm up, you will probably need to wait before sending any requests. A simple way to do this is by using wait-for-it.sh script with a specified timeout. Using Docker Compose Using services should be good enough for most of your use cases. However, sometimes you need the external services to communicate with each other. One example may be running Kafka and ZooKeeper in two separate containers (the official images are built this way). Another is running tests with a dynamic number of nodes, like selenium. For running services like this, a better solution is to use Docker Compose. version: '3' services: zookeeper: image: confluentinc/cp-zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 ports: - 9092:9092 If you’re running your own GitLab runners on a trusted server, you can use Shell executor to run Docker Compose. Another option may be using Docker in Docker (dind) container. Read this first, though. One way of using Compose is to set up the environment, run tests and tear it down. A simple bash script would look like this: docker-compose up -d ./run_tests.sh localhost:9092 docker-compose down This solution is fine as long as your tests can run in a minimal environment. It may happen though, that you will need some dependencies installed. There is another way of running tests in Docker Compose that lets you built your own docker image with the test environment. You make one of the containers run the tests and exit with proper code. version: '3' services: zookeeper: image: confluentinc/cp-zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 tests: image: registry.example.com/some-image command: ./run_tests.sh kafka:9092 Notice that we got rid of the ports mapping. In this example, tests can communicate directly with all services. Tests can now be executed with a single command: docker-compose up --exit-code-from tests The --exit-code-from option implies --abort-on-container-exit, which means the whole environment started by docker-compose up will be stopped after one of the containers exits. The exit code of that command will be equal to the exit code of the chosen service (tests in the example above). So if the command running your tests exits with a non-zero code, the whole docker-compose up command will exit with this code. Using labels as CI tags A word of warning: this is rather an unusual idea, but one I’ve found very useful and flexible. As you probably know, GitLab has Labels feature on project and group levels. Labels can be attached to issues and merge requests. They have no relation to the pipelines, though. With a bit of hacking, you can access the merge request labels in the job scripts. As of GitLab 11.6, this is now even easier, as there is a CI_MERGE_REQUEST_IID environment variable (yes, it’s IID, not ID) available if your pipeline is using only: merge_requests. Note If you’re not using only: merge_requests or running an older version of GitLab, you can still get the MR with an API call. curl "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/commits/$CI_COMMIT_SHA/merge_requests?private_token=$GITLAB_TOKEN" iid is the field you’re after. Just be aware that this could return multiple MRs for a given commit. When you get the MR IID, all that’s left is to call the Merge Requests API and use the labels field from the response. curl "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID?private_token=$GITLAB_TOKEN" Authorization Sadly, using $CI_JOB_TOKEN to access project’s API is not possible at the moment (at least unless the project is public). If your project has restricted access (internal or private), you will need to generate a personal API token to authorize with the GitLab API. This is not the most secure solution, though, so be careful. If this token would be leaked, someone could gain write access to all of your projects. One way of reducing the risk is to create a separate account with read-only access to the repository and generate a personal token for this account. Note How safe are your variables? A few versions ago, the Variables section has been called Secret Variables, which could sound like they were designed to store credentials and sensitive data safe. Frankly, the variables are just hidden from users who don’t have Maintainer permissions. They are not encrypted on disk and can be easily leaked as environment variables in your scripts. Keep it in mind when adding any variables and consider storing your secrets in safer solutions (e.g. HashiCorp’s Vault). Use cases It’s now up to you to decide what to do with the list of labels. Some ideas: Use it for tests segmentation. Introduce key-value semantics with a colon (e.g. labels like tests:auth, tests:user) Enable some specific features for jobs. Allow debugging of specific jobs when a label is present. Calling external APIs While GitLab comes with a suite of included features, it’s very likely you still use other tools that could be integrated with your pipelines. The simplest way to do it is of course by good old curl calls. If you’re writing your own tools, you could also have it listen to GitLab’s Webhooks (look up the Integrations tab in the project’s settings). If you’d use it with some critical systems though, make sure they are highly-available. Example: Grafana annotations If you’re using Grafana, annotations are a nice way of marking on graphs an event that happened in time. While it’s possible to add them by clicking in the GUI, they can also be added by the Grafana REST API. You will need to generate an API Key to access the API. Consider creating a dedicated user with restricted access. Set two variables in your project settings: GRAFANA_URL - the URL of your Grafana instance (e.g. https://grafana.example.com) GRAFANA_APIKEY - the generated API Key To keep it re-usable, you can put the script in the common scripts repository: #!/bin/bash set -e if [ $# -lt 2 ]; then echo "Usage: $0 <text> <tag>" exit 1 fi readonly text="$1" readonly tag="$2" readonly time="$(date +%s)000" cat >./payload.json <<EOF { "text": "$text", "tags": ["$tag"], "time": $time, "timeEnd": $time } EOF curl -X POST "$GRAFANA_URL/api/annotations" \ -H "Authorization: Bearer $GRAFANA_APIKEY" \ -H "content-type: application/json" \ -d @./payload.json Now call it with proper parameters in the CI definition: deploy: stage: deploy script: - $SCRIPTS_DIR/deploy.sh production - $SCRIPTS_DIR/grafana-annotation.sh "$VERSION deployed to production" deploy-production You could also put it in the deploy.sh script to keep the CI definition even simpler. Bonus quick tips GitLab has a great documentation on all possible keywords in the CI definition. I don’t want to duplicate the content here, but I’d like to point out a few useful use cases. Click on the headers to view the docs. Advanced only/except usage Use pattern matching on variables to enable custom builds for some of the branches. You don’t want to overuse it, but if you quickly need to push a hotfix, this might help. only: refs: - branches variables: - $CI_COMMIT_REF_NAME =~ /^hotfix/ GitLab includes a lot of predefined variables in each CI job, make use of them. YAML anchors Use them to avoid duplication. Since 11.3 you can also use the extends keyword. .common_before_script: &common_before_script before_script: - ... - ... deploy: <<: *common_before_script Skipping dependencies By default, all artifacts built in the pipeline will be passed to all following jobs. You can save some time and disk space by explicitly listing artifacts the jobs depends on: dependencies: - build Alternatively, skip them entirely if none are required: dependencies: [] Git strategy Skip cloning the repository if the job won’t use its files. variables: GIT_STRATEGY: none That’s it! Thanks for reading! Hit me up with feedback or questions on Twitter or Reddit. For more GitLab tips, check out my previous posts: Keeping common scripts in GitLab CI Automatic Semantic Versioning in GitLab CI

This post is a quick how-to for starting a new project in Go. It features: Hot code reloading Running multiple Docker containers with Docker Compose Using Go Modules for managing dependencies It’s best to show the above working together with an example project. We’re going to set up two separate services communicating with messages over NATS. The first one will receive messages on an HTTP endpoint and then publish them to a NATS topic. The other will subscribe to this topic and print incoming messages on the standard output. It’s pretty simple architecture, right? To keep the examples short and make handling messages easier, we’re going to use the Watermill library. I won’t go through all Go code here, but you can clone the example repository. Requirements You’re going to need Go 1.11+, Docker and Docker Compose installed on your local machine. Go Modules Go Modules were introduced in Go 1.11 and are (hopefully) the final solution for managing dependencies. I recommend keeping the repository outside of GOPATH if you’re going to use them. If you’d prefer to use GOPATH anyway, make sure you’ve set GO111MODULE=on variable. Let’s initialize modules for each of our packages (inside subscriber and publisher directories). go mod init github.com/ThreeDotsLabs/nats-example/subscriber go mod init github.com/ThreeDotsLabs/nats-example/publisher Download dependencies: go get github.com/ThreeDotsLabs/watermill go get github.com/nats-io/go-nats-streaming Add an extra ULID library only for the publisher: go get github.com/oklog/ulid Dockerfile The next step is creating a Docker image for containers that will run both applications. The Dockerfile is fairly simple: FROM golang:1.11.2-stretch RUN go get github.com/cespare/reflex COPY reflex.conf / ENTRYPOINT ["reflex", "-c", "/reflex.conf"] The image is based on the official Go image. Usually, you’d want to stick with the smaller alpine version, but since it is missing some dependencies for compiling reflex, I’m using the stretch version here. It installs and runs reflex which will be used for hot recompiling the code. It can be very useful for quickly testing your changes. Reflex is configured by reflex.conf, which in our case is just one line: -r '(\.go$|go\.mod)' -s go run . What the command means is: “watch for changes to go.mod and all files ending in .go and execute go run . when it happens. The -s flag stands for service and will make reflex kill previously run command before starting it again, which is exactly what we want. Note Reflex is not a regular dependency. It is installed and used only in the docker image and won’t be deployed with your application. Thanks to /u/habarnam for pointing that out. Docker Compose We’ll use Docker Compose for running multiple services within a shared network. It allows you to spin up containers with services or databases without cluttering your local machine. Also, other developers can set up the project by cloning the repository and running one command, instead of manually installing dependencies and compiling the application. Put the docker-compose.yml file in the repository: version: '3' services: publisher: build: . volumes: - ./publisher:/app - $GOPATH/pkg/mod/cache:/go/pkg/mod/cache working_dir: /app env_file: - .env ports: - 5000:5000 subscriber: build: . volumes: - ./subscriber:/app - $GOPATH/pkg/mod/cache:/go/pkg/mod/cache working_dir: /app env_file: - .env nats: image: nats-streaming:0.11.2 restart: on-failure It defines three services. Both publisher and subscriber are based on the Dockerfile we’ve created before (this is done by the build: . option). Each application’s code is mounted in the working directory, so go run . will run the proper package. The ports part in the publisher service will bind port 5000 on your local system and map it to the port inside container. I’ve also mounted $GOPATH/pkg/mod/cache directory, to use existing cache for dependencies, speeding up downloads. The services are configured by environment variables, kept in a separate file: PORT=5000 NATS_TOPIC=example NATS_CLUSTER_ID=test-cluster NATS_URL=nats://nats:4222 Notice the nats hostname in the NATS_URL variable - this is the name of the service defined in the docker-compose.yml file. Running To start the environment, run: docker-compose up You’ll see the output of all three services. In another terminal, try sending an example request to the publisher service with curl: $ curl -X POST http://localhost:5000 --data "this is my message" Sent message: this is my message with ID 01D09P02SBW5D0QPWP14QQZJWH You should see the message delivered in the subscriber log output: subscriber_1 | [00] 2019/01/03 11:01:27 received message: 01D09P02SBW5D0QPWP14QQZJWH, payload: this is my message Now, try editing either of the main.go files in your editor. After you save the file, appropriate service should be recompiled and restarted. When finished, you can kill the environment with Ctrl-C in the terminal it is running. You can also delete all containers by running docker-compose down. What’s next? Now that you’ve built Docker images of the services it’s the first step towards deploying it. We’ll look into it in the next post. If you’d like to explore more docker-compose definitions, check Watermill Getting started for Kafka and Google Cloud Pub/Sub examples.

Let me start by thanking all contributors for feedback on Watermill - it drives us to add new features. Thanks! It’s been almost a month since the initial release of Watermill. However, it’s just the beginning and we are still working hard to ship new features. What is new in Watermill 0.2? Documentation - watermill.io Godoc is great. However, it’s functionality is sometimes too limited to express more complicated documentation. For that reason, we’ve created watermill.io. The most important parts of the documentation are: Getting Started guide which will explain you the most basic concepts of the Watermill by the examples. Pub/Sub’s implementations which contains detailed documentation of all built-in Pub/Sub’s implementations. Router documentation which explains how Messages Router is working. All documentation topics Of course, we are not abandoning godoc. You can still find all important information in the comments (in fact, a large part of watermill.io is generated directly from godocs).` Added context.Context to the message. From now on, the Message holds a Context, which carries deadlines, cancelation signals, and other message-scoped values. Unlike metadata, it is not serialized nor sent to the Pub/Sub. Replaced Kafka Pub/Sub implementation with Sarama Watermill development started when Sarama didn’t have support for consumer groups yet. Fortunately, it is now available. Thanks to this fact we could replace librdkafka implementation with Sarama. Using Sarama allowed us to get entirely rid of the cgo dependency in Watermill. Another great bonus is a great improvement of messages publishing performance (quick benchmarks showed upgrade from 10k messages/s to 75k messages/s). Because the client implementation has been replaced, this change is not backward compatible. Kafka Breaking changes and API cleanups: kafka.NewCustomPublisher has been removed, please use NewPublisher instead. kafka.ConfluentConsumerConstructor has been removed and is no longer needed. kafka.SubscriberConfig.NoConsumerGroup is no longer needed. You can now just pass empty kafka.SubscriberConfig.ConsumerGroup. kafka.AutoOffsetReset has been removed. Please use github.com/Shopify/sarama.Config.Consumer.Offsets.Initial passed to NewSubscriber overwriteSaramaConfig argument instead. kafka.ConsumersCount has been removed and is no longer needed. kafka.KafkaConfigOverwrite has been removed and is no longer needed. You can now pass sarama.Config to NewSubscriber and NewPublisher. kafka.NewConfluentSubscriber has been removed. Please use kafka.NewSubscriber instead. kafka.NewCustomConfluentSubscriber has been removed. Please use kafka.NewSubscriber instead. kafka.DefaultConfluentConsumerConstructor has been removed and is no longer needed. kafka.Marshaler and kafka.Unmarshaler interfaces have been changed to be compatible with the Sarama API. Google Cloud Pub/Sub Cloud Pub/Sub brings the flexibility and reliability of enterprise message-oriented middleware to the cloud. At the same time, Cloud Pub/Sub is scalable, durable event ingestion and delivery system that serves as a foundation for modern stream analytics pipelines. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication among independently written applications. Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate systems hosted on the Google Cloud Platform and externally. Detailed documentation can be found at watermill.io and in the Getting Started Google Cloud section. Credits goes to @maclav3. Thanks for the contribution! NATS Streaming Pub/Sub NATS Streaming is a data streaming system powered by NATS, and written in the Go programming language. The executable name for the NATS Streaming server is nats-streaming-server. NATS Streaming embeds, extends, and interoperates seamlessly with the core NATS platform. Detailed documentation can be found at watermill.io and in Getting Started NATS section. Built in cooperation with @Nykakin, @maclav3 and @m110. What’s next? If there’s something missing in Watermill, feel free to post a new feature request or give thumbs up for the existing issue. For now, Event sourcing, Sagas support and MySQL Binlog subscriber seem to be the most popular. Also, the HTTP Publisher sounds like not too much of work and may provide some interesting functionalities. Support If you didn’t find an answer to your question in the documentation, you can hit me at Twitter: @roblaszczak. Also, check out the #watermill channel on gophers.slack.com (invite here).

In the previous post I showed how to keep all the scripts used in the CI in one repository. Let’s see what more advanced scripts you could put in there. This time I’d like to show how to add automatic versioning to your pipeline. You will also see how to push commits to your repository within the CI jobs. But first, let’s start with some background. Picking your flow One of the things I love about GitLab is its flexibility for setting up your own CI workflow. Whether you’re doing one release each week, every other month, following classic gitflow or practicing continuous deployment, it’s just a matter of the config. As a proponent of continuous delivery principles, I usually stick to a protected master branch and short-lived feature branches that are merged after code review. Each commit on the master is automatically deployed to some sort of staging environment (preferably identical with prod) and then can be manually promoted to production. If you have a solid suite of multi-layer tests, this workflow will let you deploy multiple times a day. Versioning Whatever flow you pick, versioning of your application becomes essential in many steps of the pipeline. For example, it is crucial to know precisely to which version you should rollback if something goes wrong. You should always be able to find the commit for a given version number quickly. The versioning system is up to you and will depend on your workflow. I usually stick with Semantic Versioning, as it feels natural and it is already a standard for many projects, especially for libraries. So the first commit on the master branch would get version 1.0.0, the next one 1.0.1, then 1.0.2, and so on (alternatively, you can start with 0.0.1 and treat 1.0.0 as first public release). Where to keep the versions? Some good practices from my experience: Don’t keep it in the code or a file committed to the repository. Git tags are a perfect fit for keeping versions tied with commits. Automate it. Don’t waste developers’ time on wondering which version number to pick. So basically, you want to have some automated version tagging wired into your CI system. Implementation The essential piece handling versioning is a script that runs on every job on the master branch. This script should take the most recent version, bump it, add a tag and push it to the remote repository. Note You probably don’t need to generate versions on other branches. There are some valid use cases for this, but that’s a story for another post. This time I’m going to use Python and the python-semver library. If you’d prefer to stick with bash, take a look at the semver-tool. Let’s see what the script should do. 1. Pick the most recent version and bump it. This is simple enough. You can extract the most recent tag by running git describe --tags. The semver library can then be used to bump the version. Our main function might look something like this (full source): def main(): try: latest = git("describe", "--tags").decode().strip() except subprocess.CalledProcessError: # No tags in the repository version = "1.0.0" else: # Skip already tagged commits if '-' not in latest: print(latest) return 0 version = bump(latest) tag_repo(version) print(version) return 0 There is one catch, though - how will you decide whether to bump patch, minor or major version? def bump(latest): # TODO decide what to bump # Then use bump_patch, bump_minor or bump_major return semver.bump_patch(latest) You will have to mark it in the commit somehow. I won’t implement this in the example to keep it simple, but here are some ideas that could work: Make bumping patch the default behavior, as it is the most common operation. An intention to bump minor or major version can be marked in the commit message, e.g., by using some phrase like #minor or bump-minor. Similarly, you could attach labels to the Merge Requests called bump-minor and bump-major. You could come up with a script that detects whether any breaking changes were introduced or new functionalities added. 2. Add a new tag and push it to the remote repository. Authentication This step requires the CI job to have write access to the repository. Sadly, as of now, GitLab still doesn’t support pushing changes back to the repository out of the box. Deploy tokens shown in the previous post can’t be used here, since they allow only read-only access. So we’re left with Deploy Keys. First, generate a new key on your local machine (no passphrase): ssh-keygen -t rsa -b 4096 Add the public part as a new Deploy Key in the Settings -> Repository section. Make sure to check the “Write access allowed” option. Add the private part as a new Variable in the CI/CD section. The name is up to you; I’ll stick with SSH_PRIVATE_KEY. After you’ve saved the keys in GitLab, it is a good idea to delete the private key file (or better yet, shred it). All that’s left is to add the SSH key to the CI definition. There are several ways to do it, one of them looks like this (change gitlab.com with your hostname if you’re using self-hosted GitLab): script: - mkdir -p ~/.ssh && chmod 700 ~/.ssh - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts && chmod 644 ~/.ssh/known_hosts - eval $(ssh-agent -s) - ssh-add <(echo "$SSH_PRIVATE_KEY") Pushing the tag A simple git push won’t work right away, as the repository is cloned using HTTPS, not SSH. The simplest solution is to transform the remote push URL by using a regex. def tag_repo(tag): url = os.environ["CI_REPOSITORY_URL"] # Transforms the repository URL to the SSH URL # Example input: https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/threedotslabs/ci-examples.git # Example output: git@gitlab.com:threedotslabs/ci-examples.git push_url = re.sub(r'.+@([^/]+)/', r'git@\1:', url) git("remote", "set-url", "--push", "origin", push_url) git("tag", tag) git("push", "origin", tag) 3. Pass information about the version to the next build steps. It’s very likely that more than one job in your pipeline will need to know the generated version. The idiomatic way to achieve this is to pass a file with the version as an artifact. version: image: python:3.7-stretch stage: version script: - pip install semver - $SCRIPTS_DIR/common/gen-semver > version artifacts: paths: - version only: - branches build: image: golang:1.11 stage: build script: - export VERSION="unknown" - "[ -f ./version ] && export VERSION=$(cat ./version)" - $SCRIPTS_DIR/golang/build-semver . example-server main.Version "$VERSION" artifacts: paths: - bin/ only: - branches Note If you missed the common scripts post or don’t want to use it, you can also commit the script in your application repository. The next steps can now read the ./version file. You can also make it more convenient with a one-liner placed in before_script: before_script: - [ -f ./version ] && export VERSION=$(cat ./version) Don’t run on tags Remember to put proper only setting in your step definition or your automatic tag pushes will trigger new pipelines. Limiting it to branches should be easy enough: only: - branches What about changelogs? Some workflows generate the version based on a CHANGELOG file in the repository. I don’t recommend this approach, as this forces developers to come up with versions, duplicates the commit messages and is prone to merge conflicts. Instead, you can treat the git log itself as a changelog (focus on great commit messages). With the automated versioning set up, you have everything you need in the commit - author, date, version, and message. Have the CI generate a changelog file for you and upload it somewhere with each new version. Summary This is just a basic setup that can be tweaked for more specific cases. Hit me on Twitter if you have any questions or if you’d like to share your own versioning process. See the full examples here: gen-semver script build-semver script .gitlab-ci.yaml definition External links: Semantic versioning Take a look at the semantic-release tool to compare some ideas.

Watermill is a Go library for working efficiently with message streams. It is intended as a library for building event-driven applications, enabling event sourcing, CQRS, RPC over messages, sagas. Why? Lack of standard messaging library There are many third party and standard library tools which help to implement standardized RPC or HTTP communication in Golang. There are also multiple third party HTTP routers and frameworks. But when you want to build a message-driven application, there are no libraries which are infrastructure-agnostic and not opinionated. Watermill is a library which will work with any implementation of Publisher and Subscriber interfaces: type Publisher interface { Publish(topic string, messages ...*Message) error Close() error } type Subscriber interface { Subscribe(topic string) (chan *Message, error) Close() error } For now, Watermill comes with the following implementations: Kafka Pub/Sub Go channel (intraprocess) Pub/Sub HTTP Subscriber In close future, there are plans for RabbitMQ (AMQP) Pub/Sub, Google Cloud PubSub and MySQL Binlog subscriber. High entry level When implementing a HTTP REST API nowadays, how do you go about it? Do you send it via TCP like this: Source: Golang code if _, ok := w.(io.ByteWriter); !ok { bw = bufio.NewWriter(w) w = bw } _, err = fmt.Fprintf(w, "%s %s HTTP/1.1\r\n", valueOrDefault(r.Method, "GET"), ruri) if err != nil { return err } // Header lines _, err = fmt.Fprintf(w, "Host: %s\r\n", host) if err != nil { return err } // [74 lines later] err = tw.writeBody(w) // [a lot of crazy stuff in writeBody] or just by calling: helloHandler := func(w http.ResponseWriter, req *http.Request) { io.WriteString(w, "Hello, world!\n") } For the same reason, you probably shouldn’t use your PubSub’s raw driver (for example for Kafka). Instead of this, you should use something that will deal with stuff like acknowledgment, errors handling and publishing created messages in a universal and consistent manner. It is also a lot easier for less experienced developers. Watermill works with high level handler functions, with abstraction level similar to Golang HTTP handlers: type HandlerFunc func(msg *Message) ([]*Message, error) All you need to do to process a message is to create a function with the HandlerFunc signature: func process(msg *message.Message) ([]*message.Message, error) { log.Println("received message: ", string(msg.Payload)) producedMessages := []*message.Message{ message.NewMessage(uuid.NewV4().String(), []byte(time.Now().String()), } return producedMessages, nil } Extensibility and elasticity Every organization and project is different. This is why Watermill is built so that every part that may differ between organizations can be configured. As said before, you can use any Pub/Sub or implement your own when needed. If you are using different message format, you can also inject your own message marshaller. Nowadays, any modern service must support metrics and distributed tracing. When building a message-driven application, often we need a posion queue or retrying support. Also, sometimes we need to limit the rate of message processing with throttling. Every one of these functionalities can be implemented by a middleware: type HandlerMiddleware func(h HandlerFunc) HandlerFunc Watermill is shipped with ready-to-use middlewares for these functionalities. Also, error handling can be different for some services. By default, we want to send an Ack when the processing was successful. But sometimes, we want to send an Ack just after receiving the message. It can be also done by using a middleware: func InstantAck(h message.HandlerFunc) message.HandlerFunc { return func(message *message.Message) ([]*message.Message, error) { message.Ack() return h(message) } } Fast Use of message-driven architecture and Golang is often dictated by high performance requirements. For that reason, Watermill can process hundreds of thousands of messages per second with low latency. But there’s still a lot of work to be done in this field. Performance updates and benchmarks are on the way! How to start? In https://watermill.io/ you can find detailed documentation of Watermill. I recommend to start with the Getting Started guide. Future plans For now, Watermill is in beta phase. Currently, our number one priority is to create a good documentation. We have also plans to implement some more components basing on our Messages Router and Pub/Sub: Sagas, CQRS , Event Sourcing . As mentioned before, we are planning to implement RabbitMQ (AMQP) Pub/Sub, Google Cloud PubSub and MySQL Binlog subscriber. If you need something that Watermill is missing, feel free to add a new issue. If one of the previously mentioned features is important for you, please add a thumbs up in GitHub issue.

With this post, I’d like to start a series of CI-related tips, targeted mostly at GitLab, since that’s my go-to tool for things CI/CD-related. I’m sure most of them could be easily applied to other CI systems, though. While GitLab does a great job at many things (repository hosting and related stuff, like MRs, pipelines, issue boards, etc.), it sometimes lacks more specialized features in some areas. With heavy usage of pipelines in my projects, I often had to look for workarounds and smart tricks. So I’d like to share some of them now, based on my experiences. The issue An example CI definition could look something like this: stages: - test - build - deploy test: image: golang:1.11 stage: test script: - go test ./... - go vet ./... build: image: golang:1.11 stage: build script: - go build -o ./bin/my-app ./cmd/my-app artifacts: paths: - bin/ deploy: stage: deploy script: - ... # upload bin/ somewhere I have some issues with using bare bash commands like this. See what happens when you need to change something in one of your jobs: You need to push a new commit in the application repository, triggering full pipeline for the project. Your changes won’t propagate to other branches, unless your commit from master is merged by other developers. Your changes will affect only this one repository. If you maintain more similar projects, you need to push the changes to all of them. Common scripts repository One of the things I missed when starting out with GitLab pipelines was a place for common scripts, which could be used by multiple repositories. At that time include was not yet available, but even now it hardly solves the issue for me. Reusing parts of the YAML definition is one thing, but the other is reusing the standard scripts across all of your projects. I’d prefer things like “build app”, “package app”, “deploy app” to be one-liners in the pipeline. I also feel that this fits nicely into the DevOps mindset: ops prepare the scripts, which are easy to use by developers. The scripts are developed using the same good practices as the application code itself: with code reviews and their own pipeline including unit tests and static checks. I usually end up with a mix of bash and Python scripts. Bash works well for simple scripts, but for more complex stuff, going with Python (or other scripting language you like) should save you some future headaches. Python is usually easier to read by other people too, but keep in mind it may not be included in every docker image you’ll be using in your jobs. Implementation The idea is pretty simple: fetch the scripts repository every time a job is run. This should be as simple as adding global settings in your .gitlab-ci.yml: variables: SCRIPTS_REPO: https://gitlab.com/threedotslabs/ci-scripts before_script: - export SCRIPTS_DIR=$(mktemp -d) - git clone -q --depth 1 "$SCRIPTS_REPO" "$SCRIPTS_DIR" You could also move the variables to the project’s CI/CD variables. Or include this config from another location. Or use your own docker images with the scripts repository baked in and run git pull in it. The basic idea stays the same. Note Passing --depth 1 option results in a shallow clone of the repository, i.e. truncated to the most recent commit from the master branch. Thanks to this, the overhead for each job is minimal. Learn more in docs. Example script Let’s say you have a bunch of Golang applications and you’d prefer to have a consistent build process for all of them. Why would you want it, when a simple go build should be enough? For example, you might want to bake the version number into every one of your services: it’s usually useful to know which version you’re running! Putting that into a common script ensures that none of your applications will skip this step. In the next post, I’ll show you how to generate semver versions for commits. For now, we’ll stick with git commit hash as our version number. #!/bin/bash # Build generic golang application. # # Example: # build-go cmd/server example-server pkg.version.Version set -e if [ "$#" -ne 3 ]; then echo "Usage: $0 <package> <target_binary> <version_var>" exit 1 fi readonly package="$1" readonly target_binary="$2" readonly version_var="$3" readonly bin_dir="$CI_PROJECT_DIR/bin/" mkdir -p "$bin_dir" go build -ldflags="-X $version_var=$CI_COMMIT_SHA" -o "$bin_dir/$target_binary" "$package" Note Don’t forget the set -e part or your bash scripts may silently fail and cause some hard to find bugs. This script expects that the resulting binary will be kept in the bin directory. It’s just another thing you might want to have consistent across repositories. Note Note that I used CI_ variables provided by GitLab. While they make the scripts really simple to run, it comes with the cost of more complicated testing, since you need to set up some environment variables before running scripts locally. If you prefer more explicit approach, just pass those variables as arguments to your script. This script should be put in your scripts repository. Don’t forget to set the execute permissions (chmod +x). The last thing you need to do is use the script in your application repository: build: image: golang:1.11 stage: build script: - $SCRIPTS_DIR/golang/build . example-server main.Version artifacts: paths: - bin/ And this is it! Your scripts should now always be up-to-date across your repositories. Of course this setup won’t make much sense if you’re using just one repository, so keep that in mind. Don’t overcomplicate it, if you can. Authentication If you’re using a private repository for your scripts, you’ll need to add some form of authentication to clone it. You can do this with either Deploy tokens or SSH deploy keys. Using deploy tokens is a bit easier, so let’s see it in action. In the scripts repository: generate new deploy token. You’ll find it in the Settings -> Repository -> Deploy tokens section. In the application repository: add the generated user and token as new variables called SCRIPTS_USER and SCRIPTS_TOKEN. You can do this in the Settings -> CI / CD -> Variables section. Modify the SCRIPTS_REPO variable in your application CI definition: variables: SCRIPTS_REPO: https://$SCRIPTS_USER:$SCRIPTS_TOKEN@gitlab.com/threedotslabs/ci-scripts Note Managing secrets across multiple projects is easier if you add variables at the group level. Testing the changes If you’ve worked with multiple repositories before, you should already know that it adds some synchronization overhead. For example, pushing a broken script to the master branch could break builds in all your projects. That’s why code reviews and unit tests are very much welcome. If unit testing the scripts is hard or just not possible, you could try testing them on branches first, before merging to master. You would just need to checkout to the chosen branch after cloning. What’s cool, you can retry the failed job this way just after pushing changes to the script, without running the complete pipeline again. What about Makefiles? Moving all bash commands to Makefile can tidy up your CI definition and allows developers to run the same checks and tests that will be run in the pipeline. For example, you could introduce make test target that will run unit tests, make lint for static checks and so on. Sadly, this will mean duplicating the Makefile across all your repositories and it doesn’t really solve the issues mentioned above. Makefiles also have their own quirks, so better make sure you know what you’re doing or you might run into unexpected issues (e.g. silently passing on errors). Wrapping up While the script we’ve used is really trivial, this shows how this integration can be used for keeping all CI-related scripts in one place. I’ll show more advanced examples in the next posts, so stay tuned! If you have any questions or you’d like to share other ideas for similar workflow, hit me on Twitter. Helpful resources example scripts repository example application repository Go tools and GitLab: How to do continuous integration like a boss Configuration of your jobs with .gitlab-ci.yml

Nowadays we can often hear that monolithic architecture is obsolete and responsible for all evil in IT. We often hear that microservices architecture is a silver bullet which helps to kill all this monolithic evil. But you probably know that there are almost no silver bullets in IT and every decision entails trade-offs. One of the most favored advantages of microservices architecture is good modules separation. You can deploy every service independently and services are easier to scale. Also, every team can have their own repository and use the technology of their choice. We can easily rewrite the entire service very fast. In return, we get possible network problems, latency, limited bandwidth. You must deal with potential bugs in the communication layer. It is much harder to debug and maintain from the developer and ops perspective. We must keep services API’s contracts up to date and fight with compatibility issues. We also can’t forget about potential performance drawbacks. The overall complexity is greater in many areas, starting with developing and ending in administration. In summary, regardless of all advantages, it is hard to build good microservices architecture. If you don’t trust my opinion, you can trust this guy: Almost all the cases where I’ve heard of a system that was built as a microservice system from scratch, it has ended up in serious trouble. Martin Fowler martinfowler.com Almost all the successful microservice stories have started with a monolith that got too big and was broken up. Martin Fowler martinfowler.com Let’s assume a crazy idea, which will not follow actual trends - we are developing the new application and we want to start with a monolith. Does it mean that we must live with all monolith disadvantages? Not at all. The truth is that a lot of these problems are usually not because of the monolith architecture, but because of lack of good programming practices and good design. Let’s look how an example good designed monolith application will look, compared to the microservices architecture version. In this article, we will name it Clean Monolith. Both are built with Clean Architecture rules. Clean architecture In a nutshell, Clean architecture assumes that your service should be separated into 4 layers: Domain - our domain logic and entities Application - some kind of glue for domain layer. For example, get the entity from the repository, call some method on this entity, save the entity in the repository. Also, it should take responsibility for all cross-cutting concerns like logging, transactions, instrumentation etc. It is also responsible for providing views (read models). Interfaces - thin layer which allows us to use the application, for example REST API, CLI Interface, Queue, etc. Infrastructure - database adapters, rest clients, in general implementation of interfaces from Domain/Application layer. Note Clean Architecture is a very similar concept to Hexagonal/Ports and adapters architecture/Onion Architecture of which you could already hear about. The key concept of clean architecture is that any layer cannot know anything about outside layer. For example, the domain should not be aware of application or infrastructure layer (so it doesn’t know that domain entities are persisted in MySQL for example). Also, the application layer should not be aware of how it is called (REST API, CLI, Queue listener, whatever). But how to achieve it? The answer is Inversion of Control. In short: hide implementation behind interfaces. Uncle Bob has already described it pretty good. At the bottom of the article I will provide a link with a full explanation of Clean Architecture. Monolith vs Microservices For the purposes of the article let’s use an example of a simple shop. This shop will allow us to: list products, place order, initialize payment in remote payments provider, receive notifications about payments and mark order as paid. Let’s take a look at the architecture diagram. I’ve created diagrams for both microservices and monolith example. Both are made in the spirit of Clean Architecture. (click to see in full resolution) Big picture microservices architecture diagram. Typically monolith architecture will look like Big Ball of Mud: Big Ball of Mud. But when we design architecture with the spirit of Clean Architecture it will look like this: (click to see in full resolution) Big picture monolithic architecture diagram. Do you see differences in this example? If you look closely, you can see that there are only in interfaces/infrastructure layers. The domain and application layers are basically the same (which is expected when we use DDD). I marked differences with orange arrows to make it more visible ;) Microservices can avoid it because we have a natural barrier between services (for example because of separated repositories), so we have enforced modules separation by design. But we can achieve some separation with monolith application and Clean Architecture helps here a lot, because we are limited to communicate only between infrastructure and interfaces layer. For example, we are not allowed to use anything from other bounded context’s domain/application/infrastructure in our bounded context. It is extremely important to not void this rule. The best way to enforce it is to have a tool which can validate it and plug it into the CI. Note I’ve developed a tool, which can do it for Golang: go-cleanarch. I’ve found a similar tool for PHP some time ago, but I cannot find it. There is a chance that tools like it exist for other languages. If you know any tool like this, please send it to me on Twitter: @roblaszczak and I’ll put it here :) Unfortunately, the fact that we have modules doesn’t mean that our architecture is good. Good modules separation is critical for proper working monolith and microservices. To make it good we should check the concept of Bounded Contexts (at the bottom of the article). Another great tool which will show (physically!) where your Bounded Contexts are and show you how your domain is working is Event Storming. There are a lot of people who say that persistence is just an implementation detail (Frameworks and Drivers part). They also say that persistence layer should be implemented so as to replace driver without any impact on any other layer than infrastructure. Let’s look at the big picture diagram and go one step further: let’s say that whether the application is a microservice or a monolith is an implementation detail. If it is true, we can start developing our application as Clean Monolith and when good moment comes, migrate it to microservices without much work and without touching other layers than interfaces/infrastructure. It is a similar approach to starting implementation of database driver in filesystem or memory, to defer the decision about choice of the database as long as we can. Of course, it may still require some optimizations while migrating to microservices (or in the opposite direction, because why not?) it will be pretty easy operation, compared to classical monolith refactoring (or even worse, reimplementing). I want not to be groundless. Here are some snippets from the source code of the application presented on the schema. Show me the code In this example we will follow simple shop flow which includes: placing the order, initializing payment and simulated asynchronous payment receiving. Placing the order should be made synchronous, initializing and receiving payments is async. Placing the order In interfaces it’s nothing special. Just parsing HTTP request and executing a command in the application layer. Note In the top of every snippet you can check where it is located in the repository. You can also read bounded context and layer from it. For example pkg/orders/interfaces/public/http/something.go is interfaces layer of orders bounded context. // pkg/orders/interfaces/public/http/orders.go func (o ordersResource) Post(w http.ResponseWriter, r *http.Request) { req := PostOrderRequest{} if err := render.Decode(r, &req); err != nil { _ = render.Render(w, r, common_http.ErrBadRequest(err)) return } cmd := application.PlaceOrderCommand{ OrderID: orders.ID(uuid.NewV1().String()), ProductID: req.ProductID, Address: application.PlaceOrderCommandAddress(req.Address), } if err := o.service.PlaceOrder(cmd); err != nil { _ = render.Render(w, r, common_http.ErrInternal(err)) return } w.WriteHeader(http.StatusOK) render.JSON(w, r, PostOrdersResponse{ OrderID: string(cmd.OrderID), }) } Note For people not familiar with Golang: in Go you don’t need to explicitly say that you are implementing an interface. You just need to implement interface’s methods. A good example is io.Reader interface. type Reader interface { Read(p []byte) (n int, err error) } for example: // no Foo implements io.Reader needed type Foo struct {} func (f Foo) Read(b []byte) (int, error) { // ... } func SomeFunc(r io.Reader) { buf := make([]byte, 10) // ... r.Read(buf) // ... } // ... SomeFunc(Foo{}) It’s more interesting in application service in pkg/orders/application/orders.go: First of all, interfaces of Products Service (to get product data from Shop bounded context) and Payments Service (to initialize payment in Payments bounded context): // pkg/orders/application/orders.go type productsService interface { ProductByID(id orders.ProductID) (orders.Product, error) } type paymentsService interface { InitializeOrderPayment(id orders.ID, price price.Price) error } And finally the Application service. This is exactly the same for monolith and microservices. We just inject different productsService and paymentsService implementations. We also use domain objects here and repository to persist Order in the database (in our case we use memory implementation, but it’s also a detail and can be changed to any storage). // pkg/orders/application/orders.go type OrdersService struct { productsService productsService paymentsService paymentsService ordersRepository orders.Repository } // ... func (s OrdersService) PlaceOrder(cmd PlaceOrderCommand) error { address, err := orders.NewAddress( cmd.Address.Name, cmd.Address.Street, cmd.Address.City, cmd.Address.PostCode, cmd.Address.Country, ) if err != nil { return errors.Wrap(err, "invalid address") } product, err := s.productsService.ProductByID(cmd.ProductID) if err != nil { return errors.Wrap(err, "cannot get product") } newOrder, err := orders.NewOrder(cmd.OrderID, product, address) if err != nil { return errors.Wrap(err, "cannot create order") } if err := s.ordersRepository.Save(newOrder); err != nil { return errors.Wrap(err, "cannot save order") } if err := s.paymentsService.InitializeOrderPayment(newOrder.ID(), newOrder.Product().Price()); err != nil { return errors.Wrap(err, "cannot initialize payment") } log.Printf("order %s placed", cmd.OrderID) return nil } productsService implementation Microservice In microservices version we use HTTP (REST) interface to get product info. I’ve separated REST API’s to private (internal) and public (accessed by frontend, for example). // pkg/orders/infrastructure/shop/http.go import ( // ... http_interface "github.com/ThreeDotsLabs/monolith-shop/pkg/shop/interfaces/private/http" // ... ) func (h HTTPClient) ProductByID(id orders.ProductID) (orders.Product, error) { resp, err := http.Get(fmt.Sprintf("%s/products/%s", h.address, id)) if err != nil { return orders.Product{}, errors.Wrap(err, "request to shop failed") } // ... productView := http_interface.ProductView{} if err := json.Unmarshal(b, &productView); err != nil { return orders.Product{}, errors.Wrapf(err, "cannot decode response: %s", b) } return OrderProductFromHTTP(productView) } The REST endpoint in Shop bounded context looks like this: // pkg/shop/interfaces/private/http/products.go type productsResource struct { repo products_domain.Repository } // ... func (p productsResource) Get(w http.ResponseWriter, r *http.Request) { product, err := p.repo.ByID(products_domain.ID(chi.URLParam(r, "id"))) if err != nil { _ = render.Render(w, r, common_http.ErrInternal(err)) return } render.Respond(w, r, ProductView{ string(product.ID()), product.Name(), product.Description(), priceViewFromPrice(product.Price()), }) } We also have a simple type which is used in HTTP response. In theory we can make Domain type serializable to JSON, but if we will do it every domain change will change our API contract, and every request for API contract change will change the domain. Doesn’t sound good and doesn’t have much in common with DDD and Clean Architecture. // pkg/shop/interfaces/private/http/products.go type ProductView struct { ID string `json:"id"` Name string `json:"name"` Description string `json:"description"` Price PriceView `json:"price"` } type PriceView struct { Cents uint `json:"cents"` Currency string `json:"currency"` } You can notice that ProductView is imported in pkg/orders/infrastructure/shop/http.go (example above), because, as I said before, imports from interfaces to infrastructure between bounded contexts are totally fine. Monolith In monolith version it’s pretty simple: in Orders bounded context we just call the function from Shop bounded context (intraprocess.ProductInterface:ProductByID) instead of calling REST API. // pkg/orders/infrastructure/shop/intraprocess.go import ( "github.com/ThreeDotsLabs/monolith-shop/pkg/orders/domain/orders" "github.com/ThreeDotsLabs/monolith-shop/pkg/shop/interfaces/private/intraprocess" ) type IntraprocessService struct { intraprocessInterface intraprocess.ProductInterface } func NewIntraprocessService(intraprocessInterface intraprocess.ProductInterface) IntraprocessService { return IntraprocessService{intraprocessInterface} } func (i IntraprocessService) ProductByID(id orders.ProductID) (orders.Product, error) { shopProduct, err := i.intraprocessInterface.ProductByID(string(id)) if err != nil { return orders.Product{}, err } return OrderProductFromIntraprocess(shopProduct) } And in Shop bounded context: // pkg/shop/interfaces/private/intraprocess/products.go type ProductInterface struct { repo products.Repository } // ... func (i ProductInterface) ProductByID(id string) (Product, error) { domainProduct, err := i.repo.ByID(products.ID(id)) if err != nil { return Product{}, errors.Wrap(err, "cannot get product") } return ProductFromDomainProduct(*domainProduct), nil } You can notice that in Orders bounded context we don’t import anything outside of Shops bounded context (as Clean Architecture assumes). So, we need some kind of transport type which can be imported in Shops BC. type Product struct { ID string Name string Description string Price price.Price } It can look redundant and duplicated, but in practice it helps with keeping constant contract between bounded contexts. For example we can totally replace application and domain layer and don’t touch this type. You need to keep in mind that cost of avoiding duplication increases with scale. Also, duplication of data is not the same as duplication of behaviour. Do you see some analogy to ProductView in microservices version? Note Looking back, “ipc” wasn’t the best name for this pattern. It’s too easy to confuse with Unix IPC (pipes, shared memory, message queues). Today we’d call this pattern “module contracts.” The intraprocess package would become something like shop/api/module/client, where the shop module defines its contract. If you want to practice this pattern hands-on, check out our Go Backend Masterclass. You build a real Go backend from scratch, including module contracts, HTTP/gRPC APIs, repositories, and testing. Initializing payment In the previous example we replaced HTTP Call with a Function call, which was synchronous. But how to deal with asynchronous operations? It depends. In Go it’s easy because of concurrency primitives. If it’s hard to achieve in your language you can just use Rabbit in the monolith. Like in the previous example, both versions look the same in application and domain layer. // pkg/orders/application/orders.go type paymentsService interface { InitializeOrderPayment(id orders.ID, price price.Price) error } func (s OrdersService) PlaceOrder(cmd PlaceOrderCommand) error { // .. if err := s.paymentsService.InitializeOrderPayment(newOrder.ID(), newOrder.Product().Price()); err != nil { return errors.Wrap(err, "cannot initialize payment") } // .. } Microservices In microservices we are using RabbitMQ to send message: // pkg/orders/infrastructure/payments/amqp.go // ... func (i AMQPService) InitializeOrderPayment(id orders.ID, price price.Price) error { order := payments_amqp_interface.OrderToProcessView{ ID: string(id), Price: payments_amqp_interface.PriceView{ Cents: price.Cents(), Currency: price.Currency(), }, } b, err := json.Marshal(order) if err != nil { return errors.Wrap(err, "cannot marshal order for amqp") } err = i.channel.Publish( "", i.queue.Name, false, false, amqp.Publishing{ ContentType: "application/json", Body: b, }) if err != nil { return errors.Wrap(err, "cannot send order to amqp") } log.Printf("sent order %s to amqp", id) return nil } and to receive messages // pkg/orders/interfaces/public/http/orders.go // ... type PaymentsInterface struct { conn *amqp.Connection queue amqp.Queue channel *amqp.Channel service application.PaymentsService } // ... func (o PaymentsInterface) Run(ctx context.Context) error { // ... for { select { case msg := <-msgs: err := o.processMsg(msg) if err != nil { log.Printf("cannot process msg: %s, err: %s", msg.Body, err) } case <-done: return nil } } } func (o PaymentsInterface) processMsg(msg amqp.Delivery) error { orderView := OrderToProcessView{} err := json.Unmarshal(msg.Body, &orderView) if err != nil { log.Printf("cannot decode msg: %s, error: %s", string(msg.Body), err) } orderPrice, err := price.NewPrice(orderView.Price.Cents, orderView.Price.Currency) if err != nil { log.Printf("cannot decode price for msg %s: %s", string(msg.Body), err) } return o.service.InitializeOrderPayment(orderView.ID, orderPrice) } Monolith In monolith version sending to channel is stupid easy // pkg/orders/infrastructure/payments/intraprocess.go type IntraprocessService struct { orders chan <- intraprocess.OrderToProcess } func NewIntraprocessService(ordersChannel chan <- intraprocess.OrderToProcess) IntraprocessService { return IntraprocessService{ordersChannel} } func (i IntraprocessService) InitializeOrderPayment(id orders.ID, price price.Price) error { i.orders <- intraprocess.OrderToProcess{string(id), price} return nil } and receiving (I only removed shutdown (close) support, to not complicate the code): // pkg/payments/interfaces/intraprocess/orders.go // ... type PaymentsInterface struct { orders <-chan OrderToProcess service application.PaymentsService orderProcessingWg *sync.WaitGroup runEnded chan struct{} } // .. func (o PaymentsInterface) Run() { // ... for order := range o.orders { go func(orderToPay OrderToProcess) { // ... if err := o.service.InitializeOrderPayment(orderToPay.ID, orderToPay.Price); err != nil { log.Print("Cannot initialize payment:", err) } }(order) } } And more… Marking order as paid works almost the same as placing the order (REST API/Function call). If you’re curious how it works, please check the full source code. Full source can be found here: https://github.com/ThreeDotsLabs/monolith-microservice-shop I’ve implemented some acceptance tests which will check that all flow works exactly the same for both monolith and microservices. Tests can be found in tests/acceptance_test.go. You can find more info on how to run the project and tests in README.md. There is still some code, which is not covered here. If you want to get a deeper understanding of this code, please follow me on the Twitter (@roblaszczak) or subscribe to our newsletter - you will be notified when the article is ready. You will learn some basic concepts of the Golang, if you don’t know it already. I will also make this code more production grade. We are also planning to write some DevOps articles (Packer, Terraform, Ansible). Summary What about another microservices advantage? It will be less flexible, but you can still deploy monolith independently, although you must use different flow. For example, you can use feature branches where the master branch will be production and commits for not released changes should be merged into this feature branches. These branches should be used in the staging environment. This is true that monolith is harder to scale because you must scale entire application, not a single module. But in many cases, it is good enough. In this case we cannot assign one team per repository, but fortunately there won’t be many conflicts if modules are separated well. The only conflicts will occur in layers responsible for cross modules communication, but the same conflicts will occur in REST/Queue API’s. In monolith though it will have compile check unlike in microservices, where you must validate contracts, what requires extra work. Also, with monolith you will receive compile-check of shared types. The fast rewrite of the module (microservice in microservices architecture) is a matter of good modules separation - if the module is properly separated and designed you can replace it without touching anything outside. Note And I will repeat it again: there is no silver bullet. It always depends. But probably in most cases Clean Monolith will be enough for the beginning. If it will be well designed, the decision to move to microservices architecture will be not a problem. In some cases (for example a small projects) it can be an overkill to write this all extra code for interfaces/infrastructure layer. But what is a small project? Well, it depends… Plan before you start implementing. There is no “lack of design”, it is only good design and bad design. Event storming is a good idea to kick-off the project. If you want to receive notification about the next article, you can subscribe to our newsletter or follow me on Twitter: @roblaszczak I’m aware that I’ve introduced a lot of techniques, of which a major part might be new for you. The good news is that to have good architecture you don’t need to learn all of these techniques at the same time. It’s also hard to properly understand these techniques simultaneously. I would recommend starting with Clean Architecture, then check some basics of CQRS and then you can drive into DDD. At the end of the article, I will provide some useful resources, which I had used when I was learning these techniques. If you have any questions, please write to me on Twitter. Thanks for the inspiration for this article: @unclebobmartin, @vaughnvernon, @martinfowler and @mariuszgil.

Privacy Policy Your privacy is important to us. It is Three Dots Labs R. Laszczak M. Smolka s.c.'s policy to respect your privacy and comply with any applicable law and regulation regarding any personal information we may collect about you, including across our website, https://threedots.tech/, and other sites we own and operate. Personal information is any information about you which can be used to identify you. This includes information about you as a person (such as name, address, and date of birth), your devices, payment details, and even information about how you use a website or online service. In the event our site contains links to third-party sites and services, please be aware that those sites and services have their own privacy policies. After following a link to any third-party content, you should read their posted privacy policy information about how they collect and use personal information. This Privacy Policy does not apply to any of your activities after you leave our site. This policy is effective as of 27 May 2022. Last updated: 27 May 2022 Information We Collect Information we collect falls into one of two categories: “voluntarily provided” information and “automatically collected” information. “Voluntarily provided” information refers to any information you knowingly and actively provide us when using or participating in any of our services and promotions. “Automatically collected” information refers to any information automatically sent by your devices in the course of accessing our products and services. Log Data When you visit our website, our servers may automatically log the standard data provided by your web browser. It may include your device’s Internet Protocol (IP) address, your browser type and version, the pages you visit, the time and date of your visit, the time spent on each page, and other details about your visit. Additionally, if you encounter certain errors while using the site, we may automatically collect data about the error and the circumstances surrounding its occurrence. This data may include technical details about your device, what you were trying to do when the error happened, and other technical information relating to the problem. You may or may not receive notice of such errors, even in the moment they occur, that they have occurred, or what the nature of the error is. Please be aware that while this information may not be personally identifying by itself, it may be possible to combine it with other data to personally identify individual persons. Personal Information We may ask for personal information — for example, when you register an account or when you contact us — which may include one or more of the following: Name Email Legitimate Reasons for Processing Your Personal Information We only collect and use your personal information when we have a legitimate reason for doing so. In which instance, we only collect personal information that is reasonably necessary to provide our services to you. Collection and Use of Information We may collect personal information from you when you do any of the following on our website: Register for an account Use a mobile device or web browser to access our content Contact us via email, social media, or on any similar technologies When you mention us on social media We may collect, hold, use, and disclose information for the following purposes, and personal information will not be further processed in a manner that is incompatible with these purposes: to provide you with our platform's core features and services to contact and communicate with you to enable you to access and use our website, associated applications, and associated social media platforms We may combine voluntarily provided and automatically collected personal information with general information or research data we receive from other trusted sources. For example, Our marketing and market research activities may uncover data and insights, which we may combine with information about how visitors use our site to improve our site and your experience on it. Security of Your Personal Information When we collect and process personal information, and while we retain this information, we will protect it within commercially acceptable means to prevent loss and theft, as well as unauthorized access, disclosure, copying, use, or modification. Although we will do our best to protect the personal information you provide to us, we advise that no method of electronic transmission or storage is 100% secure, and no one can guarantee absolute data security. You are responsible for selecting any password and its overall security strength, ensuring the security of your own information within the bounds of our services. For example, ensuring any passwords associated with accessing your personal information and accounts are secure and confidential. How Long We Keep Your Personal Information We keep your personal information only for as long as we need to. This time period may depend on what we are using your information for, in accordance with this privacy policy. For example, if you have provided us with personal information as part of creating an account with us, we may retain this information for the duration your account exists on our system. If your personal information is no longer required for this purpose, we will delete it or make it anonymous by removing all details that identify you. However, if necessary, we may retain your personal information for our compliance with a legal, accounting, or reporting obligation or for archiving purposes in the public interest, scientific, or historical research purposes or statistical purposes. Children’s Privacy We do not aim any of our products or services directly at children under the age of 13, and we do not knowingly collect personal information about children under 13. Disclosure of Personal Information to Third Parties We may disclose personal information to: a parent, subsidiary, or affiliate of our company third-party service providers for the purpose of enabling them to provide their services, including (without limitation) IT service providers, data storage, hosting and server providers, ad networks, error loggers, debt collectors, maintenance or problem-solving providers, marketing or advertising providers, professional advisors, and payment systems operators our employees, contractors, and/or related entities our existing or potential agents or business partners credit reporting agencies, courts, tribunals, and regulatory authorities, in the event you fail to pay for goods or services we have provided to you courts, tribunals, regulatory authorities, and law enforcement officers, as required by law, in connection with any actual or prospective legal proceedings, or in order to establish, exercise, or defend our legal rights third parties, including agents or sub-contractors, who assist us in providing information, products, services, or direct marketing to you third parties to collect and process data an entity that buys, or to which we transfer all or substantially all of our assets and business Third parties we currently use include: ConvertKit Google Adsense Facebook Ads Paddle International Transfers of Personal Information The personal information we collect is stored and/or processed in United States, Belgium, and Poland, or where we or our partners, affiliates, and third-party providers maintain facilities. The countries to which we store, process, or transfer your personal information may not have the same data protection laws as the country in which you initially provided the information. If we transfer your personal information to third parties in other countries: (i) we will perform those transfers in accordance with the requirements of applicable law; and (ii) we will protect the transferred personal information in accordance with this privacy policy. Your Rights and Controlling Your Personal Information Your choice: By providing personal information to us, you understand we will collect, hold, use, and disclose your personal information in accordance with this privacy policy. You do not have to provide personal information to us, however, if you do not, it may affect your use of our website or the products and/or services offered on or through it. Information from third parties: If we receive personal information about you from a third party, we will protect it as set out in this privacy policy. If you are a third party providing personal information about somebody else, you represent and warrant that you have such person’s consent to provide the personal information to us. Marketing permission: If you have previously agreed to us using your personal information for direct marketing purposes, you may change your mind at any time by contacting us using the details below. Access: You may request details of the personal information that we hold about you. Correction: If you believe that any information we hold about you is inaccurate, out of date, incomplete, irrelevant, or misleading, please contact us using the details provided in this privacy policy. We will take reasonable steps to correct any information found to be inaccurate, incomplete, misleading, or out of date. Non-discrimination: We will not discriminate against you for exercising any of your rights over your personal information. Unless your personal information is required to provide you with a particular service or offer (for example providing user support), we will not deny you goods or services and/or charge you different prices or rates for goods or services, including through granting discounts or other benefits, or imposing penalties, or provide you with a different level or quality of goods or services. Notification of data breaches: We will comply with laws applicable to us in respect of any data breach. Complaints: If you believe that we have breached a relevant data protection law and wish to make a complaint, please contact us using the details below and provide us with full details of the alleged breach. We will promptly investigate your complaint and respond to you, in writing, setting out the outcome of our investigation and the steps we will take to deal with your complaint. You also have the right to contact a regulatory body or data protection authority in relation to your complaint. Unsubscribe: To unsubscribe from our email database or opt-out of communications (including marketing communications), please contact us using the details provided in this privacy policy, or opt-out using the opt-out facilities provided in the communication. We may need to request specific information from you to help us confirm your identity. Use of Cookies We use “cookies” to collect information about you and your activity across our site. A cookie is a small piece of data that our website stores on your computer, and accesses each time you visit, so we can understand how you use our site. This helps us serve you content based on preferences you have specified. Please refer to our Cookie Policy for more information. Business Transfers If we or our assets are acquired, or in the unlikely event that we go out of business or enter bankruptcy, we would include data, including your personal information, among the assets transferred to any parties who acquire us. You acknowledge that such transfers may occur, and that any parties who acquire us may, to the extent permitted by applicable law, continue to use your personal information according to this policy, which they will be required to assume as it is the basis for any ownership or use rights we have over such information. Limits of Our Policy Our website may link to external sites that are not operated by us. Please be aware that we have no control over the content and policies of those sites, and cannot accept responsibility or liability for their respective privacy practices. Changes to This Policy At our discretion, we may change our privacy policy to reflect updates to our business processes, current acceptable practices, or legislative or regulatory changes. If we decide to change this privacy policy, we will post the changes here at the same link by which you are accessing this privacy policy. If the changes are significant, or if required by applicable law, we will contact you (based on your selected preferences for communications from us) and all our registered users with the new details and links to the updated or changed policy. If required by law, we will get your permission or give you the opportunity to opt in to or opt out of, as applicable, any new uses of your personal information. Additional Disclosures for Australian Privacy Act Compliance (AU) International Transfers of Personal Information Where the disclosure of your personal information is solely subject to Australian privacy laws, you acknowledge that some third parties may not be regulated by the Privacy Act and the Australian Privacy Principles in the Privacy Act. You acknowledge that if any such third party engages in any act or practice that contravenes the Australian Privacy Principles, it would not be accountable under the Privacy Act, and you will not be able to seek redress under the Privacy Act. Additional Disclosures for General Data Protection Regulation (GDPR) Compliance (EU) Data Controller / Data Processor The GDPR distinguishes between organisations that process personal information for their own purposes (known as “data controllers”) and organizations that process personal information on behalf of other organizations (known as “data processors”). We, Three Dots Labs R. Laszczak M. Smolka s.c., located at the address provided in our Contact Us section, are a Data Controller with respect to the personal information you provide to us. Legal Bases for Processing Your Personal Information We will only collect and use your personal information when we have a legal right to do so. In which case, we will collect and use your personal information lawfully, fairly, and in a transparent manner. If we seek your consent to process your personal information, and you are under 16 years of age, we will seek your parent or legal guardian’s consent to process your personal information for that specific purpose. Our lawful bases depend on the services you use and how you use them. This means we only collect and use your information on the following grounds: Consent From You Where you give us consent to collect and use your personal information for a specific purpose. You may withdraw your consent at any time using the facilities we provide; however this will not affect any use of your information that has already taken place. When you contact us, you may consent to your name and email address being used so we can respond to your enquiry. While you may request that we delete your contact details at any time, we cannot recall any email we have already sent. If you have any further enquiries about how to withdraw your consent, please feel free to enquire using the details provided in the Contact Us section of this privacy policy. Performance of a Contract or Transaction Where you have entered into a contract or transaction with us, or in order to take preparatory steps prior to our entering into a contract or transaction with you. For example, if you contact us with an enquiry, we may require personal information such as your name and contact details in order to respond. Our Legitimate Interests Where we assess it is necessary for our legitimate interests, such as for us to provide, operate, improve and communicate our services. We consider our legitimate interests to include research and development, understanding our audience, marketing and promoting our services, measures taken to operate our services efficiently, marketing analysis, and measures taken to protect our legal rights and interests. Compliance with Law In some cases, we may have a legal obligation to use or keep your personal information. Such cases may include (but are not limited to) court orders, criminal investigations, government requests, and regulatory obligations. If you have any further enquiries about how we retain personal information in order to comply with the law, please feel free to enquire using the details provided in the Contact Us section of this privacy policy. International Transfers Outside of the European Economic Area (EEA) We will ensure that any transfer of personal information from countries in the European Economic Area (EEA) to countries outside the EEA will be protected by appropriate safeguards, for example by using standard data protection clauses approved by the European Commission, or the use of binding corporate rules or other legally accepted means. Your Rights and Controlling Your Personal Information Restrict: You have the right to request that we restrict the processing of your personal information if (i) you are concerned about the accuracy of your personal information; (ii) you believe your personal information has been unlawfully processed; (iii) you need us to maintain the personal information solely for the purpose of a legal claim; or (iv) we are in the process of considering your objection in relation to processing on the basis of legitimate interests. Objecting to processing: You have the right to object to processing of your personal information that is based on our legitimate interests or public interest. If this is done, we must provide compelling legitimate grounds for the processing which overrides your interests, rights, and freedoms, in order to proceed with the processing of your personal information. Data portability: You may have the right to request a copy of the personal information we hold about you. Where possible, we will provide this information in CSV format or other easily readable machine format. You may also have the right to request that we transfer this personal information to a third party. Deletion: You may have a right to request that we delete the personal information we hold about you at any time, and we will take reasonable steps to delete your personal information from our current records. If you ask us to delete your personal information, we will let you know how the deletion affects your use of our website or products and services. There may be exceptions to this right for specific legal reasons which, if applicable, we will set out for you in response to your request. If you terminate or delete your account, we will delete your personal information within 28 days of the deletion of your account. Please be aware that search engines and similar third parties may still retain copies of your personal information that has been made public at least once, like certain profile information and public comments, even after you have deleted the information from our services or deactivated your account. Additional Disclosures for California Compliance (US) Under California Civil Code Section 1798.83, if you live in California and your business relationship with us is mainly for personal, family, or household purposes, you may ask us about the information we release to other organizations for their marketing purposes. To make such a request, please contact us using the details provided in this privacy policy with “Request for California privacy information” in the subject line. You may make this type of request once every calendar year. We will email you a list of categories of personal information we revealed to other organisations for their marketing purposes in the last calendar year, along with their names and addresses. Not all personal information shared in this way is covered by Section 1798.83 of the California Civil Code. Do Not Track Some browsers have a “Do Not Track” feature that lets you tell websites that you do not want to have your online activities tracked. At this time, we do not respond to browser “Do Not Track” signals. We adhere to the standards outlined in this privacy policy, ensuring we collect and process personal information lawfully, fairly, transparently, and with legitimate, legal reasons for doing so. Cookies and Pixels At all times, you may decline cookies from our site if your browser permits. Most browsers allow you to activate settings on your browser to refuse the setting of all or some cookies. Accordingly, your ability to limit cookies is based only on your browser’s capabilities. Please refer to the Cookies section of this privacy policy for more information. CCPA-permitted financial incentives In accordance with your right to non-discrimination, we may offer you certain financial incentives permitted by the CCPA that can result in different prices, rates, or quality levels for the goods or services we provide. Any CCPA-permitted financial incentive we offer will reasonably relate to the value of your personal information, and we will provide written terms that describe clearly the nature of such an offer. Participation in a financial incentive program requires your prior opt-in consent, which you may revoke at any time. California Notice of Collection In the past 12 months, we have collected the following categories of personal information enumerated in the California Consumer Privacy Act: Identifiers, such as name, email address, phone number account name, IP address, and an ID or number assigned to your account. Commercial information, such as products or services history and purchases. For more information on information we collect, including the sources we receive information from, review the “Information We Collect” section. We collect and use these categories of personal information for the business purposes described in the “Collection and Use of Information” section, including to provide and manage our Service. Right to Know and Delete If you are a California resident, you have rights to delete your personal information we collected and know certain information about our data practices in the preceding 12 months. In particular, you have the right to request the following from us: The categories of personal information we have collected about you; The categories of sources from which the personal information was collected; The categories of personal information about you we disclosed for a business purpose or sold; The categories of third parties to whom the personal information was disclosed for a business purpose or sold; The business or commercial purpose for collecting or selling the personal information; and The specific pieces of personal information we have collected about you. To exercise any of these rights, please contact us using the details provided in this privacy policy. Shine the Light If you are a California resident, in addition to the rights discussed above, you have the right to request information from us regarding the manner in which we share certain personal information as defined by California’s “Shine the Light” with third parties and affiliates for their own direct marketing purposes. To receive this information, send us a request using the contact details provided in this privacy policy. Requests must include “California Privacy Rights Request” in the first line of the description and include your name, street address, city, state, and ZIP code. Contact Us For any questions or concerns regarding your privacy, you may contact us using the following details: Robert Laszczak contact@threedotslabs.com Terms of Service These Terms of Service govern your use of the website located at https://threedots.tech/ and any related services provided by Three Dots Labs R. Laszczak M. Smolka s.c.. By accessing https://threedots.tech/, you agree to abide by these Terms of Service and to comply with all applicable laws and regulations. If you do not agree with these Terms of Service, you are prohibited from using or accessing this website or using any other services provided by Three Dots Labs R. Laszczak M. Smolka s.c.. We, Three Dots Labs R. Laszczak M. Smolka s.c., reserve the right to review and amend any of these Terms of Service at our sole discretion. Upon doing so, we will update this page. Any changes to these Terms of Service will take effect immediately from the date of publication. These Terms of Service were last updated on 27 May 2022. Limitations of Use By using this website, you warrant on behalf of yourself, your users, and other parties you represent that you will not: modify, copy, prepare derivative works of, decompile, or reverse engineer any materials and software contained on this website; remove any copyright or other proprietary notations from any materials and software on this website; transfer the materials to another person or “mirror” the materials on any other server; knowingly or negligently use this website or any of its associated services in a way that abuses or disrupts our networks or any other service Three Dots Labs R. Laszczak M. Smolka s.c. provides; use this website or its associated services to transmit or publish any harassing, indecent, obscene, fraudulent, or unlawful material; use this website or its associated services in violation of any applicable laws or regulations; use this website in conjunction with sending unauthorized advertising or spam; harvest, collect, or gather user data without the user’s consent; or use this website or its associated services in such a way that may infringe the privacy, intellectual property rights, or other rights of third parties. Intellectual Property The intellectual property in the materials contained in this website are owned by or licensed to Three Dots Labs R. Laszczak M. Smolka s.c. and are protected by applicable copyright and trademark law. We grant our users permission to download one copy of the materials for personal, non-commercial transitory use. This constitutes the grant of a license, not a transfer of title. This license shall automatically terminate if you violate any of these restrictions or the Terms of Service, and may be terminated by Three Dots Labs R. Laszczak M. Smolka s.c. at any time. Liability Our website and the materials on our website are provided on an 'as is' basis. To the extent permitted by law, Three Dots Labs R. Laszczak M. Smolka s.c. makes no warranties, expressed or implied, and hereby disclaims and negates all other warranties including, without limitation, implied warranties or conditions of merchantability, fitness for a particular purpose, or non-infringement of intellectual property, or other violation of rights. In no event shall Three Dots Labs R. Laszczak M. Smolka s.c. or its suppliers be liable for any consequential loss suffered or incurred by you or any third party arising from the use or inability to use this website or the materials on this website, even if Three Dots Labs R. Laszczak M. Smolka s.c. or an authorized representative has been notified, orally or in writing, of the possibility of such damage. In the context of this agreement, “consequential loss” includes any consequential loss, indirect loss, real or anticipated loss of profit, loss of benefit, loss of revenue, loss of business, loss of goodwill, loss of opportunity, loss of savings, loss of reputation, loss of use and/or loss or corruption of data, whether under statute, contract, equity, tort (including negligence), indemnity, or otherwise. Because some jurisdictions do not allow limitations on implied warranties, or limitations of liability for consequential or incidental damages, these limitations may not apply to you. Accuracy of Materials The materials appearing on our website are not comprehensive and are for general information purposes only. Three Dots Labs R. Laszczak M. Smolka s.c. does not warrant or make any representations concerning the accuracy, likely results, or reliability of the use of the materials on this website, or otherwise relating to such materials or on any resources linked to this website. Links Three Dots Labs R. Laszczak M. Smolka s.c. has not reviewed all of the sites linked to its website and is not responsible for the contents of any such linked site. The inclusion of any link does not imply endorsement, approval, or control by Three Dots Labs R. Laszczak M. Smolka s.c. of the site. Use of any such linked site is at your own risk and we strongly advise you make your own investigations with respect to the suitability of those sites. Right to Terminate We may suspend or terminate your right to use our website and terminate these Terms of Service immediately upon written notice to you for any breach of these Terms of Service. Severance Any term of these Terms of Service which is wholly or partially void or unenforceable is severed to the extent that it is void or unenforceable. The validity of the remainder of these Terms of Service is not affected. Governing Law These Terms of Service are governed by and construed in accordance with the laws of Poland. You irrevocably submit to the exclusive jurisdiction of the courts in that State or location. Cookie Policy We use cookies to help improve your experience of our website at https://threedots.tech/. This cookie policy is part of Three Dots Labs R. Laszczak M. Smolka s.c.'s privacy policy. It covers the use of cookies between your device and our site. We also provide basic information on third-party services we may use, who may also use cookies as part of their service. This policy does not cover their cookies. If you don’t wish to accept cookies from us, you should instruct your browser to refuse cookies from https://threedots.tech/. In such a case, we may be unable to provide you with some of your desired content and services. What is a cookie? A cookie is a small piece of data that a website stores on your device when you visit. It typically contains information about the website itself, a unique identifier that allows the site to recognize your web browser when you return, additional data that serves the cookie’s purpose, and the lifespan of the cookie itself. Cookies are used to enable certain features (e.g. logging in), track site usage (e.g. analytics), store your user settings (e.g. time zone, notification preferences), and to personalize your content (e.g. advertising, language). Cookies set by the website you are visiting are usually referred to as first-party cookies. They typically only track your activity on that particular site. Cookies set by other sites and companies (i.e. third parties) are called third-party cookies They can be used to track you on other websites that use the same third-party service. Types of cookies and how we use them Essential cookies Essential cookies are crucial to your experience of a website, enabling core features like user logins, account management, shopping carts, and payment processing. We use essential cookies to enable certain functions on our website. Performance cookies Performance cookies track how you use a website during your visit. Typically, this information is anonymous and aggregated, with information tracked across all site users. They help companies understand visitor usage patterns, identify and diagnose problems or errors their users may encounter, and make better strategic decisions in improving their audience’s overall website experience. These cookies may be set by the website you’re visiting (first-party) or by third-party services. They do not collect personal information about you. We do not use this type of cookie on our site. Functionality cookies Functionality cookies are used to collect information about your device and any settings you may configure on the website you’re visiting (like language and time zone settings). With this information, websites can provide you with customized, enhanced, or optimized content and services. These cookies may be set by the website you’re visiting (first-party) or by third-party services. We use functionality cookies for selected features on our site. Targeting/advertising cookies Targeting/advertising cookies help determine what promotional content is most relevant and appropriate to you and your interests. Websites may use them to deliver targeted advertising or limit the number of times you see an advertisement. This helps companies improve the effectiveness of their campaigns and the quality of content presented to you. These cookies may be set by the website you’re visiting (first-party) or by third-party services. Targeting/advertising cookies set by third-parties may be used to track you on other websites that use the same third-party service. We use targeting/advertising cookies on our site. Acceptable Use Policy This acceptable use policy covers the products, services, and technologies (collectively referred to as the “Products”) provided by Three Dots Labs R. Laszczak M. Smolka s.c. under any ongoing agreement. It’s designed to protect us, our customers, and the general Internet community from unethical, irresponsible, and illegal activity. Three Dots Labs R. Laszczak M. Smolka s.c. customers found engaging in activities prohibited by this acceptable use policy can be liable for service suspension and account termination. In extreme cases, we may be legally obliged to report such customers to the relevant authorities. This policy was last reviewed on 27 May 2022. Fair use We provide our facilities with the assumption your use will be “business as usual”, as per our offer schedule. If your use is considered to be excessive, then additional fees may be charged, or capacity may be restricted. We are opposed to all forms of abuse, discrimination, rights infringement, and/or any action that harms or disadvantages any group, individual, or resource. We expect our customers and, where applicable, their users (“end-users”) to likewise engage our Products with similar intent. Customer accountability We regard our customers as being responsible for their own actions as well as for the actions of anyone using our Products with the customer’s permission. This responsibility also applies to anyone using our Products on an unauthorized basis as a result of the customer’s failure to put in place reasonable security measures. By accepting Products from us, our customers agree to ensure adherence to this policy on behalf of anyone using the Products as their end users. Complaints regarding the actions of customers or their end-users will be forwarded to the nominated contact for the account in question. If a customer — or their end-user or anyone using our Products as a result of the customer — violates our acceptable use policy, we reserve the right to terminate any Products associated with the offending account or the account itself or take any remedial or preventative action we deem appropriate, without notice. To the extent permitted by law, no credit will be available for interruptions of service resulting from any violation of our acceptable use policy. Prohibited activity Copyright infringement and access to unauthorized material Our Products must not be used to transmit, distribute or store any material in violation of any applicable law. This includes but isn’t limited to: any material protected by copyright, trademark, trade secret, or other intellectual property right used without proper authorization, and any material that is obscene, defamatory, constitutes an illegal threat or violates export control laws. The customer is solely responsible for all material they input, upload, disseminate, transmit, create or publish through or on our Products, and for obtaining legal permission to use any works included in such material. SPAM and unauthorized message activity Our Products must not be used for the purpose of sending unsolicited bulk or commercial messages in violation of the laws and regulations applicable to your jurisdiction (“spam”). This includes but isn’t limited to sending spam, soliciting customers from spam sent from other service providers, and collecting replies to spam sent from other service providers. Our Products must not be used for the purpose of running unconfirmed mailing lists or telephone number lists (“messaging lists”). This includes but isn’t limited to subscribing email addresses or telephone numbers to any messaging list without the permission of the email address or telephone number owner, and storing any email addresses or telephone numbers subscribed in this way. All messaging lists run on or hosted by our Products must be “confirmed opt-in”. Verification of the address or telephone number owner’s express permission must be available for the lifespan of the messaging list. We prohibit the use of email lists, telephone number lists or databases purchased from third parties intended for spam or unconfirmed messaging list purposes on our Products. This spam and unauthorized message activity policy applies to messages sent using our Products, or to messages sent from any network by the customer or any person on the customer’s behalf, that directly or indirectly refer the recipient to a site hosted via our Products. Unethical, exploitative, and malicious activity Our Products must not be used for the purpose of advertising, transmitting, or otherwise making available any software, program, product, or service designed to violate this acceptable use policy, or the acceptable use policy of other service providers. This includes but isn’t limited to facilitating the means to send spam and the initiation of network sniffing, pinging, packet spoofing, flooding, mail-bombing, and denial-of-service attacks. Our Products must not be used to access any account or electronic resource where the group or individual attempting to gain access does not own or is not authorized to access the resource (e.g. “hacking”, “cracking”, “phreaking”, etc.). Our Products must not be used for the purpose of intentionally or recklessly introducing viruses or malicious code into our Products and systems. Our Products must not be used for purposely engaging in activities designed to harass another group or individual. Our definition of harassment includes but is not limited to denial-of-service attacks, hate-speech, advocacy of racial or ethnic intolerance, and any activity intended to threaten, abuse, infringe upon the rights of, or discriminate against any group or individual. Other activities considered unethical, exploitative, and malicious include: Obtaining (or attempting to obtain) services from us with the intent to avoid payment; Using our facilities to obtain (or attempt to obtain) services from another provider with the intent to avoid payment; The unauthorized access, alteration, or destruction (or any attempt thereof) of any information about our customers or end-users, by any means or device; Using our facilities to interfere with the use of our facilities and network by other customers or authorized individuals; Publishing or transmitting any content of links that incite violence, depict a violent act, depict child pornography, or threaten anyone’s health and safety; Any act or omission in violation of consumer protection laws and regulations; Any violation of a person’s privacy. Our Products may not be used by any person or entity, which is involved with or suspected of involvement in activities or causes relating to illegal gambling; terrorism; narcotics trafficking; arms trafficking or the proliferation, development, design, manufacture, production, stockpiling, or use of nuclear, chemical or biological weapons, weapons of mass destruction, or missiles; in each case including any affiliation with others whatsoever who support the above such activities or causes. Unauthorized use of Three Dots Labs R. Laszczak M. Smolka s.c. property We prohibit the impersonation of Three Dots Labs R. Laszczak M. Smolka s.c., the representation of a significant business relationship with Three Dots Labs R. Laszczak M. Smolka s.c., or ownership of any Three Dots Labs R. Laszczak M. Smolka s.c. property (including our Products and brand) for the purpose of fraudulently gaining service, custom, patronage, or user trust. About this policy This policy outlines a non-exclusive list of activities and intent we deem unacceptable and incompatible with our brand. We reserve the right to modify this policy at any time by publishing the revised version on our website. The revised version will be effective from the earlier of: the date the customer uses our Products after we publish the revised version on our website; or 30 days after we publish the revised version on our website.