An introduction to microservice architecture, trunk based development, domain driven design, test driven development and behaviour driven development. This presentation is aimed at new and junior developers as an introduction to these concepts and how they are inter-related.
It was presented at Code Mentoring (Melbourne) on Saturday 22nd June 2019.
Apologies for the audio quality in the video, hopefully the next one will be better.
# Video
# Slides
# Transcription
# Welcome
# Title Slide
So today we’re going to be talking about microservice architecture, trunk based development, domain driven design, test driven development, behaviour driven development.
I’ll give a quick intro to each of them and then go through and tie them together and how they can actually work in the real world.
# Thanks
Before we get started, I want to say thanks to General Assembly for giving us the venue and for giving us the venue every week for code mentoring. It’s great sponsorship from them.
Want to thank my employer MessageMedia, who actually gave me the time to write this whole presentation. It took quite a number of hours for that.
And also just want to thank all the organisers at code mentoring for enabling this to actually happen. Without code mentoring I probably never would have done it.
# About Me
So, first of all, a little bit about me.
I’m a senior PHP developer at MessageMedia. Basically, I spend my days writing APIs.
I’ve worked with Internet technology since about 1992, I think in '92 I wrote my first Web page for an Internet access provider in Adelaide. I still vaguely remember what it looked like and I wouldn’t be showing it to anyone these days.
In the mid-nineties, I started an Internet access provider, so we did dial up Internet access. Initially started with ten phone lines and by the time I finished that it was up to, I think, about one hundred phone lines. That was in the days where each phone line had an individual modem sitting on it, all plugged into a computer.
At one point during my career I worked on the tender document that won AusRegistry the rights for .AU. So they ran com.au, net.au, org.au, edu.au etcetera. So, that was an interesting point of my life and a lot of work in a very short space of time.
My previous role was lead infrastructure or devops engineer for a company. So I was maintaining all the AWS infrastructure, the servers, deployment pipeline, and that sort of thing.
And I’m also a mentor here.
I’m a big fan of agile processes. So, making sure that people get the most value possible out of their development cycles.
And also of using the Symfony framework, which is a PHP framework for development.
I’m currently in the process of trying to relearn Javascript because I want to do more stuff with serverless functionality in AWS Lambda. And after four or five years without actually doing much JavaScript, it looks very different to how I remember it.
If anyone wants, up there in the corner, I’ve got a QR code for my LinkedIn, so feel free to take a snapshot of that, add me if you would like.
# About this presentation
So a quick, high level overview of what we’re going through.
There’s microservice architecture that’s basically dividing a large product up into smaller chunks.
Trunk based development is a branching strategy used for Git to make sure that you’ve got all of your code getting committed correctly, especially when you’ve got multiple developers in a team.
Domain driven design and development is part of dividing all of the functionality up for microservices, and you divided up by the business function that it suits, and it also influences your terminology that’s used in that.
Then test driven development is predominantly about unit testing. So writing your code tests in advance, and then being able to utilise that to verify that you’ve got functionality that you want.
And behaviour driven development, which is writing tests that are written from the customer’s perspective to make sure that you’re actually fulfilling their needs.
Also, be briefly mentioning pub/sub, queues, CQRS, and touching briefly on chaos engineering. I won’t go into them in depth. If you want more information about that, we can catch up another time and talk about it. They’re probably a bit more advanced where I’ve aimed this talk at.
During the presentation I’ll also keep it open for anyone to ask questions. I’m hoping that I got it at the right sort of level for everyone. If not feel free to ask questions. I’m happy to give more information on that.
# Pre-requisites
So, basic prerequisites to actually get something useful out of this presentation is an understanding of the basics of Agile. Without that, a lot of the concepts aren’t really going to be that useful. If you’re very familiar with waterfall, for example, you’re going to be like “hang on, that just doesn’t apply to what I do”.
Also necessary to know that best practise doesn’t really match corporate reality. Yes, there are some companies out there that managed to do it, but you generally find that they’ve got high income and they’re not too worried about the cost of development. Once you get into companies that are starting to get concerned with, “how much is this item going to cost us?”, then you have to take a few shortcuts and you don’t always get your best practises there.
Another assumption is that you have the desire to learn and improve. If you don’t, then I don’t know why you’re at a meetup. So, that’s an easy one.
Need the ability to apply concepts to your existing skill set. I’ve tried to make this technology agnostic, obviously being a PHP developer, if there’s any technology information in there, I would be using PHP as an example. You need to be able to convert that to whatever the language of your choice is.
And then, need a basic knowledge of Git concepts. So how to do a pull request, how to create a commit, that sort of thing.
So nothing too complex there, just sort of the basics that you should have at a junior level anyway.
# Microservice Architecture
So we’ll kick off with microservice architecture.
# MSA Overview
So the first component of it for microservices is you need to break a product into small, self contained pieces, you need to remove the dependencies between services, and the ultimate effect of that is that you reduce what’s called the blast radius.
That means when something breaks, which is going to happen, and never be afraid of the fact that something will break, it happened to Google the other day, when they sent a tweet out saying how great their calendar is, and an hour later, their calendar service went down for three hours.
The idea is, though, that you keep the blast radius small. So when Google had an outage on their calendar service, it didn’t affect Gmail, it didn’t affect their searches. If they were operating in a monolith, which is the opposite of a microservice, then having an outage on one item would take the whole system down.
The benefits of microservice architecture is that it simplifies your features and your team expansion, so it means that when a new starter comes on board, they can learn a single feature very quickly, and be able to start coding on it. So it reduces the barriers to entry. If you’re doing open source projects, it also has the benefit there, that it means that someone can come along and fix just the bit that they need or add an improvement to just the bit they they need without understanding the bigger picture.
It reduces your technology lock-in. So you might have something that, we’ve actually got this case at work, when developing a product in PHP, we know it is not the right language to develop it in, but we need the product developed. So we’ll do it in PHP, that will see us through the first little bit and then we’ll go and redevelop it in C++ or Java, have it fully compiled. We’ll be able to just drop out one component, pop the new one in there and away it goes.
It increases the maintainability of items. If you’ve got a lot of small components, if something goes wrong, or you want to improve something you can affect just that one little area. You don’t have to concern yourself with how you go about, the potential impact on other items. So it makes life a lot easier there.
And, as I mentioned before, reduces the onboarding time, so when you first come into a project, you can get a quick overview of the bit that you need to work on and not worry about the bigger picture.
# Application
What I’ve got here is a bit of data for, an example of how microservices can be done.
The picture on the right there, may look a little complex, but if I go through and explain it, it’s going to be quite simple.
So the first thing we do is we divide everything into a logical components or services. In the case of what we’ve got on the diagram, we’ve decided we’ve got a web interface, we’ve got a mobile app, got an API client, and they need to communicate through a layer back to an ordering service, a payment service and a logging service.
So the components in this situation are the three bits that we’ve got at the bottom, where you’ve got your orders service, then your payment service, and your loggin service to keep a record of everything. There’s interfaces between each of the services that disconnect them from each other, but still enable them to communicate. And, that’s done through what’s called pub/sub or publisher and subscriber. For those who work with Amazon, an example of that would be Amazon’s SNS. Or queues, and again with Amazon, that would be SQS.
The idea with the interfaces, is that you’ve got the standard of communication between them, so both parties have agreed to it. It’s what’s called a contract. And any time something happens it emits an event to the appropriate notification queue or to the message queue, so that other services can take advantage of that change.
You avoid the, avoid interdependencies; so we can see, for example, on the diagram that I got, when orders received, it needs to notify several other services. If you’re doing that in a monolith, you’d basically take your order received notification, then you go and update the payment service with it, once that’s done, then you go and add it to the log, and then you’d be able to come back to the orders service to continue to the next point. By running it through the communication channels, when I receive an order, I send out a notification and if something else doesn’t action it, I don’t care, that’s not my problem. I leave it up to the other services to handle that.
There’s ways to make the queues hardened and keep backups of all the data there and everything, so if messages are missed you can replay them, but you don’t worry about that too much in the initial stages.
The other thing to remember, data is eventually consistent using this model. If you make a change on the orders service, it may take anywhere from a few milliseconds to several seconds, possibly even to hours before it actually makes it through to another service. So you have to allow for that in your design.
It’s fault tolerant, if I have a problem with my payment service, I can still accept orders, and I can still create logs for those orders. I just can’t accept payments. If I’ve got a problem with my logging service, that’s fine. I can still take orders and payments, I just can’t record any logs for it.
By having the queues there, it means that I’ve got a backlog of items, so when I when I get that service coming back online, I can then start recording all the data.
And the other thing to remember is to keep yout data storage separate. You end up with lots of data replicated throughout the whole system. When you’ve got the orders, they write the data they need to fulfil orders to the order service. Payments will need some information about that, it’ll probably need “what’s the total order cost?”, it’ll probably need “what’s the order ID?” so I can tell you when I’ve received a payment; but it doesn’t need to know what the products are, so it won’t record that data.
So yes, you end up with duplication, but you also end up with your fault tolerance because of that duplication.
# Trunk Based Development
So that’s a very quick overview of microservice architecture. We’ll move into trunk based development.
# TBD Overview
This is something that basically you start getting into when you’ve got a team of people, whether that be in open source, because you could have many people committing to it, or you’ve got a development team that are working together and you don’t want them to have any problems with getting data into production and code into production.
So trunk based development is an enabler of continuous integration and continuous delivery or deployment.
Now, before I go to far, continuous delivery means that the code is ready to be deployed. You might still have a deployment cycle because customers don’t want it to come out on a minutely basis or hourly basis. Maybe you’ve got a compiled software product that you’ve actually sold them. People have to install on the machine. Imagine if Microsoft Word updated every five minutes. You’d never been able to get any work done, so they have continuous delivery. They’re ready to, ready for a deployment at any time, and then they will produce a new version and release that maybe once a month.
Continuous deployment is taking that one step further and actually putting it into production every time there’s a change.
Part of trunk based development is there’s multiple commits to your master branch on Git every day and by every developer. There’s no; everything should be automated as possible and it basically, you end up putting everything into production as soon as that committee approved.
You branch by abstraction. Now that’s the; people are often used to with Git you create a branch and you might then create some sub-branches off that for a feature, once you’re happy that the whole feature is working, then merge that into master and release it. The idea with trunk based development is your first commit will always be that you put in a feature flag that turns this item off. Then you can start developing that item, and as soon as you’ve got any functionality ready, it gets merged into master, it gets deployed and your end users still can’t see it. But you can change the future flag, to “on” and your testers can see it, so everything can be tested incrementally as you go.
All of your commits, or at least the pull requests that you’ve got for it, must pass testing. So if there’s not enough tests there to know that it works, you’re going to have problems with this model. You’re going to end up with bugs in production very quickly.
Another part of it is a fix forward strategy. You’ve got a lot of, a lot of people, in Git, if they’ve got a problem, they will revert to commit. Generally, don’t do that with trunk based development because you’ll end up with a complete mess. If you find a bug, you can, disable the bit of functionality if it’s already in production, or you just create a fix and you get it merged and you continue working forward, you never do a roll back.
One of them big advantages of trunk based development is when you get massive team sizes, you end up with, I know companies for example, Netflix, which manages to do one release every seven seconds, I think, is the latest stats, do trunk based development because soon as you start putting another branch in there, for Gitflow or something like that, you’ll find that you get merge conflicts and these impediments to release and you can’t get your release cycle happening quickly.
And this solution also avoids merge conflicts, which, if you’ve ever worked in a team, is always annoying. You get a pull request created, someone reviews it, t’s all good, meanwhile, something else has just gone into production, and now you’ve got a merge conflict, so it’s, “okay”, and then I need to go and fix my merge conflict before we can continue.
# Styles
There’s two styles of trunk based development.
So you could do pure trump based development, and I wouldn’t recommend this one personally. It’s where all commits go directly into master. Now, if you’re doing that, you need to use pre-commit hooks on Git, so that the only time you can actually create a commit is when it’s working. It works best if your code pairing, so in that case, you actually have two developers. One person is describing the solution and another person is actually typing in the code. It’s an expensive way of doing things, but it’s really good for knowledge sharing, and you end up with some really good quality code. This’ll also require automatic roll backs. So there’s a whole whole subsection for the devops guys to do where they implement so if something fails on the test, it will automatically undo that commit on remove it from the history. It works really well in small, potentially even tiny teams that you’ve got maybe three or four developers, it’ll work well. Beyond that, you’re going to start running into some issues.
The second option, which is the one that I prefer, and one that I’ve used, is short lived features. So you can create a feature branch, that’s where all of your commits go to, but they’re going to last less than one business day. So that means based on most businesses, your developers are probably going to have an overhead of about three hours a day of non-coding time. So your feature branch will last for about five hours of development. So it needs to be kept small. Then, you create the pull request and you get it merged through the master.
You need a strong code review process for this. Before something is merged to master, then your developers, other developers need to review it. And the idea there is that they ask questions about anything from something as simple as “Did you make a typo in this comment?”, which I’ve had before, I actually had a pool request rejected because I’ve made a typo in a comment. Through to “I think your whole concept of how to resolve this issue is wrong”.
And this will scale to any team size.
# Requirements
The requirements for trunk based development is either excellent code review capabilities or some code pairing. Either one will work; effectively, code pairing you’re just reviewing it as it’s written. You need automated build and test services or your continuous integration pipeline. You need feature flags to turn things on and off very easily, and to prevent things from accidentally getting into production. When you’re defining your tasks to do, you need to keep them really small. Otherwise, your feature bunch is going to last too long. Your either need automated roll backs or a very good fix forward capability. There’s a few strategies around that that I can talk to people about afterwards if you’re looking at that sort of thing. And you also needed developer commitment. If you’ve got one developer, that’s like I prefer to create a full feature and then release it as one, then this won’t work. You need the developers to be on board with it.
# Short-lived Feature Branches Workflow
So what I’ve got here is a quick diagram about how to implement all of it.
So we can see that the very first step of it is we create a branch, there’s one commit on it, it’s for a feature flag. That gets reviewed; once it’s reviewed its merged into master and if you’ve got a full CI/CD process there, that will be released to production immediately. So already, before we’ve done any functionality, we have something that says “turn this functionality off”.
After that, we can then start doing our future development. We do it in very small steps. I’ve seen it taken as small as “we developed one function at a time”. So I had a pull request the other day where the entire pull request was defining a class. There was no data in the class, there was no functionality in the class, it was just “here’s a class, I’m going to need it”. There was a test suite created to make sure that when the class was instansiated it was a valid class.
Once that was through, the developer started creating other branches for individual bits of functionality. So in that case, there might be one, two, twenty commits; it doesn’t really matter. They will go through to become one pull request, which gets reviewed.
With the review process, once that’s complete, and everyone’s happy with it, it might be that you have two or three developers that have to sign off on it. Once it’s approved, you do what’s called a squash commit. That reduces all of the commits from the entire branch, so where this bunch has two commits, we reduce that down to a single commit to master that says, “This’s what I’ve added”. We don’t need to know all the development history of it, we only care about what’s actually been done in this. So you squash the commits there. The advantage of this is that master can be pushed into production at any time, either automatically or manual.
# Domain Driven Design and Development
So that’s the basics of trunk based development. So we’ll jump into domain driven design and development. This one’s a little bit more about how to break items up.
# DDD Overview
So the first bit is for your domain driven design, you need to bind everything to a business model. I’ll give an example in a minute. But the language has to be specific to that domain within the business. As an example, and I’ll get to this one in a bit more detail in a second, if you’re referring to customers or to the customer interface in a warehouse, they’re going to have one name for the customer. Management is going to have another main for it and accountants, accountants are going to call them by yet another name. So, you need to use your terminology that’s correct for the person who’s using the software. Big advantage of it is it’s going to reduce your overall support time, your developers no longer need to translate things in their head.
The down side of it is when you’re writing the services, you need to have the translation happening at that level. And the functionality of each of the components is driven by the business need. So if I’ve got a solution for customer data, accounts are going to need one set of data, my warehouse is going to need a different set of data, and my user authentication system is going to have a different set of data, so I only add in the functionality I need for that bit of the business.
# Implementation
Now, in order to implement it, you do have to understand the business domain before planning. So if you don’t understand that bit of the business you need to learn about it, and that can be, for a new project, that can be quite a significant amount of time. If you’re a developer with no accounting experience and you’re writing some software for the accounts people, you’re going to spend a few days learning their terminology, learning what they do, what do they want from the system?
All of you’re planning has to be a domain-centric, so that means don’t worry about anything that’s not directly related to it; that’s on external problem. And you need to use terminology dictated by the business function.
Here’s my example; accounts would have debtors and creditors, a warehouse would have recipients and senders, and management, would have customers and suppliers. If I get a call from the CEO they’re going to go “oh, I had an issue with this customer”, I’m not going to get them calling up going “I had an issue with this debtor”. But when accounts calls, they’re going to be telling me about the issue with a debtor, not their issue with a customer. So when I said it reduces the translation required for support, that’s what I mean by if my code calls it a debtor and my user calls it a debtor, I don’t have to think “oh, hang on, is a debtor a customer or is it a supplier? I can’t never remember”.
Now, once you’ve got that, you then need your inter-service communication to translate it. So when the changes made to a debtor, and it needs to update something for the warehouse, it needs to translate to a recipient. If that, then, has to go and put it in to a management or reporting system, it needs to translate it from a debtor or recipient into customer. Thankfully, that’s just a little bit of code.
# Test Driven Development
On to test driven development.
# TDD Overview
So the idea of test driven development is you write your unit tests before you develop anything. For those who have done development, very rarely does that actually happen and very rarely is it easy to do. If you can do it, you will get much better quality code, but it is going to be a big challenge and it requires a lot of analysis of the problem before you actually start writing code.
Test driven development does include behaviour driven development. That’s another component of this. However, I’ll cover behaviour driven development separately because there’s a distinct, there’s a difference between unit tests and behaviour tests.
Test driven development can often be interpreted as just unit tests. I’ve seen cases where people are like we’ve got one hundred percent unit test coverage, we’re doing test driven development, it’s wonderful. And you go well your unit tests don’t actually verify that anything works other than does exactly what it says. They were written after the fact, so we don’t know whether the code’s doing what we expect it to do or if you’re writing the tests to go “I wrote some code, let’s make sure it doesn’t change”.
If you’re writing tests, you’ll find errors early. So when you’re writing your, when you’re running your unit tests it will constantly be failing; failing to say “Hey, this isn’t doing what you want it to do”, that they’re doing what you told it to do it, but not what you want it to do.
Always test for failure. Assume that there’s going to be known bad cases. There’s nothing wrong with having an error appear, as long as, you know it’s the correct error.
It enables you to refactor your services with confidence. When something does need to be changed, maybe, your microservice has started becoming a monolith because it’s expanded. You can reuse those unit tests to go “when I rewrite this functionality, I know it’s still going to work”. Maybe you’re changing it into another language, so you modify your, translate your unit test for the new language first, and then you can test it against that and you know that you’re going to get the same functionality coming out of the new system.
And you mock external services. When you mock something, that basically means you’re creating a replica of it. It has no real functionality in it. By mocking your external services, tean, you’re not dependent on them as part of your tests. I’ve seen unit tests that do interface to, for example, a database, and because of a database outage all development had to stop because they couldn’t test.
# Unit Tests
The idea of unit tests, you write tests for each function before you write the function. Now, it might be that you start with a large function, which, you know, needs to do a number of things. So you write some tests for that. As you’re developing that you determine you need to move some code out into a separate function. So before you write the code for the other function, you write the unit test. That means that you know that the data in and data out is going to be a correct once you’ve finished that function.
It tests both good and bad paths. So you’ve given it some data, are you getting the correct after back? What about if you give it some bad data? So if you’re expecting to pass in a number, you pass in a string, what’s it going to do? Does it throw the correct exception.
Ensure that all of your tests are passing before you actually commit it all. If there’s a failure, you’re code’s not doing what it’s supposed to do.
And only test the code that you’re actually writing. So if there’s a dependency on an external service or another component then mock it, there’s no point testing it, it’s already tested elsewhere.
# Implementation and Procedure Enforcement
There’s a couple of interesting bits in relation to TDD though.
It can be very hard to adopt. If you’ve come from traditional background of coding, tests were never written in advance. It’s, you have to do all of your analysis before you write code. Whereas often from the more traditional development styles, you do your analysis as you’re writing the code.
You don’t always have to get one hundred percent code coverage. We have a policy on our CI/CD pipeline at work where we enforce one hundred percent code coverage. But that’s because we have tags in there that enables us to go “ignore this component, we don’t care about coverage here”. And where part of that policy is we have to justify why we’re ignoring that. I’ve seen something there of “in order to mock this service, we have to have mock five hundred dependencies”, because of the complexity of it, it’s not feasible to do it. So we’ve just gone “we trust everything else works and we know our functional tests will cover it”, where behaviour driven test will cover it. So we’ll just ignore it.
At a minimum, your tests must be written before committing. So this is where I was saying before that you have to understand, have to have the understanding that best practise and reality in the business world doesn’t align. Often you’ll start writing some tests. You get the basics done, as you develop you think of an edge case. So you had a bit more to the test and then you’ll develop that. Then once it’s all done, they’re going “Ahh, I’ve got this little bit of the end that I haven’t worked out how to handle or I haven’t got a test for it”. So then you add a test for that. It doesn’t have to be one hundred percent complete before you start development.
You need clear definitions for your functionality. What does this function do? What’s the input? What’s the output? What’s invalid input? In the event of invalid input what am I going to do? You need to have all that defined so that you can write tests for it.
Automating the running of the tests helps to enforce that it actually happens, so we have a process before any of our tests can get code reviewed, the tests have to pass.
I’ve also, on a personal project, I’ve implemented my test suite before I can create a commit, so it’s not possible for me to even commit on my local machine without the tests working.
It’s never too late to start getting unit tests in place. I’ve been, worked at a couple of places where they’ve got a monolith, there’s no testing on it, and they’re like “ahh, we can’t, there’s no point doing it, because then we don’t have, we’ve only got a very small percentage covered”. As long as you constantly increase your monitoring of test coverage. Then that’s fine. If you’ve got a unit test running on one function, you know that function is going to work from now, hopefully for eternity. So you might say, that might be point one of a percent of your code, or even less. So you say “cool, our minimum test coverage at this point is point one of a percent, and if I don’t have that level coverage, I will fail on my testing”. Once you’ve got that bit working, then you write a new function, you write some tests for it, You’ve now, got point one five of a percent so you can increase that level again. Keep working up until you get up to the seventy, eighty, ninety percent mark and you’ll eventually have a system that’s going to be a well tested.
The other thing to remember is that unit tests don’t replace functional tests or manual tests. You still need someone to sit down and play around with your product and make sure it actually does what it’s meant to do. If you haven’t understood the user requirements, unit tests aren’t going to help you. They’re only going to help you with the code level requirements.
# Behaviour Driven Development
So that leads us into the behaviour driven development side of things.
# BDD Overview
So the first part of behaviour driven development, you have very clear acceptance criteria. And that’s done in what’s called a user story. There’s some really good documentation out there about how to write user stories, a lot of it relates to your automated test tooling and you then also have to write, often end up writing code to facilitate the running of those tests, and there’s a predefined language for it.
It’s something that until recently I’ve actually struggled to do, behaviour driven development. But once I sat down and gave myself a bit of time, it turned out that it’s a lot easier than I thought, and writing the functionality to perform the test is incredibly simple.
Your acceptance criteria can then very quickly be converted to a functional test. I had one the other day where the acceptance criteria, I think, was four or five lines. I pasted it into my functional tests, I ran the test, and although the test failed because the code obviously wasn’t working (it was a bug report) all the components of the test were able to run, so our product had actually educated himself enough on the test to write it in the correct language, and I didn’t have to write any code in order to get functional test implemented.
The functional tests should also implement, should also test for real world use cases. If no one, if support are the only people that can, that ever change a password for a customer, then there’s no point testing “can the user or the customer change a password?” You only need to test it from a support perspective. Vice-versa though, if support is not allowed to change, through policy, is not allowed to change the password for a customer, then you might have a test that makes sure that they can’t, but there’s no point testing the password change functionality from support’s perspective.
Any time you get a support issue come into your team or reported to you, the first thing you should do is write failing tests. Make sure that it actually fits the customer’s problem. So, once you’ve written the test, you run it, you can verify that it’s failed, and therefore you’ve replicated the customer’s problem.
The test will often be written by the domain experts, so the domain expert is someone within the business unit that’s using your product. They should be able to write the test and say “this is what I expect to happen given this set of circumstances”. You might have to do a little bit of neatening up just to make it line up with the test suite properly, but over time they will learn how to write the tests and ultimately, you don’t end up writing them, someone else does.
And again it doesn’t replace unit tests and it doesn’t replace your manual testing.
# Functional Tests and Acceptance Criteria
So, sample acceptance criteria; given “X”, when “Y”, then “Z”. Keep it nice and simple.
Given I have a user account; when I enter my username and password; then I will be logged in. That’s a perfect functional test for my login process. It told me exactly what I need to do. There will have to be code behind it; one to create a user account in the database, make sure it’s got a password associated with it, then your “when” statement will need some functionality there so that it actually uses a username and password to login, your “then” statement needs to verify something to say “I’m actually logged in”.
If I was creating a new system, I could write that test, run it, and I know it would fail because I don’t have a user management system.
{ audience question - inaudible }
Yeah, you could put that in there. So by, what I’ve done here is by using the term “my” that implies it’s correct. If I was writing a similar test to make sure that the login was rejected. Then I would put “when I enter an invalid username or password” because one of them might be correct. I might even make it, add more clarity to it and use “when I enter my username and I enter an invalid password”. So you can divide it up a bit more. I kind of reduced down to having the two items on one line so it fits on the screen.
Your acceptance criteria will define those functional tests, and they’re primarily written by your domain experts, your product owners, and your quality assurance guys.
# Implemention and Integration
So again, your tests should all be automated. If you’re relying on someone to manually remember to run them, you may as well not have them.
The functional tests and units tests can be run in parallel. So if you’re using a CI/CD pipeline, you’d run two stages, one for your unit tests, one for your functional tests, at the same time. Otherwise you’re just adding a delay in there that’s not needed. I doesn’t matter if either one fails, you’re going to block the release, but they both need to pass in order to get it out.
The functional test should only ever mock third party services it should use all of the data that’s internal to your system. But a good example we had the other day, we’ve got a GeoIP service that is external to our system. Now, we needed to, the reason we needed to mock that is, what if the service goes down. So we need to have functional tests in there that say “When the GeoIP service is unavailable, this will happen”. We can then, through the mock, turn off the GeoIP service. But also, when we’re testing it to make sure it’s working, what happens if that external provider has a problem and they’re currently offline? We can’t afford to block our release process. So again, by mocking it we can go and we get a correct response from it, no matter what, even if they’re down.
The behaviour driven development test, your feature, functional tests don’t rely on unit tests. So, they’re completely separate items, they shouldn’t overlap at all.
And just reiterate, it doesn’t replace your unit tests. You can use that as an additional component above that that.
# Tying it all Together
Now that’s the sort of overview of everything.
Hopefully, now, I can tie it all together for you, so that it makes a bit of sense in the development cycle.
# Service Definition & Planning
So generally, the first part of generating your, a new bit of software is its service definition and planning what you’re going to do. The first bit is using BDD to divide your product up by the business domain. Is it for accounts? Is it for for the warehouse? Is it a reporting system for management? We can then use those divisions and divide them into services. So each of the domains can have multiple microservices within it.
So, you can see we’ve already used two of the ideals there.
We need to ensure the design doesn’t rely on other services. So, you’ve got your microservice architecture that helped to define the communication channels between the different components.
Always use terminology that’s local to the domain. So what, what does the end user call this? That’s what I should be calling it, because when they say “I got a problem with this”, I know what they mean.
And write your high level functional tests for the behaviour driven development. The very simple bits. Make sure you can login, make sure that you’re, one of your tests might be “when I click the generate report button”, or when I go, when I want to generate a report, I get a report emailed to me. You can always write the more specific ones as you move forward.
# CQRS
Using CQRS, I mentioned at the start that I would briefly use this. It’s command query responsibility segregation.
It’s generally only used when you’ve got massive amounts of data, and you need to optimise for reading on one database, and for writing on another database. A good example of it might be an audit trail where, looking at the situation we have at work, we get tens of thousands of requests a second, on our web servers. In order to write that, we need to have a database that optimised for write capabilities and can store large amounts of data. To read from it, most people are only going to search last couple of days, possibly a month, so we’ll keep one month’s worth of data on a second database that’s op[timised for read, it doesn’t matter if it takes a little bit longer to write.
Microservices and domain drive design work really well with CQRS. It works best in a complex business model. The read and write services are separated. They may share a data store, but generally won’t.
And when to use CRUD, which is the create, read, update, delete model, or CQRS is out of scope for this. If anyone wants to talk about it, I’m happy to, but I’d suggest that we do that another day.
# Issue Definition and Planning
Now, when you’re defining your issues, the first thing you should be doing; so this is for actually developing the items, you need to define your user stories. What do my users want to do? That’s the most important bit, if you’re not focusing on what you users want to do, then why are you writing a bit of software?
You break it down into the individual stories. So, you might have a larger picture of your use, your top level user story might be “as a user I want to be able to view the activities my co-workers have done on the service”. So you’ve now got a basic definition for an activity log or an audit trail. When you break that down into the individual stories, you start working out “hang on, they want to search, they want to filter it, they want to sort it”. So you start getting the individual stories coming through that you can then develop.
Those stories are then converted to the acceptance criteria. So your behavioural tests. The acceptance criteria can then become a functional test that you’re using.
You also then need to loop back to the people, to the product owner, to your business analyst, your end-users perhaps, and verify that the tests you’ve written are actually going to do what they want it to do. Without that, then again, you’re, as a developer you’re writing, or product owner depending on who’s writing the tests, then you’re not getting the validation that you’ve actually understood the problem.
Each of the tasks that you create should be small, less than a day’s worth of work for developer. They need to be testable. They to be user focussed. And they need to be designed, assuming that everything outside of them is going to fail. So if you have a dependency on a third party service, assume that service is going to fail regularly. It doesn’t matter how reliable it is; you’ve got an interface to Google’s search functionality, chances are it’s not going to fail, but what happens when it does? Not if, but when!
# Coding
When you’re coding, so as a developer for day to day job, use the domain based terminology. It’s going to make life easier long term. Don’t call something a customer when when you’re doing an accounting function.
Don’t rely on external systems. If you need some data, make sure you have a way to get that for yourself. Don’t put API calls in middle of your code. Set up another external system using queues or notifications or pub/sub so that the other system can tell you that the data has changed and you only rely on your local data store.
Make sure your code is human readable. Don’t use fancy terminology. Add comments where it’s needed. Make sure that
people, a junior developer can walk in, look at the code, and got “I know what that does, I can see the bug in it, I can
fix it”. Encourage everyone to use it. I’ve seen a lot of cases, for example, when people do a for-loop and they use
$i. A good example of that is that they’re meaning an index. Good, call it an index, because that way we know what it
does. We don’t have to learn programming terminology. If I have to explain it to a customer, it is much easier to say
“and then I used this index to go through it”, rather than $i.
Make sure your code’s testable and make sure that you test pass. All good having the test there, but if you’re not enforcing them, then they’ve got very little value. And again if you’re not testing the code, and you don’t know whether it’s going to work or not.
When you’re developing don’t don’t accept any scope creep. That’s always a big one, and that’s happened in almost every single issue I’ve worked on, where you start working on something, and you realise “I’ve got this other issue as well that’s closely related, but in another function. So I’ll just fix that part of this”. It doesn’t make sense to do that, because now you’re fixing two or three things, and before you know it, fixing ten things, you’ve got a large pull request going through. So create a new task; if that new task is blocking your current one, put the current one aside, and go and fix the external problem; then come back to the one you’re working on once it’s gone through the processes of getting released.
You have one task per issue. So if an issue is defined by a customer, it should be self contained enough to be a single task. If you think you need two or three tasks for it’s more than one issue, so make sure you’re clear on that, and before you even start coding, understand what the issue is. Am I creating a new function? Cool, what does it need to do? Or, is the customer experience in a bug? What’s the actual bug? Can I replicate it? Understand the impact off it. If you don’t know why you’re doing the code, then you’re probably not doing it for a good reason.
# Automated Testing & Deployments
Automated testing, ideally test before you commit, so use your commit hooks, pre-commit hooks to run your unit tests and then commit. It doesn’t always work, and I have seen a lot of cases where you have a soft fail, so there’s the ability to override the failure and still commit, because there’s also the best practice at the end of the day, before you go home, commit to Git, push it up to the server so if you lose your laptop it doesn’t matter.
At the very minimum, you must test your pull request before they merged. Make sure that you get the coverage there. Otherwise, that’s where you do get bugs creeping into production.
Automated both your unit and functional tests so that you don’t have to remember to run them. It’s just done for you. It also stopped you from going “I’ll run it and it didn’t pass, but I’ll just let it through anyway because that’s kind of what I wanted”.
Another good one is implementing production testing. A lot of companies struggle to do this because their systems aren’t written too consider it. Have automated systems set up that will test accounts on production. So, as an example, if you are a bank, you’d have dedicated accounts set up in the bank, where you can transfer money between them, you can deposit money into them, you can withdraw money; but you don’t want that to affect your end-users. So everything that happens, you also need to be able to undo it. So as part of your production test, you might deposit 10 dollars into one account, and then try to withdrawn eleven dollars; make sure that that fails. If it doesn’t, you need, it then needs to automatically reset itself, so that next time the test runs, you can see what happens, it can see what happens there.
Also worth consider chaos engineering, that’s where, I’ll touch on it brieflu, I won’t go too far into it. That’s where you just randomly turn things off, randomly break things. Most of the big providers out there in terms of Google, Facebook, Netflix, Spotify, have systems that will randomly turn things off. It could be something as simple as, we turn one machine off, or it could be we randomly break one function or, in the case of one, that I think it was Netflix did the other day, they turned off half of their servers around the world, and made sure that had no impact on customers, so their system was resilient enough that it recognised things were starting to go wrong and, the bits that were working, scaled up to take care of it. They managed to, I don’t think they had a single bug report or a single complaint about it when they did it. I would say there was probably a lot of very stressed developers thought.
When your code quality is high enough, implement continuous deployment. So, unless you’ve got a product that actually needs a full install routine, if you’re doing software as a service, just deploy as often as you can. As I mentioned before, places like Nexflix have obe release every seven seconds on average. Now that’s a hell of a lot of code going out there, and as an end-user, I never notice, it just magically updates. I might load a page, and I see that it’s got the design I’m used to, hit the refresh button and hey I’ve got a new, a whole new page now, or it’s got a new function on it.
{ audience question - inaudible }
There’s all sorts of little things. All their releases are very tiny. If you went and looked at Netflix from, say, a year ago. So you’ve got a screenshot from a year ago and compared it to a screenshot today, you’d suddenly go “there’s a lot of differences there”, and it can be little things like "ahh, they’ve added, when you move over the log in button with your mouse it highlights it in, it highlights it slightly differently, or it’s a slightly different curve on the button. Or it could be something much more complex, where the whole new set, whole new categorisation method. It doesn’t have to be, doesn’t have to be a big change. They’re usually best when they’re small.
{ audience question - inaudible }
Yeah, I’ve done a little bit on that, just sort of giving a very high level overview of each of them and then tying it all in together. If you want I’m, happy to go through it again afterwards. I’m also recording all of it, so at some point in the next, however long it takes me to work out how to go from the camera in to YouTube, I’ll get it up online as well.
# Q & A
So, that’s basically tied it all together.
Hopefully anyone got any more questions about it.
{ audience question - inaudible }
So what were do, we’ve got exactly that situation, we use PHP backend that’s all API, and we’ve got React on the frontend. So we’ve got…
{ audience question - inaudible }
So we have unit tests on react, and we have unit tests on a PHP side of things. We then have functional tests on React and functional tests on PHP. Use a thing called PACT to contract test to make sure that communications are agreed on both sides. And then we also do another step, which is integration testing. So, we have individually, on my machine I run the unit tests before I can commit, once I’ve got the commit done and I push it up on to our Git repo, that runs the unit test and the behavioural tests or functional tests. Once that’s passed, it then runs the PACT tests, to make sure that their contracts are still being honoured, The same thing happens on the React side of things, and then after, or as part of our release process the QA box runs an integration test. So you can end up with very complex deployment pipelines. But, there’s a lot of safety nets in there. And if something goes wrong, we get a report of it, pops up in Slack and usually it’s something very simple, like you’ve broken a contract or when you’re running the integration test, something has gone wrong there and it turns out that it’s a transient error, like you, the QAs’ database server scaled down without you realising. So you end-up re-running the tests.
{ audience question - inaudible }
For me to run full, because I’m using microservices for what we’re doing; full functional and behavioural tests, as well as linting of about seven or eight different file types, checking for copy and paste detection is under two minutes, through the automation pipeline. On the, on my local machine, I’ve generally only run the unit tests, depending on which microservice I’m working on, that could be anywhere up to a thirty seconds, forty seconds. So by the time I’ve worked out, I’ve hit the commit button I write in the commit message. I hit “commit” on it. I’m like “Okay, now what do I have to work on next, ahh, that’s finished”.
{ audience question - inaudible }
Yes, but that’s very controlled, and what happens if your infrastructure has an issue on production. So, it doesn’t matter how well, you’ve got everything set up on all of your pipeline getting through, there’s still a chance that production will be slightly different. Now in our case we have some cost concerns because the number of servers that we need to run to do all of our different components and tests is quite high. So we generally run very small servers or very small instances for our test, our QA, etcetera. One, and once we get into production those servers are generally, I think, ten to twelve times the size of whatever, any of our tests ones are. There’s also significantly more of them.
I had a situation, not at the current place, but a previous place where everything was all fully tested, it all worked perfectly, the load balancer was fine, the server was fine, but because the staging instances, there was only one server running in it, all the tests passed. Once it went into production that, there was multiple instances, and they had forgotten to share one bit of data between the servers on the back end. So the person would login, and when they refreshed the page it’d go “you’re not logged in”, but sometimes it’d be like “yep, cool, that worked” because they’re hitting the same server again. So, we very quickly identified that and I think it was, the bug was in production for less than twenty minutes. You can get, once you’ve got your, once you get all your processes streamlined, then if you find a simple bug, it might take you five minutes to fix it, then another two minutes for it to go through review, then you’ve still got thirteen minutes for it to actually complete the process through to production, especially if it’s been automated.
{ audience question - inaudible }
It depends a lot on the language and the framework. You should be mocking service dependencies, and ideally, all of your services should be injected. As opposed to having hard dependencies on specific external service. Depending on what your mocking, if I was mocking a web service, it might be something as simple as I can override my DNS to point to some other local service that I’m running. If I’m mocking a dependency on an external, external class, ideally, that would be injected, so there’d some level of configuration using autowiring or something to automatically put the correct class in there.
As I said before, I’ll use some PHP for examples. In PHP if you’ve got a dependency on external class, there’s two ways
you can do it. You can either go new and then the class name, and that will create a whole new version of it, or
you can put, instantiate the class in advance and then inject it in through the constructor of the current class. And
that’s the preferred way to do it. That way you only ever going “hey, go and get me this class that already exists,
that’s already ready for me to use”. So you can then mock that class and inject that instead of injecting the other, the
real class. You also need to think though, with that is it something where you’re so reliant on the other class that you
do need to test the integration of it. If your relying on a class that performs a certain function and without that your
function is useless, you’re probably best to just run the integration tests on it. It’s just where possible, you go “no,
I know that’s already tested, so I’m not going to test it”.
{ audience question - inaudible }
We have a, the current process that we’ve got depends a little bit on which component we’re working on. We’ve got some components which are still part of a monolith. If they get, there’s manual sanity tests on that before every single release. We’ve got other components which are much more microservice based they’re sanity tested in production. We assume that they’re working, all the tests have passed, they’ve good test coverage, they’ve got well written functional tests so we can trust that once it’s in production it’s going to be okay. We’re also, for the ones where we do that. Anyone who clicks the support link on the any of the pages that use that service, it escalates one priority level because our primary method of finding out there’s a bug is from a customer.
But, again, if you’ve got your automated production, then you don’t need the manual side of it. Part of the automated production testing, especially if you’ve got a front end in place, can be something as simple as automated screenshots. So when when you when you first release an item, it takes a screenshot of it. And if that screenshot changes in any future test, it will highlight where the change is and ask “did you intend to do this?” If you say yes it updates, if you say no then it goes “cool, that’s it, the test has failed”. So you then know that the functionality, and it could be that the backend service has failed and it’s now, it’s popped up a little toast message, saying “I’ve received a server error”, well that’s not intended, go and fix it on the backend, so you can actually monitor it in that way.
{ audience question - inaudible }
No, ultimately you end up with people that you’ve currently got manually testing are the ones that end up writing the functional tests for you because they understand the product, they know how people use it.
The other downside of manual testing, you still do a little bit of it, but your test, the manual testers become familiar with the product, they know how to use it, so they use it in a specific pattern, that fits best for the product; they don’t use it how a customer is going to use it. Who’s like I do this function with it, and that’s it.
So, we released a bit of functionality the other day, It was in the monolith, that went through the whole manual test process; everyone signed off on it. The product owner signed off on it, QA signed off on it, our team lead even looked at it and said, “Hey, that’s great”. Got in the hand of customers, I think we had one hundred fifty to two hundred support calls come in within the first hour of it going out there, because it wasn’t how a customer used it, it made no sense to them. We rolled it back, redeveloped it with a completely different method. So, your bigger issues end up becoming the customer’s experience, the customer’s interaction with it. Not “does my functionality work?”
{ audience question - inaudible }
We, surprisingly developers and users to, we have some test cases where they start off with as a developer. So our CI/CD pipeline, if we want to make a change to that it’s “as a developer, when this happens, I expect that”. And that’s how we would define something. Because our CI/CD pipeline is incorporated within the microservice, because we do everything, so you’ve heard of infrastructure, software based infrastructure, so all definition for every bit of code that we need is self contained. If I need to add a new queue, then I’ll define that in software and during the deployment process it will create that queue for me. So we actually have the need to say “as a developer I need this”. If we’re doing a change to the deployment pipeline, which happens surprisingly often, then we’ll put in there that “as a developer, I expect this to happen on the deployment pipeline”.
So, we write tests quite regularly. We also have to write the code to support the way the tests are written and in a lot of situations, depends on the seniority of the person, our senior devs and our team leads will write unit {meant “functional”} test themselves because they’ve got an understanding for product, they’ve got a good understanding of the customer, they’re also the sort of people we send out on-site to customers, so they’ve actually seen customers using it, they’ve sat down and talked to customers, and therefore we’ve got confidence in them effectively operating as a product owner type role to write the tests. So they can, we can throw an item at them going we need this bit of functionality, and they write the test, bring it back to someone else who then just goes “that test sounds good, now continue”. So there’s a validation step in there. But no matter who writes it, you need that validation step in there.
{ audience question - inaudible }
In terms of microservice stuff, with our QA his primary thing is to help us with writing some of the tests, to think of edge cases that developers, we haven’t thought of. What happens if, we might have a prompts there that says, “Enter a phone number”, what happens if a person puts in a Vietnamese phone number? I don’t know. What if their character set is different? What if they put a letter “o” instead of the number “0”? What if, what if they enter an invalid number? So as a developer going “were asking for a phone number, so someone is going to give me their phone number”, when you actually get into the real world, the first thing you do is install something like libphonenumber, which will validate the phone number. But then you also have to go, and we had this one come up the other day, what happens when Optus decides to get a new phone number range and linphonenumber doesn’t support it. Suddenly we’ve got this small little group of people that cannot use our service. In our case, we send SMS messages, o having someone that we can’t send an SMS message to is a problem. So the tester then then wrote some stuff that flags the message as high or flags the contact as high risk. So instead of saying we need to validate the phone number, we need to validate the phone = number and even if it’s invalid, but it’s got the right format still allow it to happen. Then through external systems we monitor, ongoing what happens with that phone number is it all getting rejected? And that’s where you can use stuff like machine learning to take it another step further and go “hey, this number is always rejected, start auto-rejecting it”, or “it never rejected, there’s a new set of phone numbers, we’ve got to add that in”.
So your QA does become much more about quality assurance, not about day-to-day testing. Part of that, and you’ll often hear the term “QA automation engineer”, they don’t need to know how to code, they need to know how to identify and write tests and the processes. They also, they’re more heavily involved in writing the tests on the behavioural side of things. And, generally we’ve found that QA starts becoming more of a support interface. We have a dedicated support team at work, when they get a propblem, they direct it to the right team and our QA person is the first person that gets it goes “I’m going to write a functional test for that that fails”. And if he can’t write that functional test, then we start going “h"ang on a minute, is it really a bug in our system, or is the customer using it differently?”, and we’ll keep talking about that until we get that failing test. So it becomes much more of a support type role.
# Thank You
Cool, well thank you all, I hope I’ve talked your ears off enough.
{ applause }
# Credits
