QA at Spotahome part 2: Testing our backend platform

This is the second post of a series where we share how we test our applications and services at Spotahome. In the previous one we commented about our frontend testing framework and how we saw it to be escalable. In this second one we will focus on our backend test framework and the alignment we have with the existing test magic in other layers of the platform.

Recalling the test pyramid

Let’s remember the testing pyramid theory so that we don’t lose focus:

In the previous post, we talked about e2e tests, also known as UI tests. These are the ones that take more time to develop, more time to run and are as well the most sensitive ones since just small changes in the frontend or the backend of your application might break them.

Let’s focus now on integration tests which for us are those working at the API level. API tests are (or must be) faster to develop and to run since they do not require any user interface to interact with and they directly hit the different endpoints designed in your application.

Following the pyramid stratification, these shall be more numerous compared to UI e2e tests, so we can leverage on them to increase our test coverage.

The platform behind

Our backend platform follows the micro-service pattern, decomposing the application in small parts (services) that take care of one isolated piece of the whole business. We follow a hexagonal architecture (aka ports and adapters) to avoid having them highly coupled and facilitating testing activities. In case you are interested in this topic you can have a look at this post.

To expose all the endpoints we make use of APIs which are consumed by our BFFs using GraphQL.

Regarding the base code, it is written in PHP and it follows the Domain driven design (DDD) pattern.

With all this what we have in the end are different bounded contexts (BCs) that completely isolate each side of the business.

The test framework

To test this we opted for Gauge which is an open source tool to write and run tests.

Gauge was created by ThoughtWorks (does this ring a bell?) and it is supported by a wide community now. It can be used in any platform and it supports many languages for writing your test code. Two key features that we liked are that it’s BDD and it supports data-driven executions, making your tests more escalable.

For writing the tests scenarios themselves Gauge uses specification and concept files. Basically, specification files are those containing the different scenarios written in a set of steps and concept files are just a collection of steps that you can reuse in the scenarios. You can read extensively about this in their docs.

Spec files

Create Booking Request
======================
Tags: booking
Create booking request and charge (OK)
- - - - - - - - - - - - - - - - - - - - - - - - - - ---------
* Login with "test-user"
* Create a booking lead for listing id "999999" with move in "2021–01–01" and move out "2021–01–31"
* Create a booking request from booking lead
* Charge booking
* Wait until the booking request is converted into booking
Create booking request for past dates (KO)
- - - - - - - - - - - - - - - - - - - - - - - - - - - ------------
* Login with "test-user"
* Create a booking lead for listing id "999999" with move in "2002–01–01" and move out "2002–01–31"
* Verify response status is "422"

Pretty understandable, isn’t it? This one above is an example of a specification file written in Markdown language with two scenarios. The scenarios just describe what they do in different steps. Each one of these steps has test code behind, but they can also relate to a concept file since they can be a set of steps themselves.

Usually, at the project directory, you would have a specs/ folder where you put all this. Inside it, we have many other directories sorted by feature where we put the spec files and have them located.

Note that Gauge lets you tag your specifications and even your scenarios with the Tag keyword, being this a kind of labeling enabling you to search and filtering them when necessary.

Concept files

# Charge booking
* JSON from file "jsonTemplate/bookings/chargeBookingRequest.json" is set as variable "chargeJson"
* Set path "/payment/booking-requests/{id}/chargeNow" as name "charge" replacing path variable "{id}" with variable "bookingRequest"
* An HttpSampler with name "chargeBR" and path in variable "charge" and type "PUT" and port "xxx" and json "chargeJson"
* Response code is equal to "204"

Here we show an example for the step “Charge booking” that we presented in our spec file. You see all the steps behind that one, meaning that every time we call “Charge booking” from a spec file, all these steps will run.

We usually do that when:

1. There are a lot of small steps required for just one high level task
2. We do not want to show so much logic at the spec file level
3. We are going to reuse such step in many test scenarios

Implementation

Let’s show as an example the implementation for the step “An HttpSampler with name “chargeBR” and path in variable “charge” and type “PUT” and port “xxx” and JSON “chargeJson””

@Step("An httpSampler with name <name> and path <path> and type <type> and port <port> and json <json>")
public void setHttpSamplerNameTypePortBodyGiven(String name, String path, String type, Integer port, String json) {
// Java code to send an http request with the passed on parameters
}

Note the @Step annotation. It denotes that the function is a step used in the spec files. For the Java code itself, I am not going to enter in detail since we’ve got lots of libraries and toolings in place to make the implementation reusable. I just wanted to show how a step relates to a function here.

This code would live in a Java class along with many other steps being somehow related and sharing context. Some examples of these classes would be:

• BookingSteps
• ListingsSteps
• HttpRequestSteps
• HttpResponseSteps
• ManageDataRequests
• …

And so on it goes… The clear objective of this is to keep the steps neatly organized and easy to find (even though if you use a gauge plugin you can jump to them directly from the spec).

Wrapping all the implementation we have JMeter to be able to run load testing with the scenarios developed. We can configure it passing on some parameters for the number of threads, ramp-up and duration and see how our platform behaves under such load.

As commented, you can write your test code in different languages like Java, Javascript, Python, C# and Golang. In our case, we opted for Java. This decision was made a while ago. Had we been presented with that question again, we’d have probably opted for Javascript as most of our frameworks currently use it.

Build

To manage and build the project we use Maven. As many would know, Maven is an extensively used build automation tool from Apache used in Java projects, although it can also be used for other languages. We’ve been working with it from the very beginning and it’s been useful to us given the big tree of dependencies that we have to maintain, both in-house and external ones.

Since we’ve got all our API tests in the same place, they are all placed in the same repository. So after updating our test collection, be it adding new tests, removing or refactoring, the build is triggered just after pushing the changes to master branch to make sure all is fine. With that, we make available the latest version of the tests for the BCs to grab and run.

Setting up the CI

If you remember from our first post, we use Brigade + GitItNow as our CI system and our BCs use them in the same way. For every BC, we’ve got a Brigade file to indicate the events that must run in such pipeline.

With that, GitItNow (in-house developed magic frontend tool), makes quite easy to interact with the pipeline and brings visibility to the whole process.

Above there’s one example of a pipeline for one of our BCs. As with our BFF pipelines, you can find a push event where we run some checks previous the deployment, a deploy event and a post deploy one were depending on the environment we may trigger different jobs and events.

For the api-tests event it gets the latest successful build from the test repository and runs it. In case anything went wrong, GitItNow makes available the logs for us to see the errors that might have happened. It also sends notifications to our Slack channels so that there’s no missing from our part and it enables us to answer swiftly.

Due to the number of tests in some BCs we opted to run them in parallel. To do this we need to be very careful with what to parallelize since we may be altering data used in other tests, so something to keep in mind when going for parallel runs.

What follows is a piece of Brigade code that enables parallel tests:

async (brigadeEvent, project) => {
try {
return Promise.all([
apiTestsJob(brigadeEvent,project,'search-dates','specs/search/dates).run(),
apiTestsJob(brigadeEvent,project,'search-filters','specs/search/filters').run(),
apiTestsJob(brigadeEvent,project,'search-no-filters','specs/search/no-filters).run(),
…

In the end what they are is independent jobs triggered in an async way, and the event that wraps it up will return OK only if all the jobs did well.

What do we have in that apiTestsJob? Well, there it is some logic followed by the execution command to run the API tests with gauge on a given directory:

mvn -s <settings> gauge:execute -DspecsDir=${specsDir}

Our strategy with API / Integration testing

In the previous post, we closed it up commenting a little bit about our strategy and how we decide whether to extend or not the e2e collection depending on the case. I want to do the same with this one as a kind of wrap up.

As you might figure out the approach is not similar since they act upon different layers of the application, so the strategy cannot be the same. We know that the integration layer is less sensitive than the e2e one and they take less time to develop and run, so they can be a good ally if what we need is speed.

In the case when a new endpoint is released the actions are clear. We will develop API tests (assuming that the development of the endpoint already comes with unit tests) to put some coverage on it. Typically we will try to cover all the possible response codes (including the error ones) and decide if we need to test more logic on it, which we usually do.

On the other hand, if what we have is a high level feature coming from a new product requirement, here we have several options. Since these requirements usually come well documented and with an end-user perspective, we might be tempted to cover it with an e2e (UI) test, but that may not be the best option. Instead, we can create an integration test at the API level to cover such logic and avoid touching the UI components. It’s just a matter of what we need and creating the smoothest verification that validates the new feature.

So the question we try to ask ourselves now for every inquiry of this kind that comes along is, Do we really need an e2e test to cover this? Can we cover it with API tests?

Hey! I am not saying that we must avoid creating an e2e test. Maybe one of the requirements is that the end-user must see some UI component in the frontend or clicking on a button… If that’s the case, then let’s go for the e2e option. Otherwise, if we only need to check the business logic or if we see that the maintenance of the UI test will be too expensive, then we might be fine with the API one.

What we try to find with this exercise is a balance between speed and coverage and the optimization of testing efforts. This is no silver bullet and I am not saying that this is the way of doing things in every situation. One just needs to find their best approach. Here at Spotahome we are continuously iterating and trying to find the best way that works for us, in our current situation and with our resources.

To us, it’s quite interesting the output that this move has had. A year ago we were more bold with creating more tests at the UI level, which of course covered the functionality we wanted to release. In the midterm we saw that many of those tests created a lot of noise in the pipelines -false negatives. Why was that? As mentioned, UI tests are quite sensitive and they suffer at many changes pushed by the squads, being backend or frontend changes.

The result, many red pipelines and many hours lost trying to find out what was happening.

We analyzed all that and decided to move some of the test scenarios to the API level. Also, for the ones that really needed to be at the UI level, we applied some trimming on what we were validating -the minimum necessary.

The result: Now we see almost no noise in the pipelines, we got some extra speed developing more API tests, we still feel safe on the checks we have and no critical bug made its way to production… at least yet!

So for the time being, we will continue pursuing this way since it’s demonstrated being pretty useful in our case.

This is what we wanted to share for this part #2. We hope you find it insightful, feedback is welcome! I want to thank you the colleagues/spotters that collaborated on the elaboration of this post since it is thanks to them why these posts bring some quality out there.