Humlix is a web API request builder and quality assurance tool that automatically generates tests. All you need to do is provide examples of how to call your API methods, and Humlix will use it to start doing its magic.
To put Humlix to a real test, we decided to try it on a real industry giant, and GitLab seemed to be a suitable candidate because of its size and that we can run it locally. Therefore, in this post, we will let Humlix create tests targeting GitLab Community Edition (gitlab-ce) and specifically its GitLab Groups API. We will use gitlab/gitlab-ce:11.2.0-ce.0 docker image and later try to repeat the same scenario on the currently latest version gitlab/gitlab-ce:13.1.0-ce.0
Let’s start with the end
By showing Humlix how to call the /api/v4/groups Gitlab Group API method, we could discover a bug that would have been hard to detect using regular unit tests. You can see that Humlix generated 1305 tests towards gitlab-ce in the screenshot below and that calling:
/api/v4/groups?page=461168601842738792
resulted in a status 500 Internal Server Error.
We can see the cause of the problem by going into the production.log file for GitLab and see that during the execution of some SQL statement, the error was:
2020-10-02_19:43:52.85686 ERROR: bigint out of range
2020-10-02_19:43:52.85689 STATEMENT: SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' ORDER BY "namespaces"."name" ASC, "namespaces"."id" ASC LIMIT 20 OFFSET 9223372036854775820
So in just a couple of minutes, we were able to detect bugs in production code.
A user has already reported this issue in the GitLab issue tracker system. To check if GitLab has fixed the bug, we can repeat the same scenario on the currently latest version gitlab/gitlab-ce:13.1.0-ce.0. Humlix once again located the same problem.
We can also verify it using this cURL command:
curl -I localhost/api/v4/groups?page=461168601842738792
HTTP/1.1 500 Internal Server Error
As you can see, this problem has existed for a long time, even between different versions, but Humlix was able to find the bug within minutes.
How to generate tests with Humlix
You need to download Humlix. Remember to sign up for the Humlix mailing list to stay up to date with the latest news. You can follow the Humlix getting started guide if you don’t know how to run it.
Select the HOME menu and click on ADD SUITE and create a test suite called GitLab Suite.
Press the Add request link and create a new test request called Get a group list by page.
Go to the ENVIRONMENT VARIABLES tab and change the environment name to GitLab Environment. Add two variables base_url and gitlab_token. Set the values according to your setup.
Go to the REQUEST tab so that you can see the Humlix test request builder. Select GET verb and Http protocol and add {{base_url}}/api/v4/groups. Add a new query parameter page of type integer according to the picture below.
Now we are ready to perform some tests. To verify that your setup is working, you can run a test with precisely the data you entered in the request builder by pressing the Run single test button.
If everything works as expected, you should receive something like this in the log area.
The result above shows that our setup works and that your Gitlab API token is correct.
Disclaimer: Don’t perform the next step unless you are sure you are targeting a test environment running GitLab that you can re-create without losing anything important. Humlix will generate many requests and could cause a heavy load on production servers.
Now you can press the Generate tests button to see some real magic.
If you don’t get the same result, you can experiment with the test request settings. E.g., you can change the Max number of generated test calls to 1000 and rerun the test generation.
Now you have seen what Humlix is capable of in just a few minutes and how easy it is to let Humlix create tests for you. Download and try it out if you haven’t already!