Writing Tests

Let's write our first few tests.

Creating a scenario

As mentioned earlier, a scenario refers to the JSON file in which we write tests. We can either create a JSON file inside a collection and fill in the fields, or, to make things easy, use ask Vibranium to create a scenario file for you, pre-filling all the essential data.

In order to create a new scenario, run the command:

vc c -c {collectionName} -s {scenarioName}

Here vc c or vc create refers to the create command in Vibranium, and {collectionName} and {scenarioName} refers to the collection name (directory/package name) and the scenario JSON file name respectively. Refer to Create command for more details

Once you run this command, Vibranium will create a new scenario file and open it in your default JSON viewer. You can either delete the entries inside the endpoints key in the scenario of keep it for reference. For the examples below, we will create fresh tests, so it is recommended that you remove them.

Write your first test case

Let's start writing our first test.

Assume that we are writing a test for the endpoint GET /api/v1/users. The simplest test case that you can write is to check if the endpoint is up and running and returns status 200 OK when called. For this, add the following JSON object into the endpoints array in the scenario:

{
    "name": "get_all_users",
    "description": "Endpoint to get all user details from the system",
    "url": "/api/v1/users"
}

The above mentioned JSON object will call the /api/v1/users using GET (if we don't specify any method, it takes GET as default) and checks whether the API returns a status of 200. Writing a basic test case is this simple!

Now say we want to validate more things, other than the status code. For this, we can use the expect key in the endpoint. For example if we want to call the users list api to get all user details and validate if

• API returns a status code of 200
• The Time to first byte should be less than 300ms
• The total time taken by API should be less than 700ms
• The response is of content type JSON
• Response should have at least one entry
• At least one user is admin
• All users have proper IDs

{
    "name": "get_all_users",
    "description": "List all users",
    "url": "/api/v1/users",
    "method": "GET",
    "expect": {
        "status": 200,
        "headers": {
            "content-type": "application/json"
        },
        "response": {
            "There should be atleast one entry": "{response.length} > 0",
            "Atleast one user should be admin": "{response.all.isAdmin}.filter(isAdmin => isAdmin).length >= 1",
            "All users have an ID of 32 characters": "{response.all.id}.every(id => id.length === 32)"
        },
        "timing": {
            "total": "<700",
            "firstByte": "<300"
        }
    }
}

Was that a little too much? let's discuss it line by line.

• Till the expect key, it is pretty much the same as the before example, so I'm leaving that part
• "status": 200 tells Vibranium that the expected HTTP status code is 200 and to fail the test if we get a different value. By default it takes 200, so even if you don't specify this line, the check is automatically handled
• "headers": {...} refers to checks related to the response headers. It is a key-value pair, with key referring to the repose header key and the value referring to the expected value for the key.
• "content-type": "application/json" as mentioned in the previous point, this means that the expected value for the content-type header in the response is application/json
• "response": {...} refers to checks related to the response body of the endpoint. This is also a key-value pair, with the key containing the test name/description and the value being simple JS snippets that does the check. There is a special response check when you specify the key as schema. We'll go through this later.
• "There should be at least one entry": "{response.length} > 0" The key refers to the check description and the value contains a variable (any string enclosed in curly brackets '{' and '}'). {response} is a variable that contains the response body. We use dot notation to parse the object, so {response.id} means the id key inside the response. For more details, refer to Variables.
• "Atleast one user should be admin": "{response.all.isAdmin}.filter(isAdmin => isAdmin).length >= 1" As mentioned in the previous point, we use dot notation here and response is a special variable with our response value in it. We have special keys like all, any, any_n etc about which we'll discuss in detailed later, but for now, it just means that we are taking all the entries in the array. For example, {response.all.id} is the same as response.map(r => r.id) in JavaScript.
• "All users have an ID of 32 characters": "{response.all.id}.every(id => id.length === 32)" If you have understood the previous point, this is very simple. {response.all.id} gives you an array of IDs and we are evaluating JS every function on the array to see if all the IDs have a length of 32.
• "timing": {...} refers to the checks related to the response timing. You can specify a max value for the response timing and fail the test if it takes more than a certain amount of time. All timing values are in milliseconds. The available timing values that you can check are total, firstByte, wait, dns, tcp and download
• "total": "<700" fail the test if the endpoint takes more than 700ms in total
• "firstByte": "<300" fail the test if the endpoint takes more than 300ms for the first byte of the response

Time for a slightly more complex test.

Let's write test to update the details for a particular user. The basic requirements of this test is that we first need to have a user in the system. There are two ways of proceeding with this. Either we can take an user from the users list api and update it, or create a new user and then update it. Since in many cases, there is no guarentee that the system has data already available, so we'll proceed by creating a new user. In most cases, we might already have the test for this, but for the sake of this example, let's say we already have a test as folllows in the scenarios:

{
    "name": "create_a_user",
    "url": "/api/v1/users",
    "method": "POST",
    "payload": {
        "name": "My awesome username"
    }
}

Now we'll learn how to use this endpoint as a dependency in our update API test.

Here is the JSON for update user test. I'll go through the important parts of this example below.

{
    "name": "update_user_details",
    "description": "Update user details",
    "url": "/api/v1/users/{userId}",
    "method": "PUT",
    "variables": {
        "newUserName": "{dataset.names}"
    },
    "payload": {
        "name": "{newUserName}"
    },
    "dependencies":[{
        "api": "create_a_user",
        "variable": {
            "userId": "response.id",
            "oldUserName": "response.name"
        }
    }]
}

• All the keys till variables are self explanatory and so I'm skipping them

• variables is a key used to define variables in the test. Head over to Variables if you need more details on this, but to explain in simple terms, it is just a key-value pair, with the key denoting the variable name and the value denoting the value for the variable.

• "newUserName": "{dataset.names}" means that we are creating a new variable named newUserName with the value {dataset.names}. dataset is a special keyword in Vibranium, used to denote pre-defined data values. {dataset.names} means use any value from the inbuilt names dataset. More details on datasets is also available in the previously mentioned Variables page.

• payload key is used to denote the payload to be used in the endpoint. It can be of any type, depending on the endpoint. A payload can also be a string, starting with the ! symbol to denote that the payload needs to be pulled from a file. So if the payload value is !payloadForUpdateUser, then the payload values is taken from the file named payloadForUpdateUser.json inside the payloads directory inside the tests directory.

• dependencies key is used to denote the list of dependencies to be executed before executing the given endpoint. In this case, we need to run the create user api before running update user, and hence we define that api as a dependency. Head over to Dependencies for more details of dependencies.

• "api": "create_a_user" indicates that the api with the name create_a_user is a dependency for this endpoint. If the dependency is in the same scenario file, you just need to mention the api name, else if it in the same collection, we have to mention both api name and the scenario name and if the api is in a different collection, we need to specify api name, scenario and the collection. In this case, the endpoint is in the same file (as mentioned above) and so we define only the api key

• variable key denotes the variables that are to be pulled from the dependency response. So if we define "userId": "response.id", it means that after the create endpoint is executed, the id field from the response is taken and assigned to the variable named userId, so that we can use that value in our endpoint.

And now, let's add some assertions...

{
    "name": "update_user_details",
    "description": "Update user details",
    "url": "/api/v1/users/{userId}",
    "method": "PUT",
    "variables": {
        "newUserName": "{dataset.names}"
    },
    "payload": "!exampleCollection/updateUserPayload",
    "dependencies":[{
        "api": "create_a_user",
        "variable": {
            "userId": "response.id",
            "oldUserName": "response.name"
        }
    }],
    "expect": {
        "status": 200,
        "headers": {
            "content-type": "application/json"
        },
        "response": {
            "User ID is valid": "'{response.id}' === '{userId}'",
            "User name should be same as input": "'{response.name}' === '{newUserName}'",
            "schema": "!exampleCollection/updateUserResponseSchema"
        },
        "timing": {
            "total": "<700",
            "firstByte": "<300"
        }
    }
}

Note that I have made changes to the payload key, and make it a file reference. So I'll have the corresponding payload file as workspace/Vibranium-Tests-Directory/payloads/exampleCollection/updateUserPayload.json

Also, I have added a new key in expect.response, called schema. As mentioned before, it is a special key used to validate the response schema. If I place a JSON schema v6 based schema definition for the expected response and place it in workspace/Vibranium-Tests-Directory/payloads/exampleCollection/updateUserResponseSchema.json, Vibranium will use the schema file to validate the response and report the status.

Hopefully this was and informative Getting started guide. For more details on specific topics, please refer to the pages in documentation section.

« Previous