Test building

 

A test is a series of actions that are executed

If an action fails or one of its assertions fail, the test will stop and the remaining actions will not be executed, unless the option to Ignore errors is enabled in the test launch screen.

 

The form to create / edit tests is organized in three main parts:

Generally, each step form is organized in three areas:

Most actions have these three areas, except for the pause action, as it's a special kind of action. Because its function is to pause the test, it doesn't need any device to run on, and cannot have assertions or metrics (since it doesn't produce any kind of result):

You can always add steps before (button 1) or after (button 2) the step you're currently editing:

The button 3 allows you to change the current step to another action (e.g. change the ping action of step 2 to an API call).

Once you click one of these buttons, you get access to a menu to select which kind of action you want to add:

The Assertions and Metrics blocks are situated below action details. If you don't see them, you must first click Add assertions... or Add custom metrics... to make them appear:

Tests being about checking the good behavior of your services, assertions are key to doing checks. An assertion is something that must hold true, verified at the end of an action. If an action fails (for any reason, e.g. because its SIM card is unavailable, the address of the server to ping is unreachable, etc), the assertions are not run at all. A failed assertion causes the whole test to fail and end (unless using the Ignore errors option in the test launch screen).

The assertion tool is extremely versatile and offers a wide range of options, specific to the kind of action you are using. For instance, if you call an API, you can assert that its response status code must be 200, or 404, etc.

If your action is a FTP file download action, you can assert that the file's MD5 checksum must be equal to some value.

If your action is an email sending action, you can assert that sending the email must take less than 3 seconds, 10 seconds, etc.

 

These are only a few examples of the power of assertions. There are many possibilities, and you can add any number of assertions too.

Assertions are basically comparisons: you assert that some value (coming from the action's result) is equal (or contains, is one of, is greater to, etc) another value. That other value used for comparison can be hardcoded as in the previous examples, or it can be a variable:

You can also enter more complex expressions in that comparison value field. If you check the box Evaluate as expression, you'll be able to enter mathematical expressions.

The language used to interpret such expressions is JavaScript. If you are not familiar with it, just know that you can use all your favorite math operators with it: +, -, /, * (multiplication), etc. For safety, we only allow a subset of JavaScript, enough for common computations (but you cannot call JavaScript functions, say encodeURIComponent for instance).

 

If you have only subscribed to our service kapptivate Testing but not kapptivate Monitoring, creating custom metrics won't be of any use for you, as custom metrics are only visible in kapptivate Monitoring.

Below the Assertions block is the Metrics block. We automatically generate metrics for any monitoring test, but you may want to collect additional metrics, for instance, the delay in the response of an API server, or the latency of a ping. Collecting such metrics can be useful in the long run, to see the evolution of such data over time.

Defining these additional, custom metrics is what the Metrics block is for. The value of a metrics can be either hardcoded, or it can come from a variable. This is what a single measurement (the name for a custom metric) looks like in the Metrics block:

What are tags and values? Both are used to attach data to the metric. However, when viewing graphs in kapptivate Monitoring interface, you can only group your data and organize graphs based on tags. Values are only meant to be displayed, not used to organize the display of your data. Think of tags as metadata to organize your values. If it doesn't feel clear enough, here is a good rule of thumb: use tags for value that doesn't change a lot (e.g. action categories, common status code, etc) and values for the rest (e.g. response duration, latency, data returned by an API).

In this example graph form (taken from kapptivate Monitoring), the GROUPED BY option (area 2) can only operate on tags, while the SELECT part (area 1) can only get values:

Metrics are a deep and complex topic on their own, so if you want to know more about them, we suggest taking a look at the section of the documentation dedicated to them.

 

In addition to the Assertions and Metrics blocks, some actions also provide a Store block. The Store block is used to save response data or response headers in a new or existing variable:

To access an element inside a JSON response, start by typing $, then a path, like you would access a value nested inside an object in JavaScript.

For XML, the syntax is the same, but there are two exceptions:

The Store block supports saving in variables values taken from JSON responses, XML responses and response headers:

 

Have a look at this example:

 

If your variable is used in an action's owner field, then setting its value means choosing the owner (SIM, Ethernet agent, etc) in a list. In the example above, the variable SIM_A is an owner variable used for an action that requires a SIM, so clicking its field will reveal the list of available SIM cards.

If your variable is not used as an owner, then you can type any value you want. That is the case of variables STREET_ADDRESS and RECIPIENT in the example above.

To make a variable private or public, just click the eye symbol on its left. Notice that if you don't provide a value for a private variable, you'll get an error message Value of private variable should be provided. However it's fine to leave the value of public variables unspecified, as their value can be assigned later, before running the test.

Have you noticed the three dots button ⋮ on the right of each variable? If you click it, you'll see that there is an option to reset (to blank) the value of the variable:

 

If your variable is a regular variable (meaning, not an owner variable), you'll get one more option labeled Set value from template:

 

When you use a variable to match some text (in a USSD or SMS Expected Message), you may need to write a regular expression to be sure to match only the kind of text that you need (it could be a number, a date, etc). Clicking Set value from template will reveal a list of commonly used regular expressions, to match many kinds of numbers, dates, percentages, etc:

This is a helpful shortcut, if you ever need to match text with a specific format. Of course, you can also write the regular expression yourself.

Global variables are a special case: when you use the name of a global variable in your test, you are not able to assign it a value. Instead, you will see all the values of that global variable (as set in the variables settings page), for each platform. Remember that a global variable can have one or more values in one or more platforms, and may not have any value at all on some platforms. In the example above, our variable has two values on Production platform and one value on TheBeta platform.

When you'll run the test, you'll be able to select exactly the value that you want to use for that global variable, and even to enter a custom value.

 

Apart from building a new test from zero, you can also clone an existing test or export it to another workspace (workspace means product here). Simply go to the tests list and click the three dots button ⋮ next to the test of your choice.

 

You can click Clone, which will take you to the test edit form with a copy of the test to operate on:

 

Or you can click Copy to other workspace, which will take you to this page where you can select which products to copy the test to:

 

Another quick way of building a test, if you have done a lot of manual testing before, is to create a test from your manual testing history. First, head to the history page: 

 

To create a new test from the history, simply click Add next to any command you wish to reuse:

When you are done, just click Create test:

 

You will be directed to the test creation page, and the test will be ready for you to save. You will only need to enter a name for it, all the actions previously selected will be present: