The biggest and most hostile challenge in software development is writing and maintaining test cases. This is no different in terms of chatbot development. At Botium, we don’t write regression tests, we generate them. This article shows how we do this with minimal investment.

For the best coverage, you need to define all possible conversations in your conversation template. Manual deployment and maintenance is really time consuming and sometimes tedious, not to mention human errors that can easily happen even with a fairly simple chatbot. We’ve implemented an indexing tool to help you do it in a very simple way.

The crawler recognizes the buttons and Quick Answers and talks to them. The discussion tree runs a slightly customized depth algorithm. Every time the crawler reaches an open question (meaning no button or quick response found), then the conversation is ended, the path is marked as done, and a new conversation is started from the beginning (from the conversation start message) to keep the context of the conversation safe. (The crawl process starts conversations with messages defined by the ‘Conversation Start Messages’ parameter.) When all the paths in the conversation tree are covered, the session ends and you get all the possible conversations as a result, so you have a complete regression test. Let’s see how it works in practice in Botium box.

For a better understanding, I use a very simple example of a bank chatbot mixed with buttons and open-ended questions.

With Quick Launch, you can configure your crawl project in three simple steps. In the first step, you need to select or register a new chatbot. Then, in the second step, you can configure some basic settings for the chat robot. In the third step, you can save the Crawl Project or you can start the first Crawl cycle right away.

Once registration is complete, you will be directed to the Crawler Project Dashboard. Here you can see previous crawl sessions and current execution settings.

During the crawl meeting, as many parallel processes are started as many “Conversation start messages” are specified in the execution settings. These processes identify all possible conversations with buttons and quick responses, as already mentioned in the Crawl Concept.

The example bank chat robot has chats initiated by the bot. The crawler is also able to detect buttons and quick replies in welcome messages, so in this case we can leave the Conversation Start Messages field blank.

1. How chat technology can automate customer service

2. Automated vs. live chats: what does the future of customer service look like?

3. Chatbots as a medical assistant in the COVID-19 pandemic

4. Chatbot vs. Smart Virtual Assistant – what’s the difference and why treatment?

And here is the highest value of the crawler: created conversations. This chatbot is quite small and simple, so in this case we only have five conversations as a result. In the case of a more complex chatbot, there are hundreds of test cases, which is a huge amount of work to do manually.

Another feature that is just as useful as created conversations is a flowchart that shows the entire detected conversation tree in visual form to get a great clear picture of your chatbot.

As you can see from the previous section at the bottom of the flowchart, there are “open questions” like ‘Which date would be best for you? We need 24 hours… ‘. In this case, the discussion is terminated from the crawler’s point of view, but with human interaction it can be continued. We also have a solution to this problem.

You can specify multiple user answers to ‘open questions’. These responses are identified in the next crawl box as if they were buttons. At the end of the unfinished conversations in the added user response, the flowchart became much larger and the conversations created were doubled.

As you can see with a few minutes of easy work, we created ten conversations for this bot. If the button is a full button, you have even less work to do, just press the start button and wait a few minutes.

But what can we do with these discussions? In other words, these are test scripts. Click Copy Test Scripts to Test Series to copy them to a new or existing test set.

A test suite is a collection of test cases that can be added to a test project. At this point, a regression test with pretty good coverage is ready for this bot.

  • Conversation start messages
    These are the so-called conversation start messages with which the crawler starts conversations. The more start-up messages you have, the more parallel jobs will be started.
  • Maximum number of discussions
    This is the depth of the debate. (One step is a pair of messages between the user and the bot.) When the specified depth is reached, the conversation is terminated and marked as successfully completed.
  • Number of welcome messages
    There are chatbots that start a conversation without user intervention. In this case, you need to specify how many greeting messages the bot will send.
    If the robot has a welcome message (s) and you do not specify any start messages, the crawler will try to find quick replies and buttons in the welcome messages and start conversations along them.
  • Wait for the prompt
    Many chat bots respond with multiple bot messages. In this case, you can specify a timeout until the crawler has to wait for bot messages after each simulated user message.
  • Exit criteria
    In the case of a complex chatbot, it often happens that we only want to test a certain part of the chat tree. In this case, you can specify exit criteria to exclude part of the tree.
    If the text of any one-time answer / button meets one of the exit criteria, that conversation will be paused there and marked as a successful conversation.
  • Combine statements
    All text messages are saved as statements. The crawler recognizes non-unique statements and combines them into a single expression.

Botium Crawler is a very powerful tool for creating regression tests. It is able to produce all test cases on a happy path without user interaction in the case of a fully button-based chatbot and with minimal user interaction in the case of a partial button-based chatbot. Use the flowchart to view the chatbot and identify circles.
It’s a brand new tool, so there’s still a lot to improve, and we already have a lot of ideas. For example, we would like to introduce different tree traversal algorithms to achieve more efficient performance so that you can choose the algorithm that best suits your chatbot. In addition, you can add RegExp as your exit criteria, we are going to make the open question feature more convenient and so on.
Without the right tools, you will lose. The crawl function is part of our ticket product Botium box which will help you in successful chatbot testing.


Please enter your comment!
Please enter your name here