Using BDD for State Transition diagram testing


In Agile software development lifecycle, Behaviour Driven Development (BDD), proved to be a useful testing technique to define the test cases based on the expected behaviour of the software. On of the main characteristics is that the defined test steps follow the below pattern :

Given some initial context, 
When an even occurs 
Then ensure some outcomes

This technique is used and there are many examples on the web on how to use it, found in different places :

To get the maximum value from the usage of this technique, the following guidelines were proposed :

  • Describe what not how, e.g. it is not particularly useful to define all the ‘clicks’ and ‘reloads’ to validate a user story
  • One test, one topic, each test should ideally be focused on one topic and each topic should be ideally described by one test. Otherwise, a test that validates multiple actions, it will be fragile and will cause a lot of maintenance. 

Source : Fifty quick ideas to improve your tests

The problem

When it comes to test workflows that involve different personas interacting with the application, it might be a challenge to create from scratch a complete test case to test the valid or non-valid transactions. 
For instance, lets consider a web based application that implements the following workflow transition diagram :

1. When a customer submits successfully a form and the ‘customer case’ object is created and its status is moved to ‘Open’

2. When the analyst assigns to himself the case, the ‘customer case’ status is moved to ‘In progress’

3. When the analyst completes his work, the status is moved to ‘Done’

4. When a customer case is in ‘Done’ state, an approver may move the case to ‘Rejected’ state or ‘Approved’ state. 

5.a If ‘Rejected’, the analyst can reassign the case to himself/herself by moving the status to ‘In progress’

5.b If ‘Approved’, the customer report is delivered to the customer 

To automatically test this workflow however, inevitably it will involve :

  • logging in and logging out as Customer/Analyst/Approver
  • Submitting a form (e.g. typing into text fields, selecting options)
  • Clicking of buttons or even
  • Checking if previously submitted information is present
  • Uploading or Downloading a file, or even 
  • Receiving email notifications

Writing the test cases

Considering the above, the use of the  ‘State Transition Testing‘ black box testing technique would enable the tester to cover all the valid and invalid state transitions. Following that technique, we would be more focused on the verification of the overall completion of the workflow rather than of the interaction of each particular element.

Possible risks by using this technique are 

  1. the execution time of a test case might be long since the complete application/system will need to be used, and 
  2. the test case might be too fragile in case of a WebUI change

Considering the above, we avoided to design too descriptive test cases like this one :

Scenario : Verify Open > In progress > Done > Approved  path 
Given I log in as customer 
When I click the button ‘Request a case’ 
And I fill in the requested form 
And I submit successfully the form 
And I log out 
Given I log in as analyst 
When I click on the submitted case page 
And I assign the case to myself 
And I compete the case analysis by uploading a PDF report 
Then I check the case is on ‘Done’ state 
And I log out 
Given I log in as approver 
When I click on the submitted case page 
And I approve the customer case 
Then I check the case is on ‘Approved’ state 
And I log out

Instead, this is how we actually design the test case:

Scenario : Verify Open > In progress > Done > Approved path
Given as a customer, I submit successfully a case
When as an analyst, I assign the case to myself
Then as an approver, I approve the case

Implementation and libraries

For the purpose of this article, we use the python-behave , a BDD implementation in Python style. 

In our second test case description, we created 3 ‘high level’ test steps in order to cover the ‘Open’ to ‘In progress’ to ‘Done’ path. These ‘high level’ steps they use and execute ‘lower level’ steps with the help of the method context.execute_steps. This method allows you to execute inside a step another step that was defined elsewhere. 

For instance, the step ‘Given as a customer, I submit successfully a case’ could have the following definition :

@given(‘as a customer, I submit successfully a case’) 
def step_impl(context): 
      context.execute_steps(”’Given I log in as customer”’)
      context.execute_steps(”’When I click the button ‘Request a case””) 
      context.execute_steps(”’When I fill in the requested form”’) 
      context.execute_steps(”’When I submit successfully the form”’)
      context.execute_steps(”’When I log out”’)


 It is important to think about what we want to achieve with the  BDD test case. So when it comes to verify long and complex state machine diagrams, for the benefit of the tester or the reviewer it could be more convenient to structure the test case with as simple steps as possible.  

Enabling Continuous Compliance with Confluence during Training Process Audit

In regulatory industry (such as the medical device , oil industry), the establishment of a Quality Management system (QMS) is an essential element of a business.

In this article, we explore how we could continuously audit the Training Process using our Confluence plugin QC Read and Understood for Confluence .


The cost of auditing a process manually

Via the internal audit, the leadership team is assured that the company operates in alignment with to standard operating procedures (SOPs). As the QMS matures and grows, the cost of operating the internal audits increases.

Current industry practices suggest that all compliance activities should account to no more than 10% of the total project delivery costs. However, this cost could very easily be increased mainly because of the growing amount of planning and executing audit activities.

Hopefully, the introduction of the risk-based audit using ISO 9001:2015, could reduce the pain by planning and prioritising the audit activities based on a risk based approach.

Screen Shot 2018-08-12 at 21.53.46.png

New hope with ‘Continuous Compliance’

Continuous compliance (CC) is about achieving compliance, and then maintaining compliance on an ongoing basis.

In the content of auditing the Training Process, we need to make sure that all appropriate employees have read and understood the latest version of the active Processes.


CC to Training Process using the QC Read and Understood for Confluence

In our case, we would like to check that all appropriate employees have read and understood the latest version of the processes.

By using the ‘QC Read and Understood for Confluence’ the Confluence users are able to confirm that they have read and understood a Confluence page that could contain a process.


Then, the QMS auditor by using the QC – Read and Understood table macro command is able to create an overview table that lists all the necessary processes along with the user’s approval statuses :

Screen Shot 2018-08-12 at 22.46.16


In this way, we have reduced the time needed for the QMS audit to check if all the employees have read and understood all the applicable processes by just creating an overview page and checking its content.

In addition, in this way we have created  transparent and quantitive way to measure compliance with the Training Process.

If think that this plugin might be useful in your case, you are now about  to try it out in Confluence Cloud.

Marketplace Badge-dark@2x

Integration with HipTest and Travis CI

HipTest is web-based test management platform that supports continuous delivery and bridges the gap from manual to automated testing. It provides a real-time environment for creating, executing and maintaining tests.

Travis CI a hosted, distributed continuous integration service used to build and test software projects hosted at GitHub.

We used both Hiptest and Travis CI to automatically executed our python behave test cases located at GitHub and here below we will demonstrate what we learned.

Installation and Configuration

  1. Installation of the overall setup is fairly easy since both Travis CI and HipTest provide quite useful documentation. We had to create:
  2. We set Travis CI to execute all the test cases every time the repository is changed. The status page of this Travis job is publicly available to everyone python-pdf-analytics-client .
  3. After the execution, all the test case results are set to be uploaded to HipTest test run execution called ‘Automated execution via Travis’.

What we learned

    • Both Travis CI and HipTest documentation was very useful and up-to-date based on the current configuration of their platforms
    • We liked Travis CI , it is so much easier to demonstrate the features and the capabilities of the python-pdf-analytics-client library in real time using tow different python versions Python 2.7 and Python 3.6  :
    • We liked HipTest too, all our test execution results are now stored stored in a single platform monitoring the performance over time is now becomes more easierhiptest.PNG



First blog post

Today, bills,  employment contracts, medical records, public announcements etc are delivered as (Portable Document Format) PDF file format.

Usually the content of the PDF is outcome of the chain of sub-tasks which are usually prompt to errors and mistakes. However, the content of the final exported PDF is what matters.

Therefore, when it comes for the inspection of the final PDF, as an inspector I would make sure that :

  • The right text is located in the right location
  • The correct images (logos, diagrams etc) are displayed as expected
  • The text is displayed in the correct font type and font size

These are the questions that the PDFAnalytics web-based tool ( try to answer. Enjoy !


Checking a payslip

In this article we will demonstrate how the PDFAnalytics web service can be used to verify automatically the content of a payslip PDF file.

We will use the PyPI library python-pdf-analytics-client which allows to automate the most common functions of the PDFAnalytics service.  The test cases will be written by using the python behave library.

The demo payslip PDF file is located inside the GitHub repository .

        1. The payslip document is located inside the GitHub repository 
        2. Get an account on PDFAnalytics and log in
        3. From the PDFAnalytics, upload the payslip to start the inspection as show belowupload_pdf
        4. Check the coordinates on where the expected text boxes are located inspector
        5. Create a directory for your behave_payslip
          $ mkdir ~/behave_payslip
        6. Setup a virtualenv python environment from the
          $ cd ~/behave_payslip
          $ virtualenv venv
          $ source venv/bin/activate
          $ pip install python-pdf-analytics-client
        7. Create the python behave file structure
          • : to run your environmental code
          • : the PDF payslip document
          • features/ : to store your feature test steps
          • steps/ : to store your python test steps
          • steps/, to store the actual test steps. For the test steps you can reuse (i.e. copy ) the definitions from the file on Github
        8. Create your feature file at features/payslip_pdf.feature . Check the example below to get an idea :
           Scenario: Verify payments and logo
           Given the pdf file "payslip.pdf" is sent to be analysed
           Then I "can" see the image "payslip_logo.png", at [left, top] ["100", "100"] on page "1" in pdf
           And I read "Total gross <br>payments:", at [left, top] ["74", "599"] on page "1" in pdf
           And I read "£1021.43", at [left, top] ["160", "584"] on page "1" in pdf
           And I read "Total <br>deductions:", at [left, top] ["250", "603"] on page "1" in pdf
           And I read "£304.92", at [left, top] ["334", "587"] on page "1" in pdf
           And I read "Net pay:", at [left, top] ["430", "570"] on page "1" in pdf
           And I read "£716.51", at [left, top] ["516", "586"] on page "1" in pdf
        9. Run you behave test cases from the command line:
          $ behave -D token=|your-personal-token-ID-from-the-PDFAnalytics-site|
        10. Done !


You have find a copy of this code at the Github/examples .

If you have any questions do not hesitate to contact us.