What is CSIT?

Continuous System Integration Tests. Basically, Suites of integration tests for testing OpenDaylight components (alone or with other projects as OpenStack for example).

If you are familiar with OpenStack testing, then  you can say it’s very similar to Tempest scenario tests.

The tests are executed automatically by OpenDaylight’s Jenkins on the lab provided by the Linux Foundation and can also be executed manually be any user, using Robot framework the integration/test repository.

This post should help you to execute the tests in your environment and publish the results with Jenkins Robot plugin.

You can also find small section of common failures in the bottom of this post.

To obtain the tests, run:

Fair warning

I feel like I have to warn you, running CSIT is not an pleasant experience. Some of the reasons I found it really frustrating:

  • Documentation is bad. At this point of time,  OpenDaylight’s documentation is really awful, and while this is our mutual responsibility to make it better, it would be nice if those who writing the documentation make no assumptions on the users or their environments.
  • You will probably fail to use it “out of the box”.  There are some variables that should not have default values. One quick example would be the following error: “Login with public key failed for user: ‘jenkins'” – I find it annoying  because I don’t have ‘jenkins’ user on my system and secondly, there is no way to override it except for changing the original Python file containing this variable.
  • Error handling – you will often find yourself wondering “what exactly went wrong here?”. For example test which fail to grep process, output the following error: “could not convert string to float” rather than output “no such process!”.

Now this is only a small part of what I think is terribly wrong with Robot + CSIT at the moment. Hopefully this post will ease the pain 🙂

Goal and Environment

CSIT contains a lot of test suites. To keep it simple and clear, we’ll focus on executing ‘netvirt/Openstack_Neutron/010__ovsdb_flow.robot’

I used several topologies while testing, for the purpose of this post, let’s assume the following setup is used:

1 OpenStack all-in-one node (runs nova, neutron, etc).

1 Tester node (clean centos/rhel vm).

1 OpenDaylight node (install and run OpenDaylight using the RPM or from source).

Setup the tester node

First, make sure you have the following RPMs installed on the tester node:

If you don’t have repository that contains those packages, you can use EPEL:

We are going to run the tests using the Robot framework, this is how the CSIT tests executed. So let’s install the robot framework related packages:

Next, clone the CSIT tests repository. This is where you can find all the test suites.

Setup variables

Before executing the tests, you need to define several variables. I usually keep those in a file so I can source it later and have all the variables defined.

The list of variables depends on which suite tests you chose to execute. I’m going to use variables related to csit/suites/netvirt/Openstack_Neutron/010__ovsdb_flow.robot suite.

This is the list of the variables you need in order to execute 010__ovsdb_flow.robot:

  • ODL_SYSTEM_IP – Your OpenDaylight node’s IP. By default it’s
  • NEUTRON – The IP of the OpenStack node where neutron server is running. No default.
  • KEYSTONE – The IP of the OpenStack node where keystone is running. No default.
  • TOOLS_SYSTEM_IP – The IP of the node where you have your ovs utils installed. In my case it’s the OpenStack all-in-one node. Default is
  • WORKSPACE – The  dirname of the path where your OpenDaylight distribution is located. If for example you have odl at ‘/opt/opendaylight’, then your workspace is ‘/opt’.
  • BUNDLEFOLDER -The  basename of the path where your OpenDaylight distribution is located. If for example you have odl at ‘/opt/opendaylight-2.1’, then your bundlefolder should be  ‘opendaylight-2.1’.
  • USER_HOME – the user home directory path (e.g /home/mario)

Execute the tests

One “short” command

pybot is the robot framework command name.

With -v you specify variables. You don’t have to do it with this syntax ${<var_nam>}, it’s simply comfortable  if you sourced file with all the variables before running this command.

The last part is the suite or or tests file to execute. You could use ‘test/csit/suite/netvirt/Opestack_Neutron’

Integration with Jenkins

There is a robot plugin for Jenkins. You can find more detailed description of it here.

First, install the plugin on your Jenkins server. Once installed, you will have a new post build action called “Publish Robot Framework test results”.

It looks like this:


After executing tests with Robot, you will have three files generated: output.xml, report.html and log.html. The plugin expects by default to find those in your workspace, if you have them in different location, change the ‘Directory of Robot output’ field.

Next, is the Threshold configuration. Set the percentage for marking the build as “PASSED” (green dot) and for marking the build as “UNSTABLE” (yellow dot).

This is what I used for Jenkins job definition (to use with jenkins job builder):

Common failures

List of common failures you may encounter:

‘ProtocolError(‘Connection aborted.’, error(110, ‘Connection timed out’))’

This usually happens because the tester node can’t reach the OpenDaylight node. The actual IP address and the port used, can be found in the test report.

To resolve this issue, you should add iptables rule to allow traffic from the tester node to openstack controller and opendaylight node.

You can enable all the traffic, like in the below example or enable the specific one that you need (which is preferred)

‘ ‘ cannot be converted to a floating point number: ValueError: could not convert string to float

You will see this error if your OpenDaylight deployment path doesn’t include the string “distribution”.

It happens when the test connects odl node and grep for java.*distribution.*karaf (Anther assumption on your environment).

It should look only for java.*karaf and I submitted a patch for fixing that, but I can’t say when it will be merged. In the meantime you should change it from java.*distribution.*karaf to java.*karaf in the test code or change path to include ‘distribution’.

No match found for ‘>’ in 30 seconds

There is a variable in test/csit/variables/Variables.py called ‘DEFAULT_LINUX_PROMPT’ and this variable is the string expected to be found in the prompt of OpenDaylight node.  By default the value is ‘>’.

Which I guess is fine if you have an easy to override it ( you don’t 🙁 ). You can either change the prompt of the server or edit the variable in Variables.py