Arie Bregman

Linux And Stuff

OpenStack Infra: How to deploy Zuul

This is the second post on Zuul, which focuses on deploying it and its services. To learn what is Zuul and how it works, I recommend to read the previous post.

Methods of deployment

I’ll introduce two ways to deploy Zuul. Both are basically the same, but in one you will have this nice Ansible role which you can execute really quickly, while using the other way will require you to execute each command manually.

For a quick deployment, use the Ansible way. To learn how exactly the deployment is done, step by step, use the manual way.

Note: This assumes you are deploying Zuul on RHEL/CentOS 7 server.

How to deploy Zuul

Pre deployment – Gerrit & Zuul connection

To have working Zuul server, you need to have access to Gerrit with the user you plan to specify in zuul.conf (let’s assume it’s named zuul, although it can be any other user).

To make sure you have the required access, run the following:

The short & automated way

First, clone ‘ansible-zuul’ project. I created this project in order to provide easy and automated way to deploy Zuul

Next, create the deploy.yml playbook (or copy it from ansible-zuul/playbooks/deploy.yml):

gerrit_server is the url of your Gerrit server and status_page is set to include the web page setup or not. By default, when you do ‘pip install zuul’, it will not setup the web page for you (No idea why…).

Now edit /etc/ansible/hosts and make sure the target server (where you want to deploy Zuul) is mentioned there

Now run the playbook (it may take couple of minutes):

Enjoy, you have Zuul 🙂

You can now proceed to adding your first pipeline and project.

The long & manual way

Wow, you actually chose the long way? nice. Let’s not waste time.

packages and user

Connect to the server and install the following packages:

If you don’t have those packages available, try to install EPEL repo first OR switch to pip installation.

Add the user ‘zuul’ (make sure it also creates the ‘zuul’ group)

project installation

Clone the Zuul repository

Install Zuul

Services

Now let’s create the services file. You might be using init or systemd, so I covered them both.

Let’s start with systemd. First service would be zuul-server

Next, zuul-merger

Next, zuul-launcher

Now the init (remember, you don’t need it, if you setup the systemd files).

I’ll put here only one of the files, since it’s quite long and basically it’s the same file, so simply switch from any ‘zuul-server’ to ‘zuul-merger’ and ‘zuul-launcher’ when you copy paste the files.

Zuul-server init file will look like this:

Yeah, that was painful, but let’s keep going.

Make sure you have permissions to execute it

Next, create this list of directories

Configuration files

Let’s start with the main configuration file (‘zuul.conf’).

Before you edit it, note that this file is very important and requires you to change some values according to your setup.

For example, you may have dedicated Gearman server, so you should not put “start = true” like in the following example. You should also point the Gerrit server option to your Gerrit and put the ssh key in place according to sshkey option.

Good,  now let’s setup the logging configration files. Again, I’ll put one example for them all (launcher, gearman, merger and server), so you should copy & paste & switch values.

Stay strong, you are almost there.

Now the exciting part – let’s start the services

Done! I’m proud of you!

There are other stuff you might need to configure/enable, but if you reached this step, it’s a piece of cake for you.

Connect Jenkins to Zuul

Or more accurately, connect Jenkins to Gearman.

If you chose to work with Jenkins, go to your Jenkins server and click in the left menu on “Manage Jenkins”. Next, click on “Configure System”.

gearman-jenkinsLook for Gearman configuration. You should see to blank fields for ‘Gearman Server Host’ and “Gearman Server Port”.

In “Gearman Server Host” put the IP address of your Gearman server or Zuul if you configured Zuul to spawn Gearman.

In the port you should put the number that match your zuul.conf configuration (in my setup it’s 4730).

Don’t  forget to check the “Enable Gearman” checkbox. It is also recommended to push on the “Test Connection” button to verify that Jenkins is able to connect the Gearman server.

Zuul web status page

Zuul includes nice status page to display all the pipelines and the active builds. Status source code

Unfortuantly, when you deploy Zuul, it won’t automatically install the status page. You can either use ‘ansible-zuul’ to deploy it (add the variable ‘status_page:yes’ in the main playbook).

Or you can follow the next manual steps.

Start by installing httpd

Next, you need to set selinux bool to permit httpd to make network connections

There are several dependencies required to run the status page. There is a script to fetch and install them.

You need to install them in your zuul httpd directory (by default it should be /var/lib/zuul/www). To do that, simply put the script there and run it

Next you need set up all the html and javascript files in Zuul httpd directory. Those files should be taken from Zuul installation path (in my setup it’s ‘/opt/zuul/etc/status/public_html’)

Now setup zuul vhost section in httpd.conf.  Insert the following section in ‘/etc/httpd/conf/httpd.conf’ and make sure to change the variables to match your environment!

It’s important to change lines 2-4 to match your environment profile.

That’s it, the only thing left for you to do, is to start httpd

You can now try to access Zuul web status page in your browser: http://<zuul_server>

Verify Zuul is installed properly

First, let’s check Zuul is in the processes table

You can also check the service is running

Add your first job

Best way to check you have Zuul started and running properly, is to add your first job.

Edit layout.yaml (You can check where it’s configured in zuul.conf under [zuul] section. In my setup it’s at /etc/zuul/config/layout.yaml).

Note: I would set the time to something close to your current time, so you can see the job is actually starts running.

Good, now reload zuul-server

Now you can see in the server.log that your job is in ‘periodic’ pipeline.

Common Zuul Failures

Authentication failed

It means Zuul couldn’t connect to the Gerrit server. There are variety of reasons as to why you might see it – connectivity issues, bad key, firewall, etc.

In your zuul.conf file, you specified under ‘[gerrit]’ section, the variables ‘port’, ‘server’, ‘sshkey’ and ‘user’. Use the value of these variables to test the connection.

You should see something similar to the following output (depends on your Gerrit server)

status.json: service is unavailable

If, when accessing the Zuul web status page you see “status.json: service is unavailable”, then you are in the right place.

To confirm you have an issue, you can also have a look in your zuul httpd log (in my environment it’s /var/log/httpd/zuul-error.log)

It means httpd can’t make network connections and this usually because SElinux isn’t configured properly.

Try to permit it with setsebool and restart httpd

Gearman: connection refused

If when trying to test the connection between Jenkins and Zuul you receive ‘test failed’ or ‘unable to connect’ then you need to change your Zuul configuration.

This is usually happens when you have under [gearman_server] section the ip address “127.0.0.1” instead of “0.0.0.0” as the value of ‘listen_address”

You should have similar section in your configuration:

Update: the user wangxf shared with us some of the issues he found and solved. Adding them here (thanks wangxf!):

“Host key verification failed. fatal: Could not read from remote repository”

If you get the following failure

You can fix them with the following steps:

“Fail to run git merge -s resolve FETCH_HEAD”

If the following command fails:  git merge -s resolve FETCH_HEAD to set your account’s default identity.

Add the following values in merger section:

/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py raises URLError

If you get the following exception:

Add the following value under gerrit section

9 Comments

  1. First of all, I’ve been reading your posts about zuul and they’re awesome, I’ve using this in particular for a solution to integrate several tools in a CI/CD solution. I noticed a couple of steps that are missing in this post.

    The first one is related with the script that starts the daemon in ubuntu /etc/init.d/zuul-server,

    1. The missing ‘#’ in the first line
    2. The closing bracket(}) in do_graceful_stop() function

    And the last one is related with the permissions on that file:

    # chmod +x /etc/init.d/zuul-server

    Thanks again for sharing this information

    • bregman

      October 1, 2016 at 10:42 am

      Thank you for letting me know 🙂
      I made some fixes.

      Appreciate it!

      • Thanks with your share!

        • I share with some problem and how to fixed it.
          issue:
          stderr: ‘Host key verification failed.
          fatal: Could not read from remote repository.
          return repo
          UnboundLocalError: local variable ‘repo’ referenced before assignment
          fix:
          1. cp id_rsa id_ras.pub known_hosts to your ssh key folder for zuul.
          2. chown -R zuul:zuul id_rsa
          3. chown -R zuul:zuul id_rsa.pub
          4. chown -R zuul:zuul known_hosts

        • issue:
          cmdline: git merge -s resolve FETCH_HEAD
          to set your account’s default identity.
          Omit –global to set the identity only in this repository.
          fix:
          add two values under merger section, like this:

          zuul_url = 127.0.0.1
          git_user_email=replaceme@awcloud.com
          git_user_name=replaceme

          • issue:
            File “/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py”, line 391, in getInfoRefs
            raise URLError(err)
            URLError:

            fix:
            add baseurl under gerrit section:
            ….
            baseurl=http://gerrit.corp.awcloud.com

        • bregman

          December 3, 2016 at 5:58 pm

          Thank you! I added your replies to the post. Much appreciated!

  2. How can this be accomplished in Ubuntu and without Ansible?

Leave a Reply

Your email address will not be published.

*

© 2017 Arie Bregman

Theme by Anders NorenUp ↑