Gerard Braad's blog

Presentation: Atomic, FUDCon Phnom Penh

2016-11-08T00:00:00+08:00

This is the presentation as given at the FUDCon Phnom Penh. In short it explains what Project Atomic is, how to use Atomic Host, and some advanced use-cases; deployment of a composed application, and customization of the Atomic images. The presentation relied heavily on showing usage in the form of a demo, therefore please see the links to the other relevant articles on this blog.

Presentation

Your browser does not support iframes.

Feedback

If you have any suggestion, please discuss below or send me an email.

Setup OpenStack Keystone for testing purposes

2016-10-27T00:00:00+08:00

OpenStack can be seen as a set of projects which combined into a configuration can deliver a management solution for different use-cases. However, several of these projects can also exist on their own to provide a certain functionality, such as Keystone, Ironic, etc. Some of these I will describe in articles on this blog, but let's start with one of the most important one's that deal with Authenticatioin, namely Keystone.

Keystone

Keystone is one of the core projects of OpenStack and is responsible for providing Identity. It offersauthentication, authorization and service discovery mechanisms via HTTP. Especially this last property, offering the API to be accessed by HTTP as a ReSTful interface offers a lot of opportunities for re-use. More about the Keystone project can be found on the project homepage.

Containerized

To simplify the deployment process, I have containerized Keystone. The image can be found at GitLab and the Docker Registry.

`Dockerfile`

The image is based on a standard CentOS 7 cloud image. To install Keystone, I used the packaged version from the RDO project. After an update, it install the repository information pointing to mitaka, and then it install the Keystone service, some useful utils to use with OpenStack and the SELinux configuration files.

RUN yum update -y && \
    yum install -y centos-release-openstack-mitaka && \
    yum install -y openstack-keystone openstack-utils openstack-selinux && \
    yum clean all

After this we expose the ports that Keystone uses for interaction.

EXPOSE 5000
EXPOSE 35357

As the CMD it will run a simple initialization script. It takes care to setup an admin token and use a MySQL connection. It then starts keystone-manage to provision the database with the schemas. And finally, it will use keystone-all to run the services on the exposed ports.

#!/bin/sh
ADMIN_TOKEN=${ADMIN_TOKEN-`openssl rand -hex 10`}
openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $ADMIN_TOKEN
openstack-config --set /etc/keystone/keystone.conf DEFAULT use_stderr True
openstack-config --set /etc/keystone/keystone.conf database connection ${CONNECTION-mysql://keystone:password@dbhost/keystone}
su keystone -s /bin/sh -c "keystone-manage db_sync"

/usr/bin/keystone-all

In the rest of the article I will describe how to use this image to setup the test environment. The first component you will need to use Keystone is the database to store the credentials and other relevant information.

Setup database

As you saw in the Dockerfile, we specified a MySQL connection. To make it easy to deploy, we will also be using a container for this database server. I prefer to use MariaDB, as this is also what normally would come with CentOS. For this article I will be using MariaDB version 10.1.

To start the container perform the following command:

$ docker run -d --name keystone-database -e MYSQL_ROOT_PASSWORD=secrete mariadb:10.1

This will pull the container and start a named instance as keystone-database. The root password to configure the database is set using the environment variable to secrete.

Using this password and container name, we will create a database for Keystone and grant priviliges:

$ docker exec keystone-database mysql -psecrete -e "create database keystone;"
$ docker exec keystone-database mysql -psecrete -e "GRANT ALL ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'password';"
$ docker exec keystone-database mysql -psecrete -e "GRANT ALL ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'password';"
$ docker exec keystone-database mysql -psecrete -e "flush privileges;"

After this, all the setting for the database are done and we can just leave it running without further configuration.

Generate token for API usage

Just like MariaDB, Keystone can use a token to perform the initial configuration. It is preferred to make this admin token something that is random and not easily guessable. A random token can be generated for instance with:

$ export TOKEN=$(openssl rand -hex 10)

In the rest of the article, this token will be used.

Start container

After the database has been setup, and a token has been decided, we can start the Keystone services.

$ docker run -d --link keystone-database:dbhost -e ADMIN_TOKEN=$TOKEN -p 5000:5000 --name keystone-server gbraad/openstack-keystone:mitaka

This will link the keystone container to the database container. If all goes well, the service will now be available on the exposed ports. The container instance will be identified by the name keystone-server.

As mentioned earlier, when the container gets started, it will provision the database with the schemas that are needed.

In the following segment we will configure Keystone.

Create service entry

To finalize the Keystone configuration, we will insert the service and endpoints for the Identity service.

$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ service-create --name=keystone --type=identity --description="Keystone Identity Service"
$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ endpoint-create --service keystone --publicurl 'http://localhost:5000/v2.0' --adminurl 'http://localhost:35357/v2.0' --internalurl 'http://localhost:5000/v2.0'

Note that these command are performed inside the keystone container.

Create admin user

To allow access to keystone without using the admin token, we will need to create a user. The following commands will create a user named admin with the password password and add this to a tenant called admin.

$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ user-create --name admin --pass password
$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ role-create --name admin
$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ tenant-create --name admin
$ docker exec keystone-server keystone --os-token $TOKEN --os-endpoint http://localhost:35357/v2.0/ user-role-add --user admin --role admin --tenant admin

More detailed setups are possible, and for this I will refer you to the Keystone documentation.

Getting a user token

To verify Keystone works, we will retrieve a token. First you need to know the IP address that has been assigned to your container.

$ docker inspect --format="{{.NetworkSettings.IPAddress}}" keystone-server
172.17.0.6

This can be helpful to communicate between different containers on the same Docker network. You could create a new container, or pull my OpenStack client container.

Using the OpenStack client

Using the OpenStack client you can retrieve a token using the following configuration:

keystonerc

OS_IDENTITY_API_VERSION=3
OS_AUTH_URL=http://172.17.0.6:5000/v3
OS_USERNAME=admin
OS_PASSWORD=password
OS_PROJECT_NAME=admin
OS_USER_DOMAIN_NAME=Default
OS_PROJECT_DOMAIN_NAME=Default

When you use the following commands you will be given a token:

$ source keystonerc
$ openstack token issue

Using cURL

Since Keystone uses a HTTP interface, you can also retrieve the token using the cURL command. When using the -i parameter, cURL will return all the headers from the response. This will include the X-SUBJECT-TOKEN we want.

$ curl -i -H "Content-Type: application/json" -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "name": "Default" },
          "password": "password"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "name": "Default" }
      }
    }
  }
}' http://172.17.0.6:5000/v3/auth/tokens

A response will follow. You need to set the value of X-Subject-Token to OS_TOKEN, or use in cURL as the X-AUTH-TOKEN header. More about this can be found in the Keystone documentation: API Examples using cURL.

How this was used for development

For the Commissaire project I added Keystone functionality using two methods; using password and token. The above described container was used to allow easy deployment and testing.

As I decided not to introduce unnecessary libraries as dependencies, the communication from Commissaire happens over the HTTP ReST interface. To authenticate using a basic authorization, using the password method, we only need to construct a simple JSON object:

{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "name": "Default" },
          "password": "password"
        }
      }
    },
    "scope": {
      "project": {
        "name": "admin",
        "domain": { "name": "Default" }
      }
    }
  }
}

In code we described this with:

        headers = {'Content-Type': 'application/json'}
        body = {'auth': {'identity': {}}}
        ident = body['auth']['identity']

        ident['methods'] = ['password']
        ident['password'] = {'user': {
            'name': user,
            'password': passwd,
            'domain': {'name': self.domain}}}

To perform the authentication, we send a request:

        response = requests.post(
            self.url,
            data=json.dumps(body),
            headers=headers)

to the endpoint exposed at: http://172.17.0.6:5000/v3/auth/tokens. When the response includes the header X-Subject-Token the authentication succeeded, else we failed and send a 403 as status code.

        if 'X-Subject-Token' in response.headers:
            return True

        # Forbid by default
        return False

The Implementation for authentication using a token is quite similar, instead we will take the X-Auth-Token from the request we received and verify this against Keystone using the following request:

{
    "auth": {
        "identity": {
            "methods": [
                "token"
            ],
            "token": {
                "id": "faa166abf235430e81c9fa12ad248533"
            }
        }
    }
}

Just like in the previous example, the endpoint is: http://172.17.0.6:5000/v3/auth/tokens. If the authentication is succesful the response will contain the X-Subject-Token header.

Conclusion

Keystone is powerful service to provide authentication for your infrastructure. As you can see from this example, using a containerized (composed) environment, it is easy to get a basic setup running. And using the ReST API it is also very simple to handle authentication outside of your own application. In future articles more advanced scenarios will be given.

At the moment, password authentication using Keystone and Commissaire is possible and hopefully soon we will also have integrated the token based authentication.

If you have comments or suggestions, please leave them below or consider sending a message to me on Twitter: @gbraad.

Flatpak; the road to CI/CD for desktop applications?

2016-10-22T00:00:00+08:00

In this presentation I will introduce Flatpak and how it changes the software distribution model for Linux. In short it will explain the negatives of using packages, how Flatpak solves this, and how to create your own applications and distribute them for use with Flatpak. This presentation was given at the GNOME 3.22 release party, organized by the Beijing GNOME User Group.

Presentation

Your browser does not support iframes.

Feedback

If you have any suggestion, please discuss below or send me an email.

Note: the original presentation can be found at: GitLab: BJGUG Flatpak. This is based on an earlier presentation about Software Dsitribution for a new era

"Task management and Personal Kanban: how I use GitLab Issues"

2016-10-09T00:00:00+08:00

Working on personal and work-related tasks can be overwhelming, especially when you lose sight of what actually needs to be done. What makes it more difficult is that you are constantly interrupted. I have tried many different things, and apps, but nothing really worked. Do not try to make your process of working to fit the tool, but make sure you have a tool that fits the way you work.

Kanban

Kanban (or 看板) originated as a scheduling system to improving manufacturing efficiency. The word literally means signboard in Japanese, and in Chinese it can be read as 'Look board'. It emphasizes on visualization of tasks that need to be done, are being worked on, and have finished.

In software development you often see people use Trello or other kanban-like boards to visualize work that is on their 'backlog', 'doing', 'done'. Although these tools work quite well, it did not fit my workflow or where I do most of my work.

GitLab issues

Quite recently, GitLab released a version of their sourcecode management tool that allows to visualize issues on a board. And I started to evaluate it, as GitLab is part of my workflow in general. I host many of my private repositories there for backup purposes, use it to publish my resume with the CI runners, etc. And I can tell you, that the issue boards was exactly what I was looking for.

Below is a screenshot of what it looks like:

Note: the image shown here is from an early iteration and still has too many items in 'doing'. This actually made me realize where my time was spent.

How to set-up

All you need to do is create a private repository. In my case, I created a 'personal' project. All the issues you create, will only be visible to you. After you have done this, you can create a board from the Issues. Just start with the defaults first and customize it along the way... This made it work best for me.

Workflow

When you create an issue, just tag it accordingly. I use it to group tasks for different topics or things I work on. Now when you open the board, you will see your issue is on the 'backlog'. From there you can drag it to a swimlane. I have several that helps to organize tasks, such as 'blog' and 'urgent'. When you drag, it will assign the label to the issue according to the name of the swimlane.

Offline use?

There is, however, a small issue with the setup. It only allows me to work online. In the end I noticed this is actually not a big problem.

Single-tasking

Kanban helps you to visualize your tasks at hand, but it still requires discipline to actually make it work. Therefore, you need to understand that 'multi-tasking' does not exist! I recently read 'Singletasking: Get More Done-One Thing at a Time', and it help me a lot to realize that I was going the right direction with my personal kanban. However, I was still trying to do too many things at once. Mostly due to interruptions. This led me to appreciate the offline problem I have.

If I have an interruption, or some other urgent matter, I would first record it on my phone in a file called 'reminder', which lives in the same personal repository. This file is synced using git, so it travels with me and it allows me to use it as a general reminder/todo record keeping, and decide later what to do with the entries.

Tools

On my desktop I only need to use a browser to keep track of tasks being worked on. Even the reminder file can be opened from here.

On my phone I use the following apps:

Writeily Pro, SGit, and Labcoat. Using SGit I sync several important repositories to my phone, such as my 'personal', 'knowledge-base', and some private specific projects for training/teaching, etc.

Writeily Pro opens the SGit repos folder as it's document folder: /sdcard/Android/data/me.sheimi.sgit/files/repo in my case. Writely's usage was not without problems for me. Especially when dealing with some 'larger' files.

Using Labcoat I am able to create and change basic information of an issue, such as closing it and adding comments. But I usually do all interactions online now, and else use my offline reminder file. I only use Labcoat only in rare cases. Probably also because the tool misses some of the more wanted features as labelling and tagging issues.

I haven't found a good way to use the command line yet. I did start with a small client, but so far it only reads and doesn't filter anything. Hope I can allocate more time to deal with this at some point. At the moment I do not really need it.

Conclusion

Task management is not something you learn by just reading a book or keeping a todo-list inside an application. It really is a discipline. And it depends on you, how you will deal with it. This article does not go into a lot of detail on my own process, as this is a personal quest. Just experiment and see what works best for you.

I have read several books, and in the section 'More information' below, you can find these resources. It involves in keeping your tasks visualized and limiting the amount you work on at the same time. It is up to you how to balance private and work.

I also learned that to keep things organized, I started to keep a public knowledge base. This way I am better able to keep knowledge available and at hand. Before, I would keep it in an earlier version of a reminder file and this cluttered everything and actually not made me remember things. I am still working on writing things done in a more general way, written towards possible others who will read it. But this is a work-in-progress, as part of my personal continous improvement.

I will certainly refine my process over time, and might even change it again completely. As long as it works for me. I'll keep you updated and might write related articles about time management and knowledge techniques I have used over the years. One I still love the most is mind-mapping... and I have applied it to organizing work before. It is a great way to organize thoughts.

More information

I suggest to read more about personal kanban.

Personal Kanban 101
Knowledge-base: Personal kanban
Knowledge-base: Singletasking
Presentation of a friend's experience with Personal Kanban

Dear recruiter

2016-10-08T00:00:00+08:00

As an engineer I love to work and talk about software and software development. Looking for a new opportunity is not an easy task. There is a story behind every choice people make, and in this article I will explain my vision and situation. This is certainly not an employer review, but rather a reflection on past and maybe future choices.

Job-hopping

Just too often I have to explain why my resume looks like I have been hopping between jobs. Part of this is true and the other has a story behind it. I love a challenge and also do not mind when a company hires me for my knowledge, but I also prefer to be employed for a longer period and be able to work on the stuff that matters. Work needs to be meaningful and satisfying...

Netherlands

In the Netherlands I worked for an employer which provides consultancy. This often involved on-site work. These clients, and consultants, like to work as close as possible, as if you are employed by them. However, they keep the flexibility to end a contract when the project you are assigned on ends or has no budget. These assignments are usually short-term as they need hands or knowledge to finish it. If you know 'the Mythical Man-month', you know that introducing engineers late in a project often does not work as expected. Although, wome of my assignments have dealt with longer term solutions, such as software rewrite, knowledge hiring, etc. I loved my work as certainly felt very appreciated.

China

After this, due to circumstances and a 'lost bet' with my wife, we moved to China. Being employed in a foreign country involves sometimes with work of an imperfect nature.

Projects

One of those is of having project based assignments, but unlike before I would be directly hired by the employer. Which means that the contract will not last longer than the allotted budget for the project. These project are challenging as they often deal with a new technology or a changed environment for the employer. I have been on at least two projects here in China and have learned a lot from dealing with an inter-cultural environment and how to assist with adopting a more 'Western' approach to management (agile methodology). Re-assignments were discussed, but often involved a big change in work or other less than ideal propositions. If in that case there is no fit anymore, it is best to move on. This choice would be in the best of interest of both the employer and employee.

Startups

The other situation is when dealing with startups. Startups often require an employee that is able to be all-round, as this allows to keep a minimal headcount. Again, the work is mostly challenging and benefits a lot from multi-disciplined engineers. I have worked for several startups, and most of the cases it is very enjoyable work. The experience has led to some good success stories, although some of them have also disappeared. The work includes a lot of uncertainty. While some startups might look like they have matured, they can still hide unpleasant surprises when dealing with management or decision-making. This can have an affect on how the atmosphere will be at work. It can be a happy place one-day... and a poisonous one the other day.

Recruiters

I understand an assessment of a possible match is difficult. There are probably many applicants, long resumes to wade through. Automated system exist to deal with analysing a 'match' based on the resume provided. And often, theese systems fail to understand what people really meant to say or provide a picture of the working history. And I understand this is somehow needed.

In my situation, the resume often happens to be sent to the local Chinese branch. And here, my resume or application, disappears into an abyss; often the recruiter does not reach out.

Recommendations

If you want to be employed into a group you like to work with, get noticed. Work with them, and show your passion. This is why I work on Open Source projects. It is what I like most, as it involves technology, but also an essential soft skill (or perhaps a thick skin). Hopefully this can lead to a recommendation. As this is still the best way to get hired.

What I want

At the moment I am looking for a new opportunity. As you have read, I am a software engineer, but have worked in several different roles related to it. Multi-disciplinary and professional. Knows a lot about Linux, Open Source and China. If you want to know more about me, please have a look at:

Resume
LinkedIn and the recommendations given.
Homepage and my blog

What I want is a great and challenging environment. This means that work is meaningful and provides satisfaction. It should provide growth potential and understand that people need to focus on continuous improvement. Agile taught me one thing, we’re never really good enough. And third, and most important is trust. Trust in colleagues, fairness and respect... without trust you can not have a great work environment. But you also need trust that people dare to admit when they are not experts at something yet, but you allow them to develop themselves.

Hope to hear from you. If you have a comment, please do not hesitate and leave one below.

"Replace scripts with Ansible: package installation"

2016-10-03T00:00:00+08:00

Installing and configuring software on one machine, for instance your own developer's environment, is an 'easy' task. But how about your team? Or production? Everything is documented and you only probably install some of the components manually, or better, you automate this using a set of shell scripts. The environment is now reproducible. Everything is fine.

Running a set of scripts is not a bad idea, but often what lacks is the maintenance on them or just the magic contained in them. I do not have to make the case for general configuration management, 'configuration as code' or even #DevOps I hope? In this article I will show why I use Ansible for almost all my scripts. Even small one of.

Environment

Although I have a preference for the operating system and distribution I use, I have to deal with many different versions and variants. On my workstation I prefer to use Fedora, but at work we deploy on CentOS. Some of my colleagues tend to prefer Ubuntu (latest), and customers are stuck on LTS releases. A very common situation. In one week, I had to deal with: Alpine, CentOS, Fedora, Ubuntu, (Open)SUSE, Debian, Windows, etc. This is of course an extreme case.

You would prefer to use a single environment for both development and production. When the production environment is fixed, the solution is easier. You will likely use a tool like Vagrant to stand up an environment, or some other virtualization soltuion. But what if you develop for an environment where both Ubuntu and CentOS are an option? In my case this happens a lot because of work for OpenStack, or just general Linux tool development.

Example

Below I will discuss a small example of dealing with package installation. Let's consider you have to use Ubuntu and Fedora. This means that you already have to deal with two different package managers, oh wait, three; apt on Ubuntu and yum or dnf on Fedora.

Bash script

As a crude solution, you could do the following:

#!/bin/sh
APTPKGS="git tmux zsh mc stow python-psutil"
RPMPKGS="git tmux zsh mc stow python-psutil"

# Crude multi-os installation option
if [ -x "/usr/bin/apt-get" ]
then
   sudo apt-get install -y $APTPKGS
elif [ -x "/usr/bin/dnf" ]
then
   sudo dnf install -y $RPMPKGS
elif [ -x "/usr/bin/yum" ]
then
   sudo yum install -y $RPMPKGS
fi

This basically checks if a certain executable is available, and will use that with a defined list of packages to install. If both yum and dnf is installed, it will in this case use dnf as it resolves first. This is likely what you want, but it is hidden logic. But what if you have installed yum on Ubuntu? Also, because of this case, apt-get will resolve first. But again, this is undocumented in this script. But this knowledge should not be needed.

Note: in this case, the packages are the same for both platforms, but this is not a guarantee. Therefore I use a variable for this.

Ansible playbook

Ansible solves this problem by offering a general package module. You can use yum and apt specific options, but I prefer to use package instead. Below is a playbook that has the same behaviour as the above mentioned script:

  tasks:
  - name: Install list of required packages
    package: name={{ item }} state=installed
    become: yes
    become_method: sudo
    with_items:
    - git
    - tmux
    - zsh
    - stow
    - python-psutil
    - mc

Because of the task name, it becomes clear what the intention is of the commands that follow. In this case, this will use the package manager based on the OS Family that Ansible will find. This means we do not need to have the knowledge of what to pick first. There is however a problem. What if the packages have different names?

Conditionals

For this Ansible offers conditionals which you can use with when. Below is an example that will install the development tools on Ubuntu or Fedora, based on the ansible_distribution:

#!/usr/bin/env ansible-playbook
---
- hosts: localhost
  #remote_user: root
  become_method: sudo

  tasks:
  - name: Install Git
    package: name=git state=installed
    become: yes

  - name: install the 'Development tools' package group
    package: name="@Development tools" state=present
    when:  ansible_distribution == 'CentOS' or
           ansible_distribution == 'Red Hat Enterprise Linux' or
           ansible_distribution == 'Fedora'
    become: yes

  - name: Install the 'build-essential' meta package
    package: name="build-essential" state=present
    when: ansible_distribution == 'Debian' or
          ansible_distribution == 'Ubuntu'
    become: yes

You can also use ansible_os_family == "RedHat" to be less specific about the distribution you are on.

Include

You can combine conditionals with almost anything in Ansible, for instance to include or not another playbook. Below is a simple solution of this when you want to split the install instructions into separate files.

- include: install-centos.yml
  when: ansible_distribution == "CentOS"

- include: install-ubuntu.yml
  when: ansible_distribution == "Ubuntu"

Conclusion

With Ansible it is possible to create complex instructions that can be more maintainable than when using just scripts. Of course, just using Ansible will not act as a silver bullet. But because a playbook is more readable, the process of refactoring is easier.

In future articles I will talk about other aspects where the move to Ansible helped me. However, the bootstrap process remains... I wish there was a way to Automate everything ;-)

Hope this article has been helpful to you. If so, please consider tweeting about it. Or leave a comment if you have suggestions...

Giving to your favourite content creators

2016-09-29T00:00:00+08:00

It is hard to remember the time I really sat down in front of the TV, waiting for a programme to show up, and really enjoy it. Mostly it was accompanied with plenty of interruptions. It might have been Star Trek on the public channels that I enjoyed the most. But even Discovery Channel had many interruptions, some were funny, or just slightly annoying, especially when you have the TV on as background sound. With the advent of YouTube, and other User Generated Content websites, or even streaming websites, it is almost unimaginable there used to be a time without being able to watch your favorites.

Some might not know, but I help(ed) a lot with music remixing and artwork. Too bad I do not have the time now, but this is so much fun. Content creation is probably the best people can do as it gives a lot of satisfaction. I consider the teaching I do as a form of content creation. I produced a lot of documents, notes, etc, and people enjoy the way I explain stuff. Even writing code is a creative process:

foreach step in I.take():
  step.move(DIRECTION_TOMORROW)

But remixing music, or posting artwork online, does not pay the bills. Unless of course you are Tiesto, etc. And asking for a donation has not been my thing, although I have received some. I am very thankful for this!

And this is why I give to the content creators that create good stuff. I love the cardgame Magic: the Gathering, and especially some of the youtube channels for this. Patreon is a great idea... and if people create content you read often, consider giving back. Even 1 USD is already good way to show you care. It can buy someone a cup of coffee... and the more people who do, the easier someone can take some time off to create the content you love!

One of the projects I really want to work on again is Gauth. it has been in a neglect, but I can't reserve enough time to do a rewrite... I ask you, consider to become a Patron of my work and hopefully I can improve by giving Gauth a new future, allow me to create more content on my blog... and maybe in the future even more.

And thank you to all who have donated in the past!

Run an example application on OpenShift

2016-09-29T00:00:00+08:00

In a previous article I have written on how easy it is to stand up a test environment of OpenShift. In this article I will describe an example application from the sourcecode to the created image and how this gets deployed. The steps are explained using manual steps, and how OpenShift does it all automated. You will notice, at no point do you have to write a Dockerfile.

Preparation

For this article it is not necessary to have a working test environment, however it does make things clearer. I would suggest you to use OpenShift Origin v1.3 on CentOS 7. Although my previous article showed how to get it up and running on Fedora 24, I experienced an issue with deployment not succeeding*. The steps in the deployment article can be performed by replacing dnf with yum.

Description of the example

The OpenShift project publishes several test applications on GitHub, of which one is a very simple Ruby applica8080. Please, have a look at: http://github.com/openshift/ruby-ex

You will see it consists of four files:

Gemfile
Gemfile.local
config.ru
README.md

The application itself is only described in config.ru and the needed dependencies are in Gemfile.

Dependencies

To make the application work, we first need:

$ gem install bundler

This will install bundler that can install dependencies as described in the Gemfile. This file describes:

source 'https://rubygems.org'
gem 'rack'
gem 'puma'

The first line says source which points to a gem repository, and each line starting wih gem are bundled packages containing libraries for use in your project. To install all of the required gems (dependencies) from the specified sources:

$ bundle install

The file Gemfile.lock is a snapshot of the Gemfile and is used internally.

config.ru

The application is specified in the file called config.ru. If you open the file you will see it contains route mappings, lines starting with map, for three urls:

/health
/lobster
/

`/health`

This is a commonly used to provide a simple health-check for applications that are automatically deployed. It allows to quickly test if the application got deployed. In projects I worked on, we also did quick dependency checks, such as a configuration file exists, or another needed endpoint is available. In this application it will respond with a HTTP status code 200 and returns 1 as value.

`/lobster`

This is a test provided by rack. It shows an ASCII-art lobster. By adding a variable to the URL querystring ?flip=left the direction can be changed.

`/`

This is the mapping to a bare route. It shows a greeting message on how to use the application using OpenShift to trigger automated builds.

Rackup

Rack is an interface for using Ruby and Ruby frameworks with webservers. It provides an application called rackup to start the application:

$ bundle exec rackup -p 8080 config.ru

Using this command the webserver will bind to port 8080, according to the description in the config.ru file. To see what the mappings do, open:

Use the example with OpenShift

Deploying an application on OpenShift from source is very simple. A single command can do this. First have a look

$ oc new-app openshift/ruby-20-centos7~https://github.com/[username]/ruby-ex

But before we do, I will explain what this command does. Oversimplified OpenShift does two things:

Build
Deploy

Note: If you want to perform the command, go ahead. Please fork the repository and change the [username] in this command.

Build: source to image

OpenShift runs container images which are in the Docker format. It will run the CMD instruction for this. So, how does OpenShift know what to run? Convention. Most frameworks have a standard way of doing things, and this is as you noticed also the case with the Ruby example. The creation of the image happens with a tool called source-to-image (S2I).

Source-to-Image (S2I) is a toolkit and workflow for building reproducible Docker images from source code. It uses a base image, and will layer the application on top, configures the runn command, which then results in a containter image for use.

$ s2i build https://github.com/[username]/ruby-ex openshift/ruby-20-centos7 ruby-ex

base image

The base image here is openshift/ruby-20-centos7. The source of this image can be found at the following GitHub repository: s2i-ruby-container

If you look at the Dockerfile source, you will see Software Collections is used to install a specific Ruby version. In this case version 2.0. Software collections solves one of the biggest complaints of using CentOS (or RHEL) as a basis as part of your delivery. It allows you to use multiple versions of software on the same system, without affecting system-wide installed packages.

The image also describes a label io.openshift.expose-services="8080:http" which inidcate that the application on port 8080 will be exposed as HTTP traffic. This also means the container does not need root privileges as the port assignment is above 1024. The application itself will be installed into the folder: /opt/app-root/src.

Running this container can be done with:

$ docker run -p 8080:8080 ruby-ex

[1] Puma starting in cluster mode...
[1] * Version 3.4.0 (ruby 2.0.0-p645), codename: Owl Bowl Brawl
[1] * Min threads: 0, max threads: 16
[1] * Environment: production
[1] * Process workers: 1
[1] * Phased restart available
[1] * Listening on tcp://0.0.0.0:8080
[1] Use Ctrl-C to stop
[1] - Worker 0 (pid: 32) booted, phase: 0

Open the links as previously stated will yield the same results.

$ curl http://localhost:8080/health

The build process can be as simple as a copy for static content, to compiling Java or C/C++ code. For the purpose of this article I will not explain more about the S2I process, but this will certainly be explained in future articles.

New application

If we now look at the previous command again:

$ oc new-app openshift/ruby-20-centos7~https://github.com/[username]/ruby-ex

you can clear see the structure. The first element openshift/ruby-20-centos7 describes the S2I container image for Ruby as hosted at the Docker hub. The second part is the source code path pointing to a git repository.

Please try the command now... OpenShift will create containers for each of the stages used: build, deploy and the final running container. You can check the containers using the command:

$ oc get pod

NAME               READY     STATUS         RESTARTS   AGE
ruby-ex-1-build    0/1       Completed      0          1m

Build stage

If you create this new application, a new container named ruby-ex-1-build. What happened is that the Source-to-image container got pulled which uses the base image and layers the source code on top.

To see what happened, as with the previous command, you can see the build configuration:

$ oc logs bc/ruby-ex

Cloning "https://github.com/gbraad/ruby-ex" ...
        Commit: f63d076b602441ebd65fd0749c5c58ea4bafaf90 (Merge pull request #2 from mfojtik/add-puma)
        Author: Michal Fojtik <mi@mifo.sk>
        Date:   Thu Jun 30 10:47:53 2016 +0200
---> Installing application source ...
---> Building your Ruby application from source ...
---> Running 'bundle install --deployment' ...
Fetching gem metadata from https://rubygems.org/...............
Installing puma (3.4.0)
Installing rack (1.6.4)
Using bundler (1.3.5)
Cannot write a changed lockfile while frozen.
Your bundle is complete!
It was installed into ./bundle
---> Cleaning up unused ruby gems ...
Pushing image 172.30.108.129:5000/myproject/ruby-ex:latest ...
Pushed 0/10 layers, 10% complete
Pushed 1/10 layers, 34% complete
Pushed 2/10 layers, 49% complete
Pushed 3/10 layers, 50% complete
Pushed 4/10 layers, 50% complete
Pushed 5/10 layers, 50% complete
Pushed 6/10 layers, 61% complete
Pushed 7/10 layers, 71% complete
Pushed 8/10 layers, 88% complete
Pushed 9/10 layers, 99% complete
Pushed 10/10 layers, 100% complete
Push successful

The difference is that the resulting image will be placed in the myproject namespace, and pushed to the local repository.

Deployment stage

After the image has been composed, OpenShift will run the container image on the scheduled node. What happens here can be checked with:

$ oc get pod

NAME              READY     STATUS      RESTARTS   AGE
ruby-ex-1-an801   1/1       Running     0          26s
ruby-ex-1-build   0/1       Completed   0          1m

This means that the build succeeded, the image got deployed and now runs in the a container identified with ruby-ex-1-an801. Note: The container ruby-ex-1-deploy is not shown here as only the logs are of importance.

The deployment configuration logs can be shown with:

$ oc logs dc/ruby-ex

[1] Puma starting in cluster mode...
[1] * Version 3.4.0 (ruby 2.0.0-p645), codename: Owl Bowl Brawl
[1] * Min threads: 0, max threads: 16
[1] * Environment: production
[1] * Process workers: 2
[1] * Phased restart available
[1] * Listening on tcp://0.0.0.0:8080
[1] Use Ctrl-C to stop
[1] - Worker 0 (pid: 32) booted, phase: 0
[1] - Worker 1 (pid: 35) booted, phase: 0

Events

To see the flow of execution, you can have a look at:

$ oc get events

This can be helpful if an error occured.

Verify

Now that the application has been deployed on OpenShift, we need to look up the IP address that has been assigned. For this we use:

$ oc get svc

NAME      CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
ruby-ex   172.30.91.160   <none>        8080/TCP   21h

Now we can open the application as http://172.30.91.160:8080/

Conclusion

OpenShift allows you to run prebuilt images or applications based on source. The Source-to-image tooling makes it possible to create reproducible images for deployment of applications based on source. This tool itself is very helpful and is certainly something I will be using, even outside the use of OpenShift. There is no need to create or modify a Dockerfile, which means that the developer can focus on the development process.

If you want to know more about the automated builds, please have a look at the README of the Ruby example. In future articles more detailed descriptions about these topics will certainly be given. I hope this has been helpful. Please consider leaving feedback or tweet this article.

Setup Docker storage to use LVM thin pool

2016-09-28T00:00:00+08:00

If you install Docker on a new Fedora or CentOS system, it is very likely that you use devicemapper. Especially in the case of Fedora cloud images, no special configuration is done to the image. While Atomic images come pre-configured with a dedicated pool. Using devicemapper with loopback can lead to unpredictable behaviour, and while OverlayFS is a nice replacement, you will not be able to use SELinux at the moment. In this short article I will show how to setup a pool for storing the Docker images.

Preparation

For this article I will be using a Fedora 24 installation on an OpenStack cloud provider*. It is a standard Cloud image, which means the root is configured as 'ext4'. I will be attaching a storage volume to the instance. Just like using Virtual Manager, the disk will be identified as /dev/vdb.

First you need to stop the Docker process and remove the existing location. This means you will loose the images, but if they are important, you can either export or push them somewhere else.

$ systemctl stop docker
$ rm -rf /var/lib/docker

After this we will create a basic LVM setup which will use the whole storage volume.

$ pvcreate /dev/vdb
$ vgcreate docker_vol /dev/vdb

We know have a volumegroup named docker_vol.

Setup Docker storage

Fedora comes with a tool that makes it easy to setup the storage for Docker, called docker-storage-setup. The configuration is done with a file, and in our case it needs to contain the identification of the volume group to use:

$ vi /etc/sysconfig/docker-storage-setup

VG="docker_vol"

Now you can run:

$ docker-storage-setup

This will create the filesystem and configures Docker to use the storage pool. After this successfully finishes, it has configured the pool to use the xfs filesystem.

Verify

To verify these changes, we will start Docker and run a basic image.

$ systemctl start docker
$ docker info

Storage Driver: devicemapper
 Pool Name: docker_vol-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 37.75 MB
 Data Space Total: 21.45 GB
 Data Space Available: 21.41 GB
 Metadata Space Used: 53.25 kB
 Metadata Space Total: 54.53 MB
 Metadata Space Available: 54.47 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.122 (2016-04-09)

$ docker pull busybox
$ docker run -it --rm busybox
/ #

Conclusion

Configuration of storage has been really simplified and shouldn't be a reason not to do this. However, still to many people are not aware of the issues with devicemapper and loopback. Having to assign additional software to use Docker can also be a reason why people do not consider doing this, but even OverlayFS is not perfect. Using overlay with SELinux will be possible in the future, and hopefully soon these steps will also not be needed. In any case, the steps involved are simple and if you use Docker on Fedora, needed!

More information

Friends Don't Let Friends Run Docker on Loopback in Production
Setting up storage for Project Atomic

Deploy an OpenShift test cluster

2016-09-27T00:00:00+08:00

In my previous article I described how I used an Ansible playbook and a few roles to stand-up a Kubernetes test environment. In that article I mentioned that to deploy a production-ready environment some work more was required. Luckily, it is now very easy to stand up a production and enterprise-ready container platform for hosting applications, called OpenShift. Through the years OpenShift has undergone a lot of changes, and the latest version Origin v1.3 is very different from the original version. OpenShift sets up a complete Kubernetes environment and with a set of tools it can take care of the whole application lifecycle, from source to deployment. In this article I will give an introduction to setting up a test environment for a developer.

Setup machine

I will setup the environment on a standard Fedora 24 installation. You can use a cloud image as all the needed packages will be specified. After installing the machine, you login as a standard user, which can do a password-less sudo.

$ ssh fedora@89.42.141.96
$ sudo su -
#

Install docker and client

From here all the commands will be run as root, unless otherwise specified.

$ dnf install -y docker curl

This will install the basic packages we need to setup the test cluster. Now from a browser you open the following page: https://github.com/openshift/origin/releases/tag/v1.3.0. This shows the current deliverables for the OpenShift Origin v1.3 release. You need to download the file called like openshift-origin-client-tools-v1.3.0-[...]-linux-64bit.tar.gz

$ curl -sSL https://github.com/openshift/origin/releases/download/v1.3.0/openshift-origin-client-tools-v1.3.0-3ab7af3d097b57f933eccef684a714f2368804e7-linux-64bit.tar.gz -o oc-client.tar.gz
$ tar -zxvf oc-client.tar.gz
$ mkdir -p /opt/openshift/client
$ cp ./openshift-origin-client-tools-v1.3.0-3ab7af3d097b57f933eccef684a714f2368804e7-linux-64bit/oc /opt/openshift/client/oc

Note: I do not install the binary in /usr/bin or /usr/sbin to prevent a conflict with a packaged version, but also because this makes it easier for me to work on a different version of the application. E.g. the current packaged version is v1.2 and does not provide the command we will be using in the next step.

Configure docker

To allow OpenShift to pull and locally cache images, it will deploy a local docker registry. But before docker would be able to use this, we need to specify an insecure registry in the configuration. For this you need to add --insecure-registry 172.30.0.0/16 to /etc/sysconfig/docker.

$ vi /etc/sysconfig/docker

OPTIONS='--selinux-enabled --log-driver=journald --insecure-registry 172.30.0.0/16'

After this we will setup to allow the standard user to communicate with the docker daemon over the docker socket. This is not a necessary step, and does not make the system more secure. It does make it easier not having to move between user and using sudo all the time.

$ groupadd docker
$ usermod -a -G docker fedora
$ chgrp docker /var/run/docker.sock

Edit: I recently created an Ansible playbook to perform these steps, as I had to do this on several Atomic hosts. I uses the current Ansible user and adds it to a group, and changes the socket permissions.

After this you can start docker and move on the actual installation of OpenShift.

$ systemctl enable docker
$ systemctl start docker

Note: we will be running this environment with devicemapper as Storage Driver. This is not an ideal situation. If you do further tests, consider changing the storage with docker-storage-setup to use a dedicated volume.

Running OpenShift

Since version 1.3 of OpenShift, the client provides a cluster up commands which stands up a very simple all-in-one cluster, with a configured registry, router, image streams, and default templates.

As the fedora user, you can check if you can access docker

$ docker ps

CONTAINER ID    IMAGE   COMMAND     CREATED     STATUS      PORTS   NAMES

No containers should be returned. This mean you can communicate with the docker daemon. Now you are ready to start the test cluster.

`cluster up`

$ export PATH=$PATH:/opt/openshift/client/
$ ./oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ... 
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ... 
   Using 10.5.0.27 as the server IP
-- Starting OpenShift container ... 
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:
       https://10.5.0.27:8443

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

And that was it! Now you are running an OpenShift environment. You can check this as follows:

$ docker ps

CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
cebba70022a6        openshift/origin-haproxy-router:v1.3.0    "/usr/bin/openshift-r"   16 seconds ago      Up 15 seconds                           k8s_router.9426645a_router-1-o3454_default_ba2e0814-8483-11e6-924a-fa163e29da46_e9a13a8d
32aa5e84a04d        openshift/origin-docker-registry:v1.3.0   "/bin/sh -c 'DOCKER_R"   17 seconds ago      Up 15 seconds                           k8s_registry.f0a205a4_docker-registry-1-v57os_default_b9fc0130-8483-11e6-924a-fa163e29da46_24863324
03ee38d125cb        openshift/origin-pod:v1.3.0               "/pod"                   18 seconds ago      Up 16 seconds                           k8s_POD.4a82dc9f_router-1-o3454_default_ba2e0814-8483-11e6-924a-fa163e29da46_ea6d1d08
44d6f8d2d9d6        openshift/origin-pod:v1.3.0               "/pod"                   18 seconds ago      Up 16 seconds                           k8s_POD.9fa2fe82_docker-registry-1-v57os_default_b9fc0130-8483-11e6-924a-fa163e29da46_76754271
60e7cc5f4e5d        openshift/origin-deployer:v1.3.0          "/usr/bin/openshift-d"   21 seconds ago      Up 19 seconds                           k8s_deployment.59c7ba3f_router-1-deploy_default_b3660c7b-8483-11e6-924a-fa163e29da46_8e02f47a
f1fe993ddcac        openshift/origin-pod:v1.3.0               "/pod"                   22 seconds ago      Up 20 seconds                           k8s_POD.4a82dc9f_router-1-deploy_default_b3660c7b-8483-11e6-924a-fa163e29da46_9a38fe5e
72068a244ac8        openshift/origin:v1.3.0                   "/usr/bin/openshift s"   49 seconds ago      Up 48 seconds                           origin

Client connection

After running the command oc cluster up you will be automatically logged in. For this it writes the login configuration in ~/.kube/. If you want to change you can login using:

$ oc login

The standard user provided is developer.

Verify

Now we need to verify if we can deploy a simple application. However, without changes, OpenShift will not run containers with a root-user process. For example an nginx container would fail with a permission denied error.

Instead, we will for now run a simple Hello container:

$ oc run hello-openshift --image=docker.io/openshift/hello-openshift:latest --port=8080 --expose

service "hello-openshift" created
deploymentconfig "hello-openshift" created

This would create the container and schedule it. You can check the progress with:

$ oc get pod

NAME                        READY     STATUS    RESTARTS   AGE
hello-openshift-1-xi7f0     1/1       Running   0          9m

You will also see a -deploy container. This is not needed for our verification.

To check the application, we need to get the IP address that has been assigned to the Pod. You can do this as follows:

$ oc get pod hello-openshift-1-xi7f0 -o yaml | grep podIP

  podIP: 172.17.0.7

All you have to do now is open the endpoint:

$ curl 172.17.0.7:8080

Hello OpenShift!

And that is it, you have a working OpenShift test cluster.

Teardown

If you are down with this, you simply do a:

$ oc cluster down

and all the containers used in the deployment will be torn down.

Conclusion

Using OpenShift's cluster up command you can easily setup an environment for developers to run and test their applications. The current of OpenShift provided with Fedora 24 does not offer this command, as the packaged version is v1.2. However, this change is in Rawhide and is therefore expected to be released as part of Fedora 25.

In future articles I will detail more about how to create applications for the OpenShift container platform, and how to check and maintain the life cycle of the deployed application. For now, take a look at the other source of information below.

More information

OpenShift Homepage
OpenShift Origin at GitHub
Knowledge-base article about OpenShift

Deploying Kubernetes using Ansible

2016-09-26T00:00:00+08:00

Recently I did a lot of tests with Atomic, such as a creating custom images for Ceph, and ways to provide an immutable infrastructure. However, Atomic is meant to be a host platform for a container platform using Kubernetes. Their Getting Started guide describes how to setup a basic environment to host containerized applications. However, this is a manual approach and with the help of Vincent* I create a way to deploy this Kubernetes environment on Atomic and a standard CentOS installation using Ansible. In this article I will describe the components and provide instructions on how to deploy this basic environment yourself.

Requirements

To deploy the Kubernetes environment you will need either Atomic Host or a standard CentOS installation. If you are using this for testing purposes, a cloud image on an OpenStack provider will do. The described Ansible scripts will work properly on both Atomic Host and CentOS cloud images. Although it has not been tested on Fedora, it should be able to make this work with minimal changes. If you do, please contribute these changes back.

You will need at least 2 (virtual) machines. One will be configured as the Kubernetes master and the remaining node(s) can be configured as minions or deployment nodes. I have used at least 4 nodes; a general controller node to perform the deployment from (also configured to install the Kubernetes client), a master node and at least two deployment nodes. Take note that this deployment does not handle docker-storage-setup and High Availability.

Setup for deployment

Almost all the deployments I perform are initiated from a short-lived controller node. This is a machine that allows incoming and outgoing traffic, and mostly gets configured with an external Floating IP. This host can be seen as a jumphost. I will configure it with a dedicated set of SSH keys for communication with the other machines. You do not have to do this, and if you have limited resources, consider this host to be the same as the Kubernetes master node.

On this machine do the following:

$ yum install -y ansible git
$ git clone https://gitlab.com/gbraad/ansible-playbook-kubernetes.git
$ cd ansible-playbook-kubernetes
$ ansible-galaxy install -r roles.txt

This will install the Ansible playbook and the required roles:

gbraad.docker
gbraad.docker-registry
gbraad.kubernetes-master
gbraad.kubernetes-node
gbraad.kubernetes-client

Each of the roles take care of the installation and/or configuration of the environment. Do take note that the Docker and Kubernetes roles do not install packages on the Atomic hosts as these already come with the needed software.

Configure the deployment

If you look at the files deploy-kubernetes.yml you will see three tasks for a different group of hosts. As mentioned before, they will each take care of the installation when needed.

- name: Deploy kubernetes Master
  hosts: k8s-master
  remote_user: centos
  become: true
  roles:
  - gbraad.docker
  - gbraad.kubernetes-master

- name: Deploy kubernetes Nodes
  hosts: k8s-nodes
  remote_user: centos
  become: true
  roles:
  - gbraad.docker
  - gbraad.kubernetes-node

- name: Install kubernetes client
  hosts: k8s-client
  remote_user: centos
  become: true
  roles:
  - gbraad.kubernetes-client

Take notice of the setting of remote_user. On an Atomic host this is a passwordless sudo user, which can also login with SSH using a passwordless key-based authentication. If you use CentOS, please configure a user and allow add an entry with echo "username ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/username.

Change the inventory

Ansible targets a playbook against a single or group of nodes that you specify in an inventory file. This file is named hosts in this playbook repository. When you open it you will see the same set of names as specified above in the hosts entry in the deploy-kubernetes.yml playbook. For our purposes you will always have to deploy a master and a node. If you do not specify the master node, the installation will fail as some of the deployment variables will be used for configuration of the nodes.

$ vi hosts

[k8s-master]
atomic-01 ansible_ssh_host=10.5.0.11

[k8s-nodes]
atomic-02 ansible_ssh_host=10.5.0.14
atomic-03 ansible_ssh_host=10.5.0.13
atomic-04 ansible_ssh_host=10.5.0.12

Group variables

At the moment the roles are not very configurable as they are mainly targeting a simple test environment. Inside the folder group_vars you will find the configration for the Kubernetes nodes. These are as follows

skydns_enable: true
dns_server: 10.254.0.10
dns_domain: kubernetes.local

Perform the deployment

After changing the variables in the hosts inventory file and the group variables, you are actually all set to perform the deployment. We will start with the following.

$ ansible-playbook -i hosts deploy-docker-registry.yml

This first step will install Docker on the master node and pull the Docker Registry container. This is needed to provide a local cache of container images that you have pulled.

After this we can install the Kubernetes environment with:

$ ansible-playbook -i hosts deploy-kubernetes.yml

Below I will describe what each part of the playbook does and some information about the functionality.

Playbook and role description

Below I will give a short description of what each part of the playbook and role does.

Role `gbraad.docker`

Each node in the playbook will be targeted with the following role: gbraad.docker. This role determines if the node is an Atomic Host or not. This check is performed in the tasks/main.yml file. It the node is not an Atomic Host, it will include install.yml to perform additional installation tasks. At the moment this is a simple package installation for docker. After this step, the role will set state started and enabled for the services; docker.

Source

Role: `gbraad.kubernetes-master`

As part of the playbook, first we will configure the master node. For this, the role gbraad.kubernetes-master is used. Just like in the previous role, file tasks/main.yml will perform a simple check to determine if an Atomic Host is used or not. If not, some packages will be installed:

kubernetes-master
flannel
etcd

Source

Configure Kubernetes

After this step Kubernetes will be configured on this hosts. Tasks are described in the file tasks/configure_k8s.yml, source.

Role

Congfigure Kubernetes: etcd

This role will create a single 'etcd' server on the master. For simplicty all IP address will be allowed by using 0.0.0.0 as the endpoint. The port used for etcd clients is 2379, but since 4001 is also widely used we will also configure to use this.

Tasks:

Configure etcd LISTEN CLIENTS
Configure etcd ADVERTISE CLIENTS

Configure Kubernetes: common services

After this the role will set an entry in the file /etc/kubernetes/config. This is a general configuration file used by all the services. It sets the local IP address, identified as default_ipv4_address as etcd server and the kubernetes master nodes.

Tasks:

Configure k8s common services

Configure Kubernetes: apiserver

For the apiserer the file /etc/kubernetes/apiserver is used. We will configure it to listen on all IP address. An admission control plug-in is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object. For this role we removed ServiceAccount which is normally used to limit the creation requests of Pods based on the service account.

Restart services

After this the services of the master node are started and enabled. These are:

etcd
kube-apiserver
kube-controller-manager
kube-scheduler

Configure flannel

For this environment we will use flannel to provide an overlay network. An overlay network exists on top of another network to provide a virtual path between nodes who use this overlay network. The steps for this are specified in tasks/configure_flannel.yml source.

The configuration of flannel is controller by etcd. We will copy a file to the master node containing:

{
  "Network": "172.16.0.0/12",
  "SubnetLen": 24,
  "Backend": {
    "Type": "vxlan"
  }
}

This file is located in the role as files/flanneld-conf.json. Using curl we will post this file to etcd on the master.

Tasks:

Copy json with network config
Configure subnet

After these steps the Kubernetes master is configured.

Role: `gbraad.kubernetes-node`

For the configuration of a Kubernetes node we use the role gbraad.kubernetes-node. Just like in the previous roles we determine in tasks/main.yml to install packages or not. For all the nodes we will configure:

SELinux
flannel
kubernetes specific settings

Source, Role

Configure SELinux

Because I use persistent storage using NFS with Docker, I configure SELinux to set the boolean virt_use_nfs.

Configure flannel

In the tasks:

Overlay | configure etcd url
Overlay | configure etcd config key
Flannel systemd | create service.d
Flannel systemd | deploy unit file

We configure all the nodes to use the etcd instance on k8s-master as the endpoint for flannel configuration. These tasks configure the networking to use the flanneld provided bridge IP and MTU settings. Using the last two tasks we configure systemd to start the service.

After this change we will restart the flanneld service.

Configure Kubernetes

In the file tasks/configure_k8s.yml we do final configuration of the Kubernetes node to point it to the master node.

In the tasks:

k8s client configuration
k8s client configuration | KUBELET_HOSTNAME
k8s client configuration | KUBELET_ARGS

we configure how the node is identified.

In the tasks:

k8s client configuration | KUBELET_API_SERVER
Configure k8s master on client | KUBE_MASTER

we set the location of the API server and the master node. Both of these will point to the IP address of the k8s-master.

After this change we will restart the kubelet and kube-proxy service.

After deployment

If all these tasks succeeded, you should have a working Kubernetes deployment. In the next steps we will perform some simple commands to verify if the environment works.

Deploy client and verify environment

As you have noticed from the hosts inventory file, there is a possibility to specify a client host:

[k8s-client]
controller ansible_ssh_host=10.5.0.3

You do not have to deploy a kubernetes client using this playbook. It will install a single package, but you could also just use the statically compiled Go binary that is provided from the Kubernetes releases:

$ wget http://storage.googleapis.com/kubernetes-release/release/v1.3.4/bin/linux/amd64/kubectl
$ chmod +x kubectl

Verify nodes

Oncae all the nodes have been deployed using the playbook, you canb verify communication. From a client node you can perform the following command:

$ kubectl --server=10.5.0.11:8080 get node`

NAME             LABELS    STATUS
10.5.0.12        <none>    Ready
10.5.0.13        <none>    Ready
10.5.0.14        <none>    Ready

Schedule workload

If all looks good, you can schedule a workload on the environment. The following commands will create a simple nginx instance.

$ vi kube-nginx.yml

apiVersion: v1
kind: Pod
metadata:
  name: www
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
          hostPort: 8080

This file described a Pod with a container called nginx using the 'nginx' image. To schedule it, do:

$ kubectl --server=10.5.0.11:8080 create -f kube-nginx.yml

pods/www

Using the following command you can check the status:

$ kubectl --server=192.168.124.40:8080 get pods

POD       IP            CONTAINER(S)   IMAGE(S)   HOST                            LABELS    STATUS    CREATED      MESSAGE
www       172.16.59.2                             10.5.0.12/10.5.0.12             <none>    Running   52 seconds
                        nginx          nginx                                                Running   24 seconds

If you now open a browser pointing to the node Kubernetes used to create it (http://10.5.0.12:8080/), you will see an nginx welcome page.

Conclusion

Setting up Kubernetes can be a daunting task as there are many different components involved. Kelsey Hightower has a very good guide, called Kubernetes The Hard Way that will teach you how to deploy kubernetes manually on different cloud providers, such as AWS and the Google Compute Engine. After gaining some experience, it is advised to look at automation to deploy an environment, such as the the Ansible scripts that can be found in the contrib repository of Kubernetes. this allows you to stand up an environment with High Availability. The scripts described in this article should only serve as an introduction to gain understanding what is needed for a Kubernetes environment.

If you want a production-ready environment, please have a look at OpenShift. OpenShift deals with setting up a cluster of kubernetes nodes, scheduling workload and most important, how to set up images for deployment. This is done using what is called 'source to image'. This and OpenShift itself will be the topic of future articles.

More information

Knowledge-base article about Kubernetes on Atomic
Knowledge-base article about Kubernestes
Hands-on-Lab: Kubernetes on OpenStack (Not all material published yet)
Deploying Kubernetes on Atomic hosts

Mentors and recommended books

2016-09-23T00:00:00+08:00

As a consultant, an engineer, or just any employee, you are responsible to provide solutions to challenges and issues you face on the workfloor. No no, let me say it this way, you are responsible to come up with ideas to improve a situation, make it more efficient. Or, in other words, you have to have a lot of knowledge and a certain way of thinking, being analytical. Simple. But knowledge is something you can develop. And one thing you always have to have on mind. You are responsible for your own personal development...

I have been employed by two companies that were very strong on this; Sogyo and ThoughtWorks. Both are solution providers, mostly focussed on the software. They both used agile methodologies, development practices, like pair programming, cod reviews, etc. All to ensure the delivery of excellence. Especially this, the delivery of quality and outstanding results, expected that engineers keep themselves up-to-date. They reserved time for this and organized meetups, for personnel and the software development community. I have learned a lot from this, but one thing that both of them did was allowing you a mentor (or sponsor).

This is not a coach that helps you to find your way around the company in the first few weeks or months, but a person that helps you with career development. It is not that he or she defines targets, but sits down with you and discuss about issues you face, knowledge gaps you might have, etc. My mentor at Sogyo, Edwin van Dillen, has been very inspirational. It made me want to mentor other people... and this what I do now for new contributors to Open Source, and software communities in general. Always, when you can, look for a mentor. Someone who can help you along the way to improve yourself. But improvement is also something you can do yourself. Never stop learning. We call this Continuous Improvement.

So, one of the things I often get is the question of 'what should I read'? I generally do not promote books that much, as a lot of technology books are dated by the time they are printed. I'd rather refer to a good blog post or living content that gets updated regularly. However, there is always this list of books that are timeless. They have been inspirational or defining in the knowledgd they provide. And this is something I want to start adding to my blog, a review or discussion of a book that helped me in my career. Books that I even bought again either as ePub or PDF after I moved to the other side of the world. I started to compile a list on my Knowledge Base * repository as a reference to others, but thought I would be a better idea to post them here with a short intro. So, look for the next post in the category 'Books'.

Object Design

2016-09-23T00:00:00+08:00

One of the books I got recommended early in my career was Object Design: Roles, Responsibilities, and Collaborations by Rebecca Wirfs-Brock and Alan McKean. The topic of this book is OOD (Object-Oriented Design). It is a wordy book, which could certainly could have been shorter, but the content makes certainly up for this issue. This write-up is based on reading this book many years ago, so bear with me if things are not completely in line with the book. Although, the knowledge in this book certainly has contributed to this point-of-view. I currently own a copy of this book as reference in a digital format.

The author, Rebecca Wirfs-Brock, is a software engineer and consultant in Object-Oriented Design, and as the title suggests the book details about the design of your object model and how it interacts with other objects.

One of the things you will learn when creating software, is that design of the interactions and relations between objects and classes probably is one of the most important and hardest things. Luckily, software design and development is an evolutionary process. You will not get it right he first time and you also shouldn't aim for this. What can be modeled today as a variable on an object today, might become a class itself in the future. If you would design this from day one, you probably over-design the software and implement functionality that might never be needed.

What I learned from this book is how this interaction and collaboration between objects happen. Where do I keep the state? who can make a change? If the object can encapsulate it, this is probably where it should live. Ask yourself, why would another object need to know about the information, and whose responsibility is it? The book details as far as I remember CRC-cards (Class Reponsibility Rollaboration) and shows a method to get this clear. In my work I have never created the actual cards, but the concept behind it is very valuable. Get this book and browse through it, keep a copy as a reference. It is well worth it.

More information about this book can be found in my Knowledge Base entry.

Document generation using Markdown and Pandoc

2016-09-13T00:00:00+08:00

Maintaining documentation can be an involved process, especially when different versions and output formats are needed. One of the documents I, and probably you also, have to maintain is your work history or resume/CV. Some while back I changed this document from an OpenOffice document to plain text and ever since, this workflow has influenced how I deal with other documentation.

Requirement and timeline

If I remember correctly the original version of my resume was written in Microsoft Word. As you can imagine, this limits the usage to Windows as the platform. Small corrections became unnecessarily difficult and painful. Of course I could use CrossOver Office or Wine and install Office 2003. But, this is never a long-term solution. So, I moved to OpenOffice as the .odt format looked promising. It was easy to use to generate the needed PDF output, but still... the workflow was far from ideal.

I wanted a workflow in which I can use just plaintext, which could be converted into the final format. Text is easy to index and to story under version management. This lead me to using XML directly and the use of FO (Formatting Objects), and Apache FOP. This was painful, but the results were decent to even very acceptable. It allowed me to describe elements with metadata, such as 'suffix', 'employment-type', etc. And all could be validated and transformed using XSLT. Although the conclusion was that the workflow was slow as there was no preview and transforming just took long. Also, describing elements was not useful as no-one would use the source material directly. Most recruiter use analysis tools that rely on extracting keywords and generate a profile based on this information. This can lead to incorrect results, but since they analyse hundreds of resumes this seems one of the best methods, besides being introduced by a reference.

I moved to a simpler approach which allows me to change the document and have a partial, and acceptable preview. By this time Markdown got more popular and I tried to convert my resume and it seemed like an acceptable approach. Using a standard parser the documents looked OK, but it missed the oomph. Then pandoc arrived and it changed everything.

Markdown

Using Markdown I would be able to write the following

# _**Gerard Braad**_


## Personal information

  * Chinese name
    : 吉拉德
  * Date of Birth
    : 22 February, 1981
  * Nationality
    : Dutch
  * Email
    : [me@gbraad.nl][personal email]

It will get rendered as follows:

This source is humanly readable, and can be parsed by something like the Webinterface of GitHub into a decent representation. And when using Markdown Preview for Sublime Text, even editing this became a pleasure.

Pandoc

Now for introducing Pandoc. This tool is like a Swiss knife for handling documents. It can convert between different formats and allows you to specify a template for specific formats, such as HTML. This tool allows me to use:

$ pandoc -o resume.html resume.md

This will generate a standard HTML output from resume.md as input. Headings are converted to h6, h5, etc. and links become a, etc.

Styling and layout

But the beauty shows when a template is used:

$ pandoc --template resume-template.html -o resume.html resume.md

This will generate a similar output, however the template refers to a stylesheet to style the look of the output. Using a template and stylesheet I am able to format the document to my liking. And since the output of markdown auto-generates names for elements in a consistent way, it is easy to style and format the document.

Take for instance the photo. Although I personally do not like photos on a resume, it is often requested. Placing it inline with the text does not look very nice.

Using CSS it is possible to convert the following:

# _**Gerard Braad**_


## Personal information

![Photo][personal photo]

  * Chinese name
    : 吉拉德

With the styling:

figure {
  float: right;
  margin: 5px;
  border: 1px solid #eee;
}

The output of this will look as follows:

Containerizing pandoc (software distribution)

Pandoc itself is written in Haskell. Although a nice language, it has a different style of programming. This means that few people can contribute, and a distribution like Fedora or Ubuntu can not always provide the latest version. This can be due to possible dependency issues, but also the effort a packager has to put into it. The developer of this tool only provides a .deb package. Since I mostly work on Fedora and CentOS, I had to find an alternative solution to install this tool.

This is a great use-case for a container, as it consolidates the dependencies of the tool, and will allow me to use the tool as packaged by the developer, on a distribution that would else not be supported. Because I use the provided package I can also file issues against a known version, which can aid in reproducibility.

This lead to me creating a container called docugen. It uses pandoc, jekyll and pandoc-ruby to integrate Jekyll with pandoc.

By setting an alias I can call pandoc as it is a local command, which actually calls the container:

$ alias pandoc='docker run -it --rm -v $PWD:/workspace gbraad/docugen pandoc'
$ pandoc -o resume.html resume.md

You can find more information about it here.

Automated generation

Since creating the containerized version of Pandoc, it was easy to setup an automated process for generation of the resume output. Because I use GitLab for almost all of my projects I only needed to add a CI runner definition to my repository. Below is a snippet of the generation, which publishes to GitLab pages.

.gitlab-ci.yml

pages:
  image: gbraad/docugen:latest
  script:
    - mkdir -p public
    - pandoc -t html5 -o public/index.html resume.md
  artifacts:
    paths:
    - public

The result of my resume gets published on GitLab at http://gbraad.gitlab.io/resume/ and syncs to GitHub, which hosts it at https://gbraad.nl/resume.

PDF generation using Pandoc -> LaTeX

Pandoc can create nice looking PDF documents by using LaTeX. The handout of my OpenStack Hands-on-Lab was produced with this.

$ pandoc -V geometry:margin=1in -o deploying-OpenStack.pdf deploying-openstack.md

Although the results look really good, I had several issues with it:

increased the size of the docugen container by more than a gigabyte.
issues with properly rendering Chinese
- needed additional installation of modules and fonts
would not respect width and margins for pre-formatted blocks

Likely, all of them could be solved with some smart hacks, but I didn't have the time and interest to learn LaTeX. This felt to me like learning how to use XSLT to transform XML into something meaningful. I still have the packages installed in the container as some documents still use xelatex for the PDF generation.

PDF generation using PhantomJS

To solve the issues mentioned above, I now use phantomjs. PhantomJS is a headless browser based on the WebKit rendering engine, and is scriptable with JavaScript. By using this approach I am able to use a single stylesheet to produce the same output on screen as on paper.

var page = require('webpage').create(), system = require('system'),
    address = system.args[1], output = system.args[2];
page.viewportSize = { width: 600, height: 600 };
page.paperSize = { format: "A4", orientation: 'portrait', margin: '1cm' };
if(system.args.length > 3) { page.zoomFactor = system.args[3]; }
page.open(address, function (status) {
   page.render(output);
   phantom.exit();
});

To generate a PDF, I call:

$ phantomjs topdf.js resume.html resume.pdf

Page breaks

To produce a nicer result, I have also added the following to my stylesheet:

h1 {
  page-break-before: always;
}

#gerard-braad {
  text-align: center;
  page-break-before: avoid;
}

When printed this inserts a page break before the header starts. So when a new topic starts, but avoids it for the start of the document.

Conclusion

Generating a document from plaintext using Markdown can be beneficial. The source will never suffer from issues with 'being unreadable' or 'deprecated software', and it can easily be indexed. Tracking changes and even publishing a change is not as easy as opening the file on my mobile phone, and commiting it using git. Automatically the result will be published.

I have moved all my documentation over to plaintext, from the handsonlabs, training and teaching material, schedules, etc. I am still tweaking some of the processes, but the convenience is that a single source material can be used for different outputs. For instance, I publish my articles of this blog in Markdown, as generated documents, and as a GitBook.

Do not settle for a solution forever. Over time requirements can change and new insights can be utilized to produce a better solution.

Endnotes

I looked for some of the older formats and output, but was unable to track them down. I do remember my businesscard was used as a reference for 'microformats'. And someone tried to introduce an XML standard for resume, called xmlresume if I recall correctly. From experience I could conclude this would not work; no applicant will provide XML and converting it in a tool seems pointless.

At the moment I am still looking for a nice tool on Android to edit the documents. I use SGit with Writeily Pro, but I am a little disappointed in both tools. Opening a file for viewing in sgit will modify the file, and it does not deal well with resetting this. And Writeily Pro becomes slow over time when editing a file. Closing it and re-opening solves this, but it breaks the writing streak. Sometimes I rather use DroidEdit, as it is just a text editor and does not suffer from this issue. Tools on the iPad, like Daedalus Touch and Textastic feel more productive. Suggestions are welcome...

Highly Available storage using GlusterFS

2016-09-12T00:00:00+08:00

In this article I will describe how you can setup a webserver environment with Highly Available (HA) storage, provided by GlusterFS. We will be setting up a simple environment in which storage between two webservers needs to be replicated.

GlusterFS is a scalable network filesystem suitable for data-intensive tasks. It is free and open source software and can utilize common off-the-shelf hardware. To learn more, please see the Gluster project home page.

Prerequisites

You will need two servers available, each with a dedicated disk for storage, such as /dev/vdb or /dev/sdb. I will be using Fedora 24, which of writing is the latest version.

The following needs to be installed on both servers:

$ dnf install -y glusterfs-server xfsprogs

This includes the server component for the distributed filesystem, and the tools needed to format the disk we will use to store the files. Gluster recommends to use XFS as the filesystem.

We will assume for this article, that the machines will be running Apache. If this is not installed already, you can do the following:

$ dnf install -y httpd

Hostnames

Make sure the hostnames and IP addresses of the servers are known to each other. For this, check the content of /etc/hosts

/etc/hosts

37.153.173.244 server01-public
37.153.173.245 server01-public
10.3.0.44 server01-private
10.3.0.45 server02-private

Note: these servers have two interfaces, Although this is not necessary, it is strongly recommended to separate the traffic from the web and what you use for the storage network. In our case, the web-facing IP addresses have not been load-balanced yet. This is a topic for future articles.

Prepare disks

Since Gluster relies on extended file attributes (xattr), we will need to increase the inode size when formatting the disks.

Perform on the following on both the servers

$ mkfs.xfs -i size=512 /dev/vdb

meta-data=/dev/vdb               isize=512    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Now we need to create a mountpoint for the filesystem. We will use /data/var-www/brick01 on server01:

$ mkdir -p /data/var-www/brick01
$ echo '/dev/vdb /data/var-www/brick01 xfs defaults 1 2' >> /etc/fstab
$ mount -a

and /data/var-www/brick02 on server02:

$ mkdir -p /data/var-www/brick02
$ echo '/dev/vdb /data/var-www/brick02 xfs defaults 1 2' >> /etc/fstab
$ mount -a

Setup Gluster

You will now create a cluster of Gluster servers, which means that we will create a trust between the two nodes. But before we can do this, we need to make sure the Gluster daemon is running on the nodes. Perform the following commands on both of the nodes:

$ systemctl enable glusterd
$ systemctl start glusterd
$ systemctl status glusterd

Now we need to create the trusted connection. The following commands will only need to be used on one of the nodes. In my case, this will be server01:

$ gluster peer probe server02-private

peer probe: success.

Note: the hostname here refers to the private IP address. This is not the same range as the public network. For security purposes you will likely separate the traffic.

You can verify is the trust exists by running:

$ gluster peer status

Number of Peers: 1

Hostname: server02-private
Uuid: af523e9b-4257-485d-aa45-ebd24f115c42
State: Peer in Cluster (Connected)

If you would run this command also on server02, you would get a similar result:

Number of Peers: 1

Hostname: server01-private
Uuid: 74c77082-b62c-4cc4-b33f-02de36083990
State: Peer in Cluster (Connected)

Create Gluster volume

In case of a failure, we do not want our data to be unavailable. Therefore we need to have two copies of our data, and we do this by creating a replicated volume.

This command will only need to be run on one node, as Gluster will take care of the related nodes.

$ gluster volume create var-www replica 2 \
          server01-private:/data/var-www/brick01/volume \
          server02-private:/data/var-www/brick02/volume

volume create: var-www: success: please start the volume to access data

$ gluster volume start var-www

volume start: var-www: success

$ gluster volume info

Volume Name: var-www
Type: Replicate
Volume ID: e91014ca-f98c-42e5-8e75-de4d2e65f569
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server01-private:/data/var-www/brick01/volume
Brick2: server02-private:/data/var-www/brick02/volume
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

Mount Gluster volume

Before you can mount the volume on /var/www to share the webpages, we need to move the original /var/www out of the way and create a new directory to mount. Perform the following commands on both the servers:

$ systemctl stop httpd
$ mv /var/www{,.orig}
$ mkdir /var/www

Now that we have the mountpoint, we can add an entry to fstab to deal with mounting of our volume. On server01 you need to do the following:

$ echo 'server01-private:/var-www /var/www glusterfs defaults,_netdev,fetch-attempts=3 0 0' >> /etc/fstab
$ mount -a

And on server02:

$ echo 'server02-private:/var-www /var/www glusterfs defaults,_netdev,fetch-attempts=3 0 0' >> /etc/fstab
$ mount -a

Now the volume has been mounted and replication is available. We use _netdev because network needs to be up before we can use this filesystem, and fetch_attempts will deal with the volume information in case of a failure. Since we have multiple IP address, this is a suggested setting.

Test replication

On server01 you can do the following:

$ mkdir -p /var/www/html
$ echo 'Hello, World!' > /var/www/html/index.html
$ systemctl start httpd

If you would now open the web facing IP address, you should see the message: Hello, World!.

Note: if you have issues opening the page, or the default Fedora test page gets returned, you might have SELinux enabled. In this case, you need to turn on the following boolean:

$ setsebool -P httpd_use_fusefs 1

which will allow Apache to use the FUSE filesystem mount as used by GlusterFS.

Now, on server02 you can check the content of the folder:

$ ls /var/www/html

and you would see an index.html file. Now start Apache and you will see that the content is also served from this server.

$ systemctl start httpd

Conclusion

Using GlusterFS it is easy to setup an environment that can handle issues with storage availability. You now have a replicate volume that is available on two servers. Actions we take on each of these servers, will be replicated to the other server.

In this case we have setup the servers to handle both the storage and act as a client to read these files. More advanced are available and are described in the documentation.

Next step

Now that we have implemented replicated storage for our webservers, we need to provide a HA solution to deal with the traffic on the webfacing IP addresses. In an OpenStack environment you could use Neutron to setup a loadbalancer and health monitor for the two servers. How to set this up in described in the following article. However, you can also do so with HAProxy and a failover mechanism, which will be the topic of a future article.

If you have questions please let me know on Twitter at @gbraad or by email.

Highlighting of Code and Text

2016-09-09T00:00:00+08:00

Code is read more often than it is written. Nothing is truer than this statement. This means that when writing code it is important to have a good idea of what you are doing. Write clean code, which is simple or easy to reason about. Since if you can't explain it, how could others even understand it well. Syntax highlighting of code can help. With this, certain keywords or structure get emphasized by using a color. I do not think it is an ideal solution, but subconsciously it seems I focus more on the code. Likely, the colors themselves are pleasing to look at.

Pleasant to the eyes

Perhaps because of this last, I tend to always implement or set the Tomorrow Theme for my color scheme, and specifically the Night Bright variant. Below you can find an example, as I also enabled it on my blog by implementing a Pygments style.

Example

server.js

'use strict';

// get port from environment settings for deployment on Heroku
var EXPRESS_PORT = process.env.OPENSHIFT_NODEJS_PORT || process.env.PORT || 4000;
var EXPRESS_IPADDR = process.env.OPENSHIFT_NODEJS_IP || process.env.IPADDR || '127.0.0.1';
var EXPRESS_ROOT = __dirname;

function startExpress(root, port, ipaddr) {
  var express = require('express');
  var app = express();
  app.use(express.static(root));
  app.listen(port, ipaddr, function() {
    console.log('Listening on %s:%d',
        ipaddr, port);
  });
}

startExpress(EXPRESS_ROOT, EXPRESS_PORT, EXPRESS_IPADDR);

Implementations

Chris Kempson now has a new color scheme called 'base16`and it has a 'Tomorrow' variant. It must be the name as I always install 'Tomorrow' first. Here are some of the Night Bright variants I have made over the years:

How about language

Since my wife and I are very involved in teaching, I have also experimented with the idea of using word or word-class highlighting for English. The results are that it can actually help. Words that need to be stressed, or emphasized, can be highlighted in a different color. Think of words like 'very', 'never', etc. Some of this can be automated, as can be seen in the following examples:

These implementations range from simple hard-coded word matching, to actual Natural Language Processing.

Example

Take for instance the following example of Alice In Wonderland - Down the Rabbit Hole.

"There was nothing so very remarkable in that; nor did Alice think it so very much out of the way to hear the Rabbit say to itself, 'Oh dear! Oh dear! I shall be late!' (when she thought it over afterwards, it occurred to her that she ought to have wondered at this, but at the time it all seemed quite natural); but when the Rabbit actually took a watch out of its waistcoat-pocket, and looked at it, and then hurried on, Alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with either a waistcoat-pocket, or a watch to take out of it, and burning with curiosity, she ran across the field after it, and fortunately was just in time to see it pop down a large rabbit-hole under the hedge."

When adverb, verb, conjunction, etc. are highlighted, the text will look as follows:

Note: this is taken from English syntax highlighter. Try and see for yourself if it can help.

Chinese

Highlighting does not only work well for English. Chinese for instance is a tonal language. Which means that a pronounciation of a character can have a different tone for a different meaning. When not proprerly mastered, this can lead to a lot of confusion or embarrassment. For learner's of Chinese it is actually common to highlight the tones. An example of this is shown here:

welcome

v.t.
欢迎 huānyíng;  迎接 yíngjiē

Welcome home again!

欢迎你又回家了!

Huānyíng nǐ yòu huíjiā le!

Conclusion

Highlighting can serve different purposes. I can not recall that highlighting actually helped me to write clean or correct code, or that it even prevented me from making a mistake or typo. However, it can certainly provide a plesant feeling, in which you can 'feel good' about your code in another way. For languages, such as English and Chinese, it can assist with the learning process.

non-root user inside a Docker container

2016-09-08T00:00:00+08:00

One of the things that you notice when using Docker, is that all commands you run from the Dockerfile with RUN or CMD are performed as the root user. This is not only a bad security practice for running internet facing services, it might even prevent certain applications from working properly. So, how do you run commands as a non-root user? For people using Red Hat-based systems, such as Fedora or CentOS I will explain how you can do this.

Let's begin

Let's say you have a Dockerfile using Fedora as a base:

FROM fedora:24

Create user

Before we can use a different user, we need to create one:

RUN adduser user

Run as user

Now you can run commands as this user by doing:

RUN su - user -c "touch me"

The container start process can be changed to:

CMD ["su", "-", "user", "-c", "/bin/bash"]

This way, a bash shell will open as the user.

Allow `sudo` usage

Since you are running a normal user, it might be handy to install the following package inside the container:

RUN dnf install -y sudo

This will allow you to use sudo to perform actions as the root user:

RUN echo "user ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/user && \
    chmod 0440 /etc/sudoers.d/user

Final `Dockerfile`

To combine all of this, your Dockerfile would look as follows:

Dockerfile

FROM fedora:24

RUN dnf install -y sudo && \
    adduser user && \
    echo "user ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/user && \
    chmod 0440 /etc/sudoers.d/user

RUN su - user -c "touch mine"

CMD ["su", "-", "user", "-c", "/bin/bash"]

Result

Building this container is simple:

$ docker build -t user .
Step 1 : FROM fedora:24
 ---> f9873d530588
Step 2 : RUN dnf install -y sudo &&     adduser user &&     echo "user ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/user &&     chmod 0440 /etc/sudoers.d/user
 ---> Using cache
 ---> 05c3f3c7e9fc
Step 3 : RUN su - user -c "touch me"
 ---> Running in 584ab0ad025b
 ---> 66ea558b9855
Removing intermediate container 584ab0ad025b
Step 4 : CMD su - user -c /bin/bash
 ---> Running in 780e0ccd5f61
 ---> b19a10012b28
Removing intermediate container 780e0ccd5f61
Successfully built b19a10012b28

Running it will produce the following result:

$ docker run -it --rm user
[user@48add6313ab6 ~]$ ls -al
total 20
drwx------ 2 user user 4096 Sep  8 08:51 .
drwxr-xr-x 3 root root 4096 Sep  8 08:46 ..
-rw-r--r-- 4 user user   18 May 17 14:22 .bash_logout
-rw-r--r-- 4 user user  193 May 17 14:22 .bash_profile
-rw-r--r-- 4 user user  231 May 17 14:22 .bashrc
-rw-rw-r-- 1 user user    0 Sep  8 08:51 me
[user@48add6313ab6 ~]$ pwd
/home/user
[user@48add6313ab6 ~]$

Base images

If you are setting up base images for re-use, it can be handy to use the USER keyword. This will run all the following commqnds as a particalur. Lets say, you want to use a service, which is Node.JS based, you can create the following Dockerfile

FROM centos:7
MAINTAINER Gerard Braad <me@gbraad.nl>

# Run update and install dependencies
RUN yum update -y && yum install -y nodejs npm

# Add the user UID:1000, GID:1000, home at /app
RUN groupadd -r app -g 1000 && useradd -u 1000 -r -g app -m -d /app -s /sbin/nologin -c "App user" app && \
    chmod 755 /app

# Set the working directory to app home directory
WORKDIR /app

# Specify the user to execute all commands below
USER app

You can then use this container as a base for NodeJS based applications. You copy the application to /app and when commands are run, they will use the user named user.

I hope this has been of some help to you. You can follow me on Twitter or leave a comment below.

Deployment of Ceph using custom Atomic images

2016-09-07T00:00:00+08:00

As part of providing a scalable infrastructure, many components in the datacenter are now moving towards software-based solutions, such as SDN (for networking) and SDS (for storage). Ceph one of these projects to provide a scalable, software-based software solution. It is often used in conjuction with OpenStack for storing disk images and block access for volumes. Setting up an environment usually involves installing a base operating system and configuring the machine to join a storage cluster. In this article I will setup a Atomic based image that allows for fast deployment of Ceph cluster. This is not a recommended solution, but an exercise that shows how to cutomize Atomic Host and using Ansible to standup a Ceph cluster.

Components

Atomic * Host is a new generation of Operating Systems that used to create an immutable infrastructure to deploy and scale containerized applications. For this, it uses a technology called OSTree * which is a git-like model for dealing with filesystem trees. It allows different filesystems to exist alongside of each other. If a filesystem is not booting properly, it is possible to move back to the previous state.

Ceph * is a distributed storage solution which provides Object Storage, Block Storage and FIlesystem access. Ceph comes with several software components which provides a role in the cluster, such as a monitor or storage daemon.

Ansible * is a configuration and system management tool which for example can do multi-node deployment of software or execution of commands

Although Atomic Hosts are primarily used to deploy Docker and Kubernetes environments, or even containerized versions of the Ceph daemons, I will show how it is possible to use ostree to deploy the Ceph software on nodes, with minimal effort. Ansible is used to send commands to these nodes and finalize the configuration.

Preparation

For this setup we will use about 7 machines. To simplify testing, I have performed these actions on an OpenStack environment, such as DreamCompute.

1 controller and compose node, running Fedora 24 with IP address 10.3.0.2
3 Ceph monitors, running CentOS Atomic with IP addresses 10.3.0.3-5
3 Ceph storage nodes, running CentOS Atomic with IP addresses 10.3.0.6-8 and attached storage volume at /dev/vdb.

To deploy you can start with setting up the Fedora node.

Install controller

To deploy our environment we setup a controller node which can access the other nodes in the network using SSH. In our case we will use Fedora 24. After installing the operating system, perform the following commands:

$ dnf update -y
$ dnf install -y  ansible rpm-ostree git python
$ ssh-keygen
$ cat ~/.ssh/id_rsa.pub

The last two commands generate an SSH authentication key and will show the result on the command line.

Atomic nodes

The installation media for Atomic can be found at the main project page. It comes in two different distribution types, CentOS or Fedora. For this article I used CentOS, and downloaded it from the following location:

$ wget http://http://cloud.centos.org/centos/7/atomic/images/CentOS-Atomic-Host-7-GenericCloud.qcow2

and uploaded it to my OpenStack environment with:

$ openstack image create "CentOS7 Atomic" --progress --file CentOS-Atomic-Host-7-GenericCloud.qcow2 --disk-format qcow2 --container-format bare --property os_distro=centos

After this, I created six identical machines, with provisioning a predefined SSH key with cloud-init which we made on the controller node. If you are not using OpenStack, you have to use a cloud-init configdrive to configure the node with the SSH key. This process is described here.

The storage nodes need to have to configured with a dedicated volume for use with the osd storage daemon of Ceph. By creating a volume and attaching them to the node, these will be identified as /dev/vdb. There is no need to prepapre a filesystem on them. This will be handles by the deployment scripts.

Note: if you want to upload disk images to DreamCompute or other Ceph-backed OpenStack image services, you have to convert the disk image to raw format first. An example of this can be found in an example for use of the OpenStack client.

Compose tree

On the controller node we will create an ostree that will be used to setup the Ceph software on the Atomic host nodes. This process involves creating a definition for a filesystem tree that uses the base defintion of CentOS Atomic with the additional packages from the CentOS-specific Ceph packages.

Atomic definition

We will clone the git repositiory containing the base definition:

$ mkdir -p /workspace
$ cd /workspace
$ git clone https://github.com/CentOS/sig-atomic-buildscripts.git \
    -b downstream \
    --depth 1 \
    ceph-atomic

Inside this folder you can add a RPM repository file where the Ceph packages are located:

ceph-atomic/centos-ceph-jewel.repo

[centos-ceph-jewel]
name=CentOS-7 - Ceph Jewel
baseurl=http://buildlogs.centos.org/centos/7/storage/x86_64/ceph-jewel/
gpgcheck=0

And then we will define an Atomic host for Ceph-Jewel and specify which additional repositories and packages are used:

ceph-atomic/ceph-atomic-host.json

{
    "include": "centos-atomic-host.json",
    "ref": "centos-atomic-host/7/x86_64/ceph-jewel",
    "automatic_version_prefix": "2016.0",
    "repos": ["centos-ceph-jewel", "epel"],
    "packages": [
        "ceph",
        "ntp",
        "ntpdate",
        "hdparm",
        "autogen-libopts"
    ]
}

Note: with include we base our image off an existing definition. In our case this will simplify the process, but it also means we will end up with an image with unneeded components. For our test this is not an issue.

After this, the defintion for the Ceph-specific Atomic host is finished. Before we can compose the ostree, we have to create and initialize the repository directory:

$ mkdir -p /srv/repo
$ ostree --repo=/srv/repo init --mode=archive-z2

When you perform the following command:

$ rpm-ostree compose tree \
    --repo=/srv/repo \
    ./ceph-atomic/ceph-atomic-host.json

packages will be fetched and committed to an ostree in /srv/repo. After this, you will have a filesystem tree which can be served from a webserver, to be used by Atomic hosts.

$ cd /srv/repo
$ python -m SimpleHTTPServer

This will serve the content from /srv/repo on the endpoint http://10.3.0.2:8000/.

Deployment

To get the Atomic hosts to use the tree, the following commands will have to be used to provision the Ceph-based filesystem tree:

$ sudo su -
$ setenforce 0
$ ostree remote add --set=gpg-verify=false atomic-ceph http://10.3.0.2:8000/
$ rpm-ostree rebase atomic-ceph:centos-atomic-host/7/x86_64/ceph-jewel
$ systemctl reboot

This is however very cumbersome as we have to do this on six nodes. To optimize this process, we will use Ansible.

Note: setenforce 0 is used to prevent issue with SELinux label during the rebase action.

OSTree provisioning using Ansible

Create a file called hosts in your home directory with the following content:

~/hosts

[mons]
atomic-01 ansible_ssh_host=10.3.0.3
atomic-02 ansible_ssh_host=10.3.0.4
atomic-03 ansible_ssh_host=10.3.0.5

[osds]
atomic-04 ansible_ssh_host=10.3.0.6
atomic-05 ansible_ssh_host=10.3.0.7
atomic-06 ansible_ssh_host=10.3.0.8

To test if the nodes are reachable, use the following command:

$ ansible -i hosts all -u centos -s -m ping

If no error occurs, you are all set.

Using Ansible and the hosts file we can perform these commands on all the nodes.

$ ansible -i hosts all -u centos -s -m shell -a "setenforce 0"
$ ansible -i hosts all -u centos -s -m shell -a "ostree remote add --set=gpg-verify=false atomic-ceph http://10.3.0.2:8000/"
$ ansible -i hosts all -u centos -s -m shell -a "rpm-ostree rebase atomic-ceph:centos-atomic-host/7/x86_64/ceph-jewel"
$ ansible -i hosts all -u centos -s -m shell -a "systemctl disable docker"
$ ansible -i hosts all -u centos -s -m shell -a "systemctl reboot"

The last command will throw errors, but this is expected as the connection to the node is dropped. To check if the nodes are up again, you can perform the previous ping test again.

Note: since we based on the original CentOS Atomic host image, it will have Docker installed. For this article we do not use it, so it is best to disable it.

Deployment using `ceph-ansible`

If the previous step succeeded you would have six atomic hosts that are ready to be configured to form a small Ceph cluster; 3 nodes will serve the role of monitor, while the other nodes will be storage nodes with an attached volume at /dev/vdb. For the deployment we will use ceph-ansible. This is a collection of playbooks to help with common scenarios for a Ceph storage cluster.

From your home folder perform the following command:

$ git clone https://github.com/ceph/ceph-ansible.git --depth 1

This will clone the latest version of the playbooks. But before we can use them, we will need to configure it for our setup. But before we do, we initialize it with the base configuration. Take note that we will re-use our previosuly used ~/hosts file for Ansible.

$ cd ceph-ansible
$ cp site.yml.sample site.yml
$ cp group_vars/all.sample group_vars/all
$ cp group_vars/mons.sample group_vars/mons
$ cp group_vars/osds.sample group_vars/osds
$ ln -s ~/hosts hosts

We will now customize the configuration by changing the following files and add the shows content to the top of the file:

group_vars/all

cluster_network: 10.3.0.0/24
generate_fsid: 'true'
ceph_origin: distro
monitor_interface: eth0
public_network: 10.3.0.0/24
journal_size: 5120
group_vars/mons
cephx: false

group_vars/mons

cephx: false

group_vars/osds

journal_collocation: true
cephx: false
devices:
   - /dev/vdb

After these changes are made, you have setup to use Ceph packages as provided by the distribution, and to deploy as a storage cluster on 10.3.0.0/24.

To deploy the storage cluster, all we need to do now is to run the playbook named site.yml and skipping the tags named with_pkg and package-install.

$ ansible-playbook -i hosts site.yml --skip-tags with_pkg,package-install

After this finishes a storage cluster should be available.

Conclusion

This article showed how it is possible to customize an Atomic host image for a specific use-case and use this for deployment of a simple Ceph cluster. Using ostree it is possible to move between different trees, allowing a collection of packages to be upgraded in an atomic fashion. And when there is a failure for some reason, it is possible to rollback to the previous tree and allow the node to function in the former state.

This is part of the setup I usefor automated builds of Ceph packages and deployment tests. If you have questions about specific parts, please let me know on Twitter at @gbraad or by email. Input is appreciated as this can help with writing of future articles.

Setting up a powerful self-hosted IDE in the cloud

2016-09-06T00:00:00+08:00

Setting up infrastructure should be as simple as playing with LEGO. With the advent of Docker, IT has underwent a dramatic change. For instance, it became easy to setup throw-away development environments. These could then be used to create clean environments to perform builds in. And eventually, containers are starting to move into the general infrastructure . This is a transaction that is currently still happening, helped by tools as Docker Swarm and Kubernetes. In this short article I will show how these ideas can help in setting up a composable infrastructure, providing you with a powerful IDE in the cloud, full with Let's Encrypt certificate generation.

Cloud9

We will be deploying C9, or Cloud9 IDE, is a full IDE which can be accessed from your browser. For convenience of use, I created a containerized version of this some while back and have ever since been using it. Although,, C9 also has a hosted environment, I grew worried after Amazon acquired the company. Support has since been slower to respond and this made me look into other ways to provide an alternative approach. After some testing with Docker Cloud it seemed like a good idea to deploy a self-hosted alternative.

But first about one of the main components. The container I created exposes port 8181, which is an unsecured endpoint. It allows you to simply setup a local instance with:

$ alias c9ide='docker run -it --rm -v $PWD:/workspace gbraad/c9ide:u1604'
$ cd ~/Projects/gauth
$ c9ide

This would however work for a local environment and does not provide a secure solution. More was needed... such as secure solution and preferably automated. For this Nginx and Let's Encrypt is used. More information about Cloud9 can be found in my Knowledge Base article *.

Preparation

To deploy the environment I used Fedora 24. Most cloud providers provide this as an image. In my case, CityCloud provided the latest version after placing a simple request. You can consider any size, but the more storage and memory, the better the performance will be. I set up a 4G and 2 cores instance, with 50G of storage. After logging in, perform the following commands

$ sudo su -
$ dnf update -y
$ dnf install -y docker
$ setenforce 0
$ sed 's/SELINUX=targeted/SELINUX=permisssive/g' /etc/selinux/config
$ systemctl enable docker
$ systemctl start docker

Note: SELinux is disabled because we will have a volume attached to the IDE container where the workspace will be located. We can however fix this, but this is something for a future blog post.

Setup DNS

After this, you host machine should be ready to host containers to will make up our environment. before we continue, we need to be sure we have DNS configured that can point to the running containers that host the C9 IDE. I used CloudFlare and setup the following wildcard for my environment.

With this each hostname, such as ubuntu.c9ide.spotsnel.net would point to the same machine that hosts our development containers.

Deploy Nginx

To allow a hostname to be resolved to a 'local' end-point a reverse proxy is needed. Jason Wilder created a container for this purpose called nginx-proxy. It needs access to the docker.sock to be notified of changes and it will accordingly create an endpoint in the hosted Nginx. For more information about this, please read nginx-proxy's description on Docker hub.

# docker run -d -p 80:80 -p 443:443 \
   --name nginx-proxy \
   -v /path/to/certs:/etc/nginx/certs:ro \
   -v /etc/nginx/vhost.d \
   -v /usr/share/nginx/html \
   -v /var/run/docker.sock:/tmp/docker.sock:ro \
   jwilder/nginx-proxy

Opening any address pointing to this host would know result in a 503 error. This means that Nginx is listening and can not forward the request. We will deal with this later.

Deploy Let's Encrypt companion

Setting up a secure connection involves certificates which can be quite a hassle to obtain, and if trust of security is concerned, even very expensive. Luckily, initiatives exist that believe security should be available to all and Let's Encrypt provides a fully automated solution. Using the letsencrypt-nginx-proxy-companion container created by Yves Blusseau, it is possible allow the proxy container to be provisioned with the needed certificates. More information about this container can be found at Docker hub.

# docker run -d \
   --name nginx-proxy-letsencrypt \
   -v /path/to/certs:/etc/nginx/certs:rw \
   --volumes-from nginx-proxy \
   -v /var/run/docker.sock:/var/run/docker.sock:ro \
   jrcs/letsencrypt-nginx-proxy-companion

Unfortunately we do not have a way to really test this. But if the container starts correctly and does not terminate, it is likely working correctly, waiting for notifications of newly created containers.

Explanation

Containers that provide metadata in the form of environment variables, indicate to these service containers what the endpoint will be.

The nginx-proxy service uses VIRTUAL_HOST to know what the hostname endpoint will be. While the Let's Encrypt companion service uses LETSENCRYPT_HOST for this. In the rest of this article the endpoints will be the same. With LETSENCRYPT_EMAIL we indicate to who the certificate will be addressed.

C9 IDE

The container hosting C9 is available in different flavors. I have created them based on CentOS 7, Fedora 24 and Ubuntu Xenail (16.04). To prepare, it is a good idea to pull the images locally:

# docker pull gbraad/c9ide:c7
# docker pull gbraad/c9ide:f24
# docker pull gbraad/c9ide:u1604

To setup the containers as endpoints, I created the following helper script. It will deal with setting up the host and email for login to the container.

./create.sh [name] [host] [email] [password] [flavor] [volume]

#!/bin/sh -x
set -e

NAME=${1:-"test"}
HOST=${2:-${NAME}".local"}
USER=${3:-"nobody@local"}
PASS=${4:-"secrete"}
FLAVOR=${5:-"u1604"}  # c7, f24, u1604
VOLUME=${6:-"/workspaces/c9ide-"${NAME}}

mkdir -p ${VOLUME}
docker run -d \
   --name ${NAME} \
   -v ${VOLUME}:/workspace:rw \
   -e "VIRTUAL_HOST=${HOST}" \
   -e "LETSENCRYPT_HOST=${HOST}" \
   -e "LETSENCRYPT_EMAIL=${USER}" \
   -e "USERNAME=${USER}" \
   -e "PASSWORD=${PASS}" \
   gbraad/c9ide:${FLAVOR}

The USERNAME and PASSWORD are used to perform basic authentication. As you can see, the email address used for the certificates is also used for the username for login. This can of course customized. After this,

# chmod +x create.sh
# mkdir /workspaces
# ./create.sh ubuntu ubuntu.c9ide.spotsnel.net me@gbraad.nl verysecrete
# ./create.sh centos quentin.c9ide.spotsnel.net me@quentinyong.nl verysecrete c7

will create two containers, hosted at ubuntu.c9ide.spotsnel.net and quentin.c9ide.spotsnel.net. Notice that I used /workspaces to provide the storage location for all the host containers. Make sure this is hosted on a separate mountpoint for instance.

Now open your IDE by opening https://ubuntu.c9ide.spotsnel.net in your browser. If all goes well, you will be created by a login prompt. The username is me@gbraad.nl and the password verysecrete.

In future articles I will detail more about the composability of infrastructure. if you enjoyed this article, consider tweeting about it. If you have comments, please find me online at @gbraad.

Hello, Internet!

2016-08-08T00:00:00+08:00

Today I decided to pick up blogging again and asked the question; what to use? I had a simple requirement; 'it has to support Markdown-formatted documents'. The reasoning behind this is simple; I want to be able to use Pandoc to generate other kinds of output if needed. Plus GitBook has an online editor which allows me to write the articles as if they are chapters in a book.

Chris Smart, who is the creator of the Korora project, suggested me to have a look at Pelican. So I did, and this resulted in the blog you are reading at the moment.

Publication is all automated using GitLab CI runners. The project for this is available at blog. For the moment it is still a test and I will try to theme it over the coming days. Anyways, progress can be tracked here...

Software Distribution for a new era

2016-07-30T00:00:00+08:00

In this presentation I will introduce Software Distribution for a new era. In short it will explain the negatives of using packages, how 'containers' improved on this, but now how to make this available for use on servers and desktops.

In short it will talk about the following topics:

container and Docker
FlatPak
OSTree
Project Atomic (Fedora, CentOS)

Presentation

Your browser does not support iframes.

Feedback

If you have any suggestion, please discuss below or send me an email.

Note: the original presentation can be found at: Software Distribution for a new era

Deploying OpenStack using TripleO

2016-05-18T00:00:00+08:00

In this article we will deploy an OpenStack environment using TripleO using the tripleo-quickstart scripts. It will create a virtualized environment which consists of 1 Undercloud node and in total 9 nodes in the Overcloud; 3 controller nodes (High Availability), 3 compute nodes and 3 Ceph storage nodes.

What is TripleO, undercloud and overcloud?

TripleO is a deployment method in which a dedicated OpenStack management environment is used to deploy another OpenStack environment which is used for the workload. The management environment is contained in a single machine, called the undercloud. This OpenStack environment takes care of the monitoring and management of what is known as the overcloud. The overcloud is the actual OpenStack environment that will run the workload.

Prerequisites

It is preferred to use a dedicated server for the following instructions. Consider using 32G of memory and have enough diskspace available in /home. This target node has to be using CentOS 7 (or RHEL7).

Prepare for deployment

The deployment can be done from a workstation targeting a server, or from the server itself. Whatever your preferred method is, you will need the tripleo-quickstart scripts.

$ curl -O https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/quickstart.sh
$ chmod u+x quickstart.sh
$ ./quickstart.sh --install-deps

This will install the dependencies used by the quickstart.sh script. Now you need to prepare the target node. Make sure you can login to this server without a password.

$ export VIRTHOST=[target]
$ ssh-copy-id root@$VIRTHOST
$ ssh root@$VIRTHOST uname -a

This article witll not describe the setup of passwordless or key-based authentication with SSH. If issues occur, please verify you have a generated keyset. If not, generate one with ssh-keygen. For further information please consult the man page or verify online.

Cache deployment images locally

Although this step is not necessary, you can pre-download the images and cache them locally. This can be helpful if you want to perform the deployment using different images and/or suffer from bad connectivity.

The location if the images is currently at http://artifacts.ci.centos.org/rdo/images/{{release}}/delorean/stable/undercloud.qcow2.

To download the mitaka image, you can do this with

$ mkdir /var/lib/oooq-images
$ curl http://artifacts.ci.centos.org/rdo/images/mitaka/delorean/stable/undercloud.qcow2 -o /var/lib/oooq-images/undercloud-mitaka.qcow2

Deployment configuration file

We will create a virtual environment for out TripleO deployment. This will provide you with the knowledge you need to perform a bare-metal installation.

You can define the node deployment inside a configuration file, eg. called deploy-config.yml. This will contain the following:

overcloud_nodes:
  - name: control_0
    flavor: control
  - name: control_1
    flavor: control
  - name: control_2
    flavor: control

  - name: compute_0
    flavor: compute
  - name: compute_1
    flavor: compute
  - name: compute_2
    flavor: compute

  - name: storage_0
    flavor: ceph
  - name: storage_1
    flavor: ceph
  - name: storage_2
    flavor: ceph

extra_args: "--control-scale 3 --ceph-storage-scale 3 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --ntp-server pool.ntp.org"

Nodes can be assigned a role by setting the flavor:

control sets up a controller node, which also handles the network.
compute sets up a Nova compute node.
ceph sets up a node for Ceph storage.

The extra arguments allow you to modify the deployment that is performed.

--control-scale 3 instructs the deployment to assign 3 nodes with the controller role.
--ceph-storage-scale 3 instructs the deployment to assign 3 nodes to be used for the Ceph storage backend.
-e puppet-pacemaker.yaml will setup pacemaker HA for the controller nodes
--ntp-server pool.ntp.org will sync time on the nodes using NTP.

Note: if you want to use all compute nodes at once, include --compute-scale 3. But in this article I am using these additional nodes for scale out.

Perform deployment

Now you can perform the undercloud deployment using:

$ ./quickstart.sh --config deploy-config.yml --undercloud-image-url file:///var/lib/oooq-images/undercloud-mitaka.qcow2 $VIRTHOST

But before you do, please continue reading about the Deployment scripts.

The previous command will target the node as specified with the $VIRTHOST environment variable, and according to the deploy-config.yml which we defined earlier.

It will login to this node and create a stack user which will be running the virtual machines. Later we will inspect this. After creating the virtual machines it will prepare the undercloud machine. After this, you still need to start the actual deployment.

Deployment scripts

To login to the undercloud:

$ ssh -F $OPT_WORKDIR/ssh.config.ansible undercloud

The undercloud is not fully prepared, you would have to do so with the following scripts.

Undercloud (management)

undercloud-install.sh will run the undercloud install and execute diskimage elements.
undercloud-post-install.sh will perform all pre-deploy steps, such as uploading the images to glance

Overcloud (workload)

overcloud-deploy.sh will deploy the overcloud, creating a heat stack and will use the nodes as defined in instack-env.json and the extra arguments given in the deployment configuration.
overcloud-deploy-post.sh will do any post-deploy configuration such as writing a local /etc/hosts file.
overcloud-validate.sh will run post-deploy validation, like a pingtest and possible tempest

You can run these scripts one by one... or install the whole undercloud and overcloud using the command:

$ ./quickstart.sh --tags all --config deploy-config.yml --undercloud-image-url file:///var/lib/oooq-images/undercloud-mitaka.qcow2 $VIRTHOST

Using --tags all will instruct ansible to perform all the steps and scripts as previously described. I suggest you to run the steps first each one by one and look into the scripts itself to understand how they interact with python-tripleoclient (eg. openstack undercloud and openstack overcloud).

[stack@undercloud ~]$ ./undercloud-install.sh
[stack@undercloud ~]$ ./undercloud-post-install.sh
[stack@undercloud ~]$ ./overcloud-deploy.sh
[stack@undercloud ~]$ ./overcloud-deploy-post.sh
[stack@undercloud ~]$ ./overcloud-validate.sh

Undercloud node

After running these commands, you will have a fully deployed environment. You can verify this from the undercloud node.

[stack@undercloud ~]$ . stackrc
[stack@undercloud ~]$ ironic node-list

+--------------------------------------+-----------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name      | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------+--------------------------------------+-------------+--------------------+-------------+
| 5c4f1ad5-3cea-41c2-8fa8-f3468e660447 | control-0 | aea0add0-f638-4a41-97ca-a2a64ac083a5 | None        | active             | True        |
| 0a0bf5e8-e903-4f77-be35-0a49f4da5109 | control-1 | 75778a5e-5e65-48bc-9934-d4cb203fad86 | None        | active             | True        |
| 12aa12d2-0023-48ac-89d2-4e138f6eef08 | control-2 | f74b2b00-0c01-44a5-917e-45fff058f2fa | None        | active             | True        |
| 6a0533a2-d91f-4f45-acbb-5f5f231c8986 | compute-0 | None                                 | None        | available          | True        |
| e8663954-4da8-4027-9226-f9f053f269d9 | compute-1 | 113afdbd-0afa-481c-a744-90276907b8e2 | None        | active             | True        |
| 06679c73-97a2-4de0-b676-193a3e182fcf | compute-2 | None                                 | None        | available          | True        |
| a6ef4bce-941c-407d-966e-85df2af3f6e1 | storage-0 | bbba571f-f2a9-46f3-a270-f154e045c5a8 | None        | active             | True        |
| 6174dbb7-28e7-425e-b675-f1653f0b731c | storage-1 | d3afc0e8-70a0-43c4-b35a-693a7e4e5fa5 | None        | active             | True        |
| 32e8122b-3a5f-45fb-8a5a-60dc4f82325f | storage-2 | 53ea0799-1bcd-4f55-8671-e6a7f7f46054 | None        | active             | True        |
+--------------------------------------+-----------+--------------------------------------+-------------+--------------------+-------------+

This will show a list of nodes that are available in the environment. This information

Login to the overcloud

From the undercloud node you can source the stack resource file and use the openstack clients as usual.

[stack@undercloud ~]$ . overcloudrc
[stack@undercloud ~]$ cat overcloudrc
[stack@undercloud ~]$ nova list

In the previous output you also see the OS_AUTH_URL and the credentials needed to login from the Horizon dashboard.

Either using ssh portforwaring, or the dynamic proxy option, you can open the dashboard.

$ ssh -F ~/.quickstart/ssh.config.ansible undercloud -D 8080

Either using Firefox (with the FoxyProxy extension) or Chrome/Vivaldi (with the SwitchySharp extension) you can set a SOCKS proxy at 127.0.0.1 and port 8080.

Login to overcloud nodes

If you need to inspect a node in the overcloud (workload), you can login to these nodes from the undercloud using the following command:

[stack@undercloud ~]$ ssh [hostname/nodeip]

Note: you can find the hostnames and IP addresses on the undercloud in the /etc/hosts file. It will use 'heat-admin' as user to login.

Scale out

After deployment, you might have noticed that ironic node-list returned a list of baremetal nodes that are in Provisioning state 'available' and do not have an Instance UUID. These nodes have not been deployed to, as --compute-scale had not been set.

To scale out to these nodes, you first need to change the Maintenance status to false. You can do this for all available at once with:

[stack@undercloud ~]$ for i in $(ironic node-list | grep available | grep -v UUID | awk ' { print $2 } '); do
> ironic node-set-maintenance $i false;
> done

You can verify the status with ironic node-list.

After this you can scale out using

[stack@undercloud ~]$ openstack overcloud deploy --templates --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --timeout 60 --ntp-server pool.ntp.org --compute-scale 3

This will change the current overcloud heat deployment and provision the remaining nodes.

Eventually the command will return with:

...
Stack overcloud UPDATE_COMPLETE
Overcloud Endpoint: http://192.0.2.6:5000/v2.0
Overcloud Deployed

after which the new nodes have been added to the overcloud. To update the hosts entries in /etc/hosts you can rerun:

[stack@undercloud ~]$ ./overcloud-deploy-post.sh

Diskimage building

The undercloud images can be created using ansible-role-tripleo-image-build. Using the following commands it will generate the images:

$ git clone https://github.com/redhat-openstack/ansible-role-tripleo-image-build.git
$ cd ansible-role-tripleo-image-build/tests/pip
$ sudo ./build.sh -i
$ ./build.sh $VIRTHOST

After the command finishes succesfully, the images can be found in /var/lib/oooq-images.

Note: The content of /var/lib/oooq-images will be cleaned on run. After this it will download a base image from http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 of about 800M. You can download this image and specify the location in a configuration file to prevent it from having to be downloaded each time.

A create a file called: override.yml

artib_minimal_base_image_url: file:///var/lib/oooq-base-images/CentOS-7-x86_64-GenericCloud.qcow2

And pass this to the build command:

$ ./build.sh -e override.yml $VIRTHOST

Conclusion

Since the introduction of TripleO Quickstart, standing up a multi-node OpenStack environment is very easy. In my knowledge-base article I describe also how you can do a baremetal deployment using the Quickstart.

If you have any suggestion, please discuss below or send me an email.

Note: the original publication can be found at: OpenStack hands-on-labs

More information

Deployment configuration options
Knowledge-Base for development/architecture notes on TripleO
Openstack deployment using RDO-Manager

Building a multi-tier application using OpenStack (PackStack)

2016-03-05T00:00:00+08:00

This is a publication of an article/training class I gave related to setting up an environment using OpenStack, to host a multi-tier applicatiom.

Introduction

In this hands-on-labs you will learn how to set-up an OpenStack environment to host a multi-tier application; front-end, back-end and the related network settings to prevent access to back-end servers, etc.

To setup the environment quickly, we will be using PackStack. PackStack is an installation utility to quickly deploy an OpenStack cloud. In our case we will use the all-in-one solutiion which allows a single node environment to test deploying our multi-tier application.

Setup packstack environment

Use a RHEL or CentOS 7 installation.

$ systemctl disable NetworkManager
$ systemctl enable network
$ systemctl stop NetworkManager.service
$ systemctl start network.service

$ yum install -y https://www.rdoproject.org/repos/rdo-release.rpm
$ sudo yum update -y
$ sudo yum install -y openstack-packstack
$ packstack --allinone --os-neutron-lbaas-install=y

Check environment

In order to start using OpenStack, you’ll need to authenticate as a tenant. To do this, run the following command in order to put the demo user’s credentials in your environment.

$ source ~/keystonerc_admin

The installation automatically creates two networks for you ‘private’ and ‘public’. The ‘public’ network we’ll use to allocate floating ips out of later. Running openstack network list will show this.

$ openstack network list

+--------------------------------------+---------+--------------------------------------+
| ID                                   | Name    | Subnets                              |
+--------------------------------------+---------+--------------------------------------+
| 84aff6b0-2291-41b5-9871-d3d24906e358 | private | 92432fb8-8c29-4abe-98d8-de8bf161a18b |
| 427becab-54af-4b43-a5d2-e292b13b6a86 | public  | 78eff45a-25f2-4904-bab8-a8795d9a7f9b |
+--------------------------------------+---------+--------------------------------------+

Setup security groups

First we’ll create the three security groups we’ll need to contain the members: web, database and ssh.

$ openstack security group create web

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | web                                  |
| id          | a98fcd2f-a828-4a88-92aa-36e3c1223a92 |
| name        | web                                  |
| rules       | []                                   |
| tenant_id   | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------+--------------------------------------+

$ openstack security group create database

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | database                             |
| id          | cf6c0380-e255-4ba8-9258-bb8e9c062fa7 |
| name        | database                             |
| rules       | []                                   |
| tenant_id   | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------+--------------------------------------+

$ openstack security group create ssh

+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| description | ssh                                  |
| id          | 141ed0d0-c004-457d-8efa-45e0fd2dc986 |
| name        | ssh                                  |
| rules       | []                                   |
| tenant_id   | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------+--------------------------------------+

$ openstack security group list

+--------------------------------------+----------+------------------------+
| ID                                   | Name     | Description            |
+--------------------------------------+----------+------------------------+
| cf6c0380-e255-4ba8-9258-bb8e9c062fa7 | database | database               |
| 379b58b2-7ca3-431e-ae1f-cd6a627a9b30 | default  | Default security group |
| 141ed0d0-c004-457d-8efa-45e0fd2dc986 | ssh      | ssh                    |
| a98fcd2f-a828-4a88-92aa-36e3c1223a92 | web      | web                    |
+--------------------------------------+----------+------------------------+

Now we’ll add rules into these security groups for their desired functionality.

Allow all HTTP traffic on port 80 to the web security group:

$ neutron security-group-rule-create --direction ingress --protocol TCP \ 
> --port-range-min 80 --port-range-max 80 web

Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | b293d93a-30c2-4854-a890-5ce65639f870 |
| port_range_max    | 80                                   |
| port_range_min    | 80                                   |
| protocol          | tcp                                  |
| remote_group_id   |                                      |
| remote_ip_prefix  |                                      |
| security_group_id | a98fcd2f-a828-4a88-92aa-36e3c1223a92 |
| tenant_id         | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------------+--------------------------------------+

Allow database servers to be accessed from the web servers:

$ neutron security-group-rule-create --direction ingress --protocol TCP \
> --port-range-min 3306 --port-range-max 3306 --remote-group-id web database

Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | dfc77fb6-48a0-41ca-9230-50c2cda27c63 |
| port_range_max    | 3306                                 |
| port_range_min    | 3306                                 |
| protocol          | tcp                                  |
| remote_group_id   | a98fcd2f-a828-4a88-92aa-36e3c1223a92 |
| remote_ip_prefix  |                                      |
| security_group_id | cf6c0380-e255-4ba8-9258-bb8e9c062fa7 |
| tenant_id         | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------------+--------------------------------------+

Allow the jump host to ssh into both the database servers and webservers

$ neutron security-group-rule-create --direction ingress --protocol TCP \ 
> --port-range-min 22 --port-range-max 22 --remote-group-id ssh database

Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 0c686a2c-304f-42be-9936-cdce46963d46 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| protocol          | tcp                                  |
| remote_group_id   | 141ed0d0-c004-457d-8efa-45e0fd2dc986 |
| remote_ip_prefix  |                                      |
| security_group_id | cf6c0380-e255-4ba8-9258-bb8e9c062fa7 |
| tenant_id         | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------------+--------------------------------------+

$ neutron security-group-rule-create --direction ingress --protocol TCP \
> --port-range-min 22 --port-range-max 22 --remote-group-id ssh web

Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | 919a6ede-8dfd-4184-bf2a-f07c0527d5bf |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| protocol          | tcp                                  |
| remote_group_id   | 141ed0d0-c004-457d-8efa-45e0fd2dc986 |
| remote_ip_prefix  |                                      |
| security_group_id | a98fcd2f-a828-4a88-92aa-36e3c1223a92 |
| tenant_id         | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------------+--------------------------------------+

Allow the outside world to be able to ssh into the jump host on port 22:

$ neutron security-group-rule-create --direction ingress --protocol tcp \
> --port-range-min 22 --port-range-max 22 ssh

Created a new security_group_rule:
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| id                | fb8dcbe6-e553-4a92-aed4-aca7f086dca4 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| protocol          | tcp                                  |
| remote_group_id   |                                      |
| remote_ip_prefix  |                                      |
| security_group_id | 141ed0d0-c004-457d-8efa-45e0fd2dc986 |
| tenant_id         | 3d44af649a1c42fcaa102ed11e3f010f     |
+-------------------+--------------------------------------+

Setup virtual machines

Now we can boot some virtual machines that will make use of these security groups. Run openstack net work list to obtain the private network uuid that we are going to be using:

$ openstack network list

+--------------------------------------+---------+--------------------------------------+
| ID                                   | Name    | Subnets                              |
+--------------------------------------+---------+--------------------------------------+
| 427becab-54af-4b43-a5d2-e292b13b6a86 | public  | 78eff45a-25f2-4904-bab8-a8795d9a7f9b |
| 84aff6b0-2291-41b5-9871-d3d24906e358 | private | 92432fb8-8c29-4abe-98d8-de8bf161a18b |
+--------------------------------------+---------+--------------------------------------+

Then, we’ll run openstack image list to determine the images available to boot our instances with. Since we’re using packstack, the script automatically uploaded an image to glance for us to use.

$ openstack image list

+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| eea0e326-8e2e-41db-80a0-1138a4bdd5a6 | cirros |
+--------------------------------------+--------+

Overview

In the next steps we will boot four instances:

2 web servers
1 database server
1 ssh jump host

We will be creating instances using the smallest flavor available.

$ openstack flavor list

+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1  | m1.tiny   |   512 |    1 |         0 |     1 | True      |
| 2  | m1.small  |  2048 |   20 |         0 |     1 | True      |
| 3  | m1.medium |  4096 |   40 |         0 |     2 | True      |
| 4  | m1.large  |  8192 |   80 |         0 |     4 | True      |
| 5  | m1.xlarge | 16384 |  160 |         0 |     8 | True      |
+----+-----------+-------+------+-----------+-------+-----------+

So we will be using flavor 1, which means instances are created with 512MB of memory and a disk of 1G.

Note:
We also have to make sure that each instances has an IP address on the private network. For this we are including the --nic net-id= option specifying the network ID of the private network.

Setup web servers

Boot two instances named web_server1 and web_server2 on the private network using the cirros image and part of the web security group:

$ nova boot --image cirros --nic net-id=84aff6b0-2291-41b5-9871-d3d24906e358 \
> --security_groups web --flavor 1 web_server1

+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | rijM8RvVKXhd                                  |
| config_drive                         |                                               |
| created                              | 2016-02-25T08:21:23Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | be6ec624-07cd-45c1-8260-211f1f2fd786          |
| image                                | cirros (eea0e326-8e2e-41db-80a0-1138a4bdd5a6) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | web_server1                                   |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | web                                           |
| status                               | BUILD                                         |
| tenant_id                            | 3d44af649a1c42fcaa102ed11e3f010f              |
| updated                              | 2016-02-25T08:21:24Z                          |
| user_id                              | a72ce317d35c47e8b8274995d0a2af92              |
+--------------------------------------+-----------------------------------------------+

$ nova boot --image cirros --nic net-id=84aff6b0-2291-41b5-9871-d3d24906e358 \ 
> --security_groups web --flavor 1 web_server2

+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | vyT4575gsqth                                  |
| config_drive                         |                                               |
| created                              | 2016-02-25T08:22:53Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | 146056ad-e8dc-4ad3-8765-97b753f3d040          |
| image                                | cirros (eea0e326-8e2e-41db-80a0-1138a4bdd5a6) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | web_server2                                   |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | web                                           |
| status                               | BUILD                                         |
| tenant_id                            | 3d44af649a1c42fcaa102ed11e3f010f              |
| updated                              | 2016-02-25T08:22:53Z                          |
| user_id                              | a72ce317d35c47e8b8274995d0a2af92              |
+--------------------------------------+-----------------------------------------------+

Setup database server

Boot database server

$ nova boot --image cirros --nic net-id=84aff6b0-2291-41b5-9871-d3d24906e358 \
> --security_groups database --flavor 1 database_server

+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | 9GJgxdvz3eFQ                                  |
| config_drive                         |                                               |
| created                              | 2016-02-25T08:23:22Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | d66ced0e-3aaf-4c14-8921-229ac6307ecd          |
| image                                | cirros (eea0e326-8e2e-41db-80a0-1138a4bdd5a6) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | database_server                               |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | database                                      |
| status                               | BUILD                                         |
| tenant_id                            | 3d44af649a1c42fcaa102ed11e3f010f              |
| updated                              | 2016-02-25T08:23:22Z                          |
| user_id                              | a72ce317d35c47e8b8274995d0a2af92              |
+--------------------------------------+-----------------------------------------------+

Setup jumphost server

Boot ssh jump host

$ nova boot --image cirros --nic net-id=84aff6b0-2291-41b5-9871-d3d24906e358 \
> --security_groups ssh --flavor 1 jumphost

+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000004                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | jwbUXkmfEK7Y                                  |
| config_drive                         |                                               |
| created                              | 2016-02-25T08:23:54Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | e540896e-e148-414a-9588-3b83d3f2b059          |
| image                                | cirros (eea0e326-8e2e-41db-80a0-1138a4bdd5a6) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | jumphost                                      |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | ssh                                           |
| status                               | BUILD                                         |
| tenant_id                            | 3d44af649a1c42fcaa102ed11e3f010f              |
| updated                              | 2016-02-25T08:23:54Z                          |
| user_id                              | a72ce317d35c47e8b8274995d0a2af92              |
+--------------------------------------+-----------------------------------------------+

Client

We will also create a client instance that we will use to access the web servers from.

Note: Since we did not specify a security group this instance will be part of a default security group which allows the instance to make outgoing connections to anyone but only accept incoming connections from members of this same security group.

$ nova boot --image cirros --nic net-id=84aff6b0-2291-41b5-9871-d3d24906e358 \
> --flavor 1 client

+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000005                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | c63QG7iE3PrY                                  |
| config_drive                         |                                               |
| created                              | 2016-02-25T08:24:27Z                          |
| flavor                               | m1.tiny (1)                                   |
| hostId                               |                                               |
| id                                   | 8e0179a3-6bf8-4c07-8035-f8916ca3183d          |
| image                                | cirros (eea0e326-8e2e-41db-80a0-1138a4bdd5a6) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | client                                        |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 3d44af649a1c42fcaa102ed11e3f010f              |
| updated                              | 2016-02-25T08:24:27Z                          |
| user_id                              | a72ce317d35c47e8b8274995d0a2af92              |
+--------------------------------------+-----------------------------------------------+

Check virtual machines

Running openstack server list will display the status of the instances. After a few seconds all of the instances should go to an ACTIVE status.

$ openstack server list

+--------------------------------------+-----------------+--------+------------------+
| ID                                   | Name            | Status | Networks         |
+--------------------------------------+-----------------+--------+------------------+
| 8e0179a3-6bf8-4c07-8035-f8916ca3183d | client          | ACTIVE | private=10.0.0.7 |
| e540896e-e148-414a-9588-3b83d3f2b059 | jumphost        | ACTIVE | private=10.0.0.6 |
| d66ced0e-3aaf-4c14-8921-229ac6307ecd | database_server | ACTIVE | private=10.0.0.5 |
| 146056ad-e8dc-4ad3-8765-97b753f3d040 | web_server2     | ACTIVE | private=10.0.0.4 |
| be6ec624-07cd-45c1-8260-211f1f2fd786 | web_server1     | ACTIVE | private=10.0.0.3 |
+--------------------------------------+-----------------+--------+------------------+

Setup public IP address for SSH

To make the jumphost publicly accessible on the internet we’ll need to assign a floating IP to it. To do this first create a floating IP via:

$ neutron floatingip-create public

Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 172.24.4.228                         |
| floating_network_id | 427becab-54af-4b43-a5d2-e292b13b6a86 |
| id                  | ce6efd31-97e9-428b-a3ac-b5a14e77a305 |
| port_id             |                                      |
| router_id           |                                      |
| status              | DOWN                                 |
| tenant_id           | 3d44af649a1c42fcaa102ed11e3f010f     |
+---------------------+--------------------------------------+

Next, we need to determine the port id of the jumpbox:

$ neutron port-list

+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                           |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 551cf7bc-802f-4b02-bd3e-c44473ffb1ff |      | fa:16:3e:48:a4:d1 | {"subnet_id": "78eff45a-25f2-4904-bab8-a8795d9a7f9b", "ip_address": "172.24.4.226"} |
| 5d98aa79-8bc0-4512-a128-078281aae2bc |      | fa:16:3e:0d:4b:55 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.6"}     |
| 6aab05f6-9176-48b4-9b0f-113593945593 |      | fa:16:3e:2d:b6:a4 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.1"}     |
| 8941a204-08e7-4547-b720-f9840358929b |      | fa:16:3e:5b:a3:c7 | {"subnet_id": "78eff45a-25f2-4904-bab8-a8795d9a7f9b", "ip_address": "172.24.4.228"} |
| 9bd695dc-4e6c-40c5-952d-44269711bd6c |      | fa:16:3e:9e:36:8f | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.3"}     |
| afd18b4f-c19f-4c03-a049-fe8aeae7da49 |      | fa:16:3e:ac:94:5e | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.5"}     |
| e63d8edb-f10c-4776-a10e-cba9ec3f3d56 |      | fa:16:3e:d4:9c:57 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.4"}     |
| f0d35941-989c-4f2b-b187-872445b0b653 |      | fa:16:3e:0c:7f:d5 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.7"}     |
| f9e097b4-4227-49ee-9442-643c4186e587 |      | fa:16:3e:a5:36:2c | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.2"}     |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+

and find the id that matches the IP address of the jumphost (10.0.0.6) and associate it via:

# neutron floatingip-associate [floating_ip id] [port-list id]
$ neutron floatingip-associate ce6efd31-97e9-428b-a3ac-b5a14e77a305 \
> 5d98aa79-8bc0-4512-a128-078281aae2bc

Associated floating IP ce6efd31-97e9-428b-a3ac-b5a14e77a305

$ neutron floatingip-list

+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| ce6efd31-97e9-428b-a3ac-b5a14e77a305 | 10.0.0.6         | 172.24.4.228        | 5d98aa79-8bc0-4512-a128-078281aae2bc |
+--------------------------------------+------------------+---------------------+--------------------------------------+

Verify SSH connectivity

Now you should be able to ssh to the jumbox via with password cubswin:) :

$ ssh cirros@172.24.4.228

The authenticity of host '172.24.4.3 (172.24.4.3)' can't be established.
RSA key fingerprint is 8b:90:ae:9c:eb:be:83:ae:3c:33:fe:84:7a:88:12:d1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.24.4.3' (RSA) to the list of known hosts.
cirros@172.24.4.3's password:
$

After logging into the jumpbox you’ll be able to ssh into your webserver1, webserver2, and database server via:

$ ssh 10.0.0.3

Host '10.0.0.3' is not in the trusted hosts file.
(fingerprint md5 e7:c3:d2:93:0b:eb:65:6d:e6:01:2e:b3:e6:72:fd:c0)
Do you want to continue connecting? (y/n) y
cirros@10.0.0.3's password:
$ exit

$ ssh 10.0.0.4

...

$ ssh 10.0.0.5

...

None of those instances will be able to ssh to each other. The point of this instance is so that you do not need to have all of your other instances publicly addressable and directly accessible via the internet.

Simulate web server

Now let’s log in to web_server1 and web_server2 (via ssh or via horizon) and setup a simple web server to handle requests and reply with who they are:

# On web_server 1 (10.0.0.3)
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\n'`hostname` | sudo nc -l -p 80 ; done &
$ exit

# on web_server 2 (10.0.0.4)
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\n'`hostname` | sudo nc -l -p 80 ; done &
$ exit

Simulate HTTP request

Now, log in to your client virtual machine (from the web console). From there if you run:

$ wget -O - http://10.0.0.3/

Connecting to 10.0.0.3 (10.0.0.3:80)
web-server1
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://10.0.0.4/

Connecting to 10.0.0.4 (10.0.0.4:80)
web-server2
               100% |************************************| 12 0:00:00 ETA

This demonstrates that our simple web server is working on our two web server instances.

Optional check

We can demonstrate that the web security group is working correctly by killing our simple web server and changing the port number. (Note: to kill the web server you may need to hold control + c for a second in order for it to break out of the while loop before another instance of nc is created.)

# On web_server 1 
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\n'`hostname` | sudo nc -l -p 81 ; done &

Now on the client run:

$ wget -O - http://10.0.0.3:81

Connecting to 10.0.0.3 (10.0.0.3:81)
wget: can't connect to remote host (10.0.0.3): Connection timed out

As you can see the request is never answered as expected because our web security group does not allow port 81 ingress. Now let’s set the web server to run on port 80 again.

Provision loadbalancer

At this point were going to provision loadbalancer via neutron in order to load balance requests between our two web server instances.

$ neutron subnet-list

+--------------------------------------+----------------+-----------------+--------------------------------------------------+
| id                                   | name           | cidr            | allocation_pools                                 |
+--------------------------------------+----------------+-----------------+--------------------------------------------------+
| 78eff45a-25f2-4904-bab8-a8795d9a7f9b | public_subnet  | 172.24.4.224/28 | {"start": "172.24.4.226", "end": "172.24.4.238"} |
| 92432fb8-8c29-4abe-98d8-de8bf161a18b | private_subnet | 10.0.0.0/24     | {"start": "10.0.0.2", "end": "10.0.0.254"}       |
+--------------------------------------+----------------+-----------------+--------------------------------------------------+

Create a loadbalancer pool:

$ neutron lb-pool-create --name http-pool --lb-method ROUND_ROBIN \
> --protocol HTTP --subnet-id 92432fb8-8c29-4abe-98d8-de8bf161a18b

Created a new pool:
+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| admin_state_up         | True                                 |
| description            |                                      |
| health_monitors        |                                      |
| health_monitors_status |                                      |
| id                     | 51ca1962-a24a-4f43-920f-162a03a45c51 |
| lb_method              | ROUND_ROBIN                          |
| members                |                                      |
| name                   | http-pool                            |
| protocol               | HTTP                                 |
| provider               | haproxy                              |
| status                 | PENDING_CREATE                       |
| status_description     |                                      |
| subnet_id              | 92432fb8-8c29-4abe-98d8-de8bf161a18b |
| tenant_id              | 3d44af649a1c42fcaa102ed11e3f010f     |
| vip_id                 |                                      |
+------------------------+--------------------------------------+

You can verify the creation of the http-pool with the neutron lb-pool-list and neutron lb-pool-show http-pool command.

$ neutron lb-pool-list

+--------------------------------------+-----------+----------+-------------+----------+----------------+--------+
| id                                   | name      | provider | lb_method   | protocol | admin_state_up | status |
+--------------------------------------+-----------+----------+-------------+----------+----------------+--------+
| 51ca1962-a24a-4f43-920f-162a03a45c51 | http-pool | haproxy  | ROUND_ROBIN | HTTP     | True           | ACTIVE |
+--------------------------------------+-----------+----------+-------------+----------+----------------+--------+

$ neutron lb-pool-show http-pool

+------------------------+--------------------------------------+
| Field                  | Value                                |
+------------------------+--------------------------------------+
| admin_state_up         | True                                 |
| description            |                                      |
| health_monitors        |                                      |
| health_monitors_status |                                      |
| id                     | 51ca1962-a24a-4f43-920f-162a03a45c51 |
| lb_method              | ROUND_ROBIN                          |
| members                |                                      |
| name                   | http-pool                            |
| protocol               | HTTP                                 |
| provider               | haproxy                              |
| status                 | ACTIVE                               |
| status_description     |                                      |
| subnet_id              | 92432fb8-8c29-4abe-98d8-de8bf161a18b |
| tenant_id              | 3d44af649a1c42fcaa102ed11e3f010f     |
| vip_id                 |                                      |
+------------------------+--------------------------------------+

Now, let’s create a health monitor, which checks to make sure our instances are still running and associate that with the pool:

$ neutron lbaas-healthmonitor-create --delay 3 --type HTTP --max-retries 3 \
> --timeout 3 --pool webserver-pool

Adding loadbalancer members

$ neutron lb-member-create --address 10.0.0.3 --protocol-port 80 http-pool

Created a new member:
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| address            | 10.0.0.3                             |
| admin_state_up     | True                                 |
| id                 | 36b01f84-a273-417f-9588-15cceea18868 |
| pool_id            | 51ca1962-a24a-4f43-920f-162a03a45c51 |
| protocol_port      | 80                                   |
| status             | PENDING_CREATE                       |
| status_description |                                      |
| tenant_id          | 3d44af649a1c42fcaa102ed11e3f010f     |
| weight             | 1                                    |
+--------------------+--------------------------------------+

$ neutron lb-member-create --address 10.0.0.4 --protocol-port 80 http-pool

Created a new member:
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| address            | 10.0.0.4                             |
| admin_state_up     | True                                 |
| id                 | 5d68b992-d09f-408b-8049-353e702cc990 |
| pool_id            | 51ca1962-a24a-4f43-920f-162a03a45c51 |
| protocol_port      | 80                                   |
| status             | PENDING_CREATE                       |
| status_description |                                      |
| tenant_id          | 3d44af649a1c42fcaa102ed11e3f010f     |
| weight             | 1                                    |
+--------------------+--------------------------------------+

After this you can verify the nodes have been added to the loadbalancer pool.

$ neutron lb-member-list

+--------------------------------------+----------+---------------+--------+----------------+--------+
| id                                   | address  | protocol_port | weight | admin_state_up | status |
+--------------------------------------+----------+---------------+--------+----------------+--------+
| 36b01f84-a273-417f-9588-15cceea18868 | 10.0.0.3 |            80 |      1 | True           | ACTIVE |
| 5d68b992-d09f-408b-8049-353e702cc990 | 10.0.0.4 |            80 |      1 | True           | ACTIVE |
+--------------------------------------+----------+---------------+--------+----------------+--------+

Add healthmonitor

We need to create a health monitor, which will check our instances to make sure they are still running

$ neutron lb-healthmonitor-create --delay 3 --type HTTP --max-retries 3 \
> --timeout 3

Created a new health_monitor:
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| admin_state_up | True                                 |
| delay          | 3                                    |
| expected_codes | 200                                  |
| http_method    | GET                                  |
| id             | 05cbf7f9-d01b-483b-934e-8955b14a1653 |
| max_retries    | 3                                    |
| pools          |                                      |
| tenant_id      | 3d44af649a1c42fcaa102ed11e3f010f     |
| timeout        | 3                                    |
| type           | HTTP                                 |
| url_path       | /                                    |
+----------------+--------------------------------------+

Now we have to associate the health monitor to the previously created loadbalancer pool.

$ neutron lb-healthmonitor-associate 05cbf7f9-d01b-483b-934e-8955b14a1653 http-pool

Associated health monitor 05cbf7f9-d01b-483b-934e-8955b14a1653

To make the loadbalancer and associated health monitor available, we need to assign it a Virual IP. The address will be allocated from the private subnet. This address will redirect the request to either instance within the pool to handle the request.

$ neutron lb-vip-create --name webserver-vip --protocol-port 80 \
> --protocol HTTP --subnet-id 92432fb8-8c29-4abe-98d8-de8bf161a18b http-pool

Created a new vip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 10.0.0.8                             |
| admin_state_up      | True                                 |
| connection_limit    | -1                                   |
| description         |                                      |
| id                  | b9ca33b0-b09c-4e7d-8ab7-77fe5207add6 |
| name                | webserver-vip                        |
| pool_id             | 51ca1962-a24a-4f43-920f-162a03a45c51 |
| port_id             | ecbadc53-5266-4146-bbb3-d1b6d383be10 |
| protocol            | HTTP                                 |
| protocol_port       | 80                                   |
| session_persistence |                                      |
| status              | PENDING_CREATE                       |
| status_description  |                                      |
| subnet_id           | 92432fb8-8c29-4abe-98d8-de8bf161a18b |
| tenant_id           | 3d44af649a1c42fcaa102ed11e3f010f     |
+---------------------+--------------------------------------+

Verify loadbalancer

Finally, let’s test out the loadbalancer. From the client instance we should be able to run wget at 10.0.0.8 and see that it loadbalancers our requests.

$ for i in $(seq 1 4) ; do wget -O - http://10.0.0.8/ ; done

Connecting to 10.0.0.8 (10.0.0.8:80)
web-server1
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 10.0.0.8 (10.0.0.8:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 10.0.0.8 (10.0.0.8:80)
web-server1
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 10.0.0.8 (10.0.0.8:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA

From the output above you can see that the the requests are being handled by web_server1 then web_server2 in an alternating fashion according to the round robin method.

Setup public IP address for web traffic

Now to make our VIP publicly accessible via the internet we need to create another floating IP:

$ neutron floatingip-create public

Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 172.24.4.229                         |
| floating_network_id | 427becab-54af-4b43-a5d2-e292b13b6a86 |
| id                  | 3c15f8a4-bdc6-4154-8bfa-e8b0674079ca |
| port_id             |                                      |
| router_id           |                                      |
| status              | DOWN                                 |
| tenant_id           | 3d44af649a1c42fcaa102ed11e3f010f     |
+---------------------+--------------------------------------+

Determine the port_id for the Virtual IP we created earlier:

$ neutron port-list

+--------------------------------------+------------------------------------------+-------------------+-------------------------------------------------------------------------------------+
| id                                   | name                                     | mac_address       | fixed_ips                                                                           |
+--------------------------------------+------------------------------------------+-------------------+-------------------------------------------------------------------------------------+
| 54b17392-dce7-4d6a-8cce-153fccb5d441 |                                          | fa:16:3e:f0:c7:cf | {"subnet_id": "78eff45a-25f2-4904-bab8-a8795d9a7f9b", "ip_address": "172.24.4.229"} |
| 551cf7bc-802f-4b02-bd3e-c44473ffb1ff |                                          | fa:16:3e:48:a4:d1 | {"subnet_id": "78eff45a-25f2-4904-bab8-a8795d9a7f9b", "ip_address": "172.24.4.226"} |
| 5d98aa79-8bc0-4512-a128-078281aae2bc |                                          | fa:16:3e:0d:4b:55 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.6"}     |
| 6aab05f6-9176-48b4-9b0f-113593945593 |                                          | fa:16:3e:2d:b6:a4 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.1"}     |
| 8941a204-08e7-4547-b720-f9840358929b |                                          | fa:16:3e:5b:a3:c7 | {"subnet_id": "78eff45a-25f2-4904-bab8-a8795d9a7f9b", "ip_address": "172.24.4.228"} |
| 9bd695dc-4e6c-40c5-952d-44269711bd6c |                                          | fa:16:3e:9e:36:8f | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.3"}     |
| afd18b4f-c19f-4c03-a049-fe8aeae7da49 |                                          | fa:16:3e:ac:94:5e | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.5"}     |
| e63d8edb-f10c-4776-a10e-cba9ec3f3d56 |                                          | fa:16:3e:d4:9c:57 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.4"}     |
| ecbadc53-5266-4146-bbb3-d1b6d383be10 | vip-b9ca33b0-b09c-4e7d-8ab7-77fe5207add6 | fa:16:3e:41:1a:eb | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.8"}     |
| f0d35941-989c-4f2b-b187-872445b0b653 |                                          | fa:16:3e:0c:7f:d5 | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.7"}     |
| f9e097b4-4227-49ee-9442-643c4186e587 |                                          | fa:16:3e:a5:36:2c | {"subnet_id": "92432fb8-8c29-4abe-98d8-de8bf161a18b", "ip_address": "10.0.0.2"}     |
+--------------------------------------+------------------------------------------+-------------------+-------------------------------------------------------------------------------------+

Associate VIP port with floating IP:

$ neutron floatingip-associate 3c15f8a4-bdc6-4154-8bfa-e8b0674079ca \
> ecbadc53-5266-4146-bbb3-d1b6d383be10

Associated floating IP 3c15f8a4-bdc6-4154-8bfa-e8b0674079ca

At this point the Virtual IP is a member of the default security group which does not allow ingress traffic unless you are also part of a security group which allows incoming traffic. We need to update the VIP to be a member of the web security group so that requests from the internet are allowed to pass (not just from our client instance).

Get the web security group uuid:

$ neutron security-group-list

+--------------------------------------+----------+--------------------------------------------------------------------------------+
| id                                   | name     | security_group_rules                                                           |
+--------------------------------------+----------+--------------------------------------------------------------------------------+
| 141ed0d0-c004-457d-8efa-45e0fd2dc986 | ssh      | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, 22/tcp                                                          |
| 379b58b2-7ca3-431e-ae1f-cd6a627a9b30 | default  | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, remote_group_id: 379b58b2-7ca3-431e-ae1f-cd6a627a9b30           |
|                                      |          | ingress, IPv6, remote_group_id: 379b58b2-7ca3-431e-ae1f-cd6a627a9b30           |
| 5e0d0d57-2df6-4fbc-ad18-5908aacdf799 | default  | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, remote_group_id: 5e0d0d57-2df6-4fbc-ad18-5908aacdf799           |
|                                      |          | ingress, IPv6, remote_group_id: 5e0d0d57-2df6-4fbc-ad18-5908aacdf799           |
| a98fcd2f-a828-4a88-92aa-36e3c1223a92 | web      | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, 22/tcp, remote_group_id: 141ed0d0-c004-457d-8efa-45e0fd2dc986   |
|                                      |          | ingress, IPv4, 80/tcp                                                          |
| b2c914cb-3bc1-4ee6-9f30-66c459af5f4c | default  | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, remote_group_id: b2c914cb-3bc1-4ee6-9f30-66c459af5f4c           |
|                                      |          | ingress, IPv6, remote_group_id: b2c914cb-3bc1-4ee6-9f30-66c459af5f4c           |
| cf6c0380-e255-4ba8-9258-bb8e9c062fa7 | database | egress, IPv4                                                                   |
|                                      |          | egress, IPv6                                                                   |
|                                      |          | ingress, IPv4, 22/tcp, remote_group_id: 141ed0d0-c004-457d-8efa-45e0fd2dc986   |
|                                      |          | ingress, IPv4, 3306/tcp, remote_group_id: a98fcd2f-a828-4a88-92aa-36e3c1223a92 |
+--------------------------------------+----------+--------------------------------------------------------------------------------+

Update VIP port to be a member of the web security group:

$ neutron port-update ecbadc53-5266-4146-bbb3-d1b6d383be10 --security_groups \
> list=true a98fcd2f-a828-4a88-92aa-36e3c1223a92

Updated port: ecbadc53-5266-4146-bbb3-d1b6d383be10

Verify loadbalancer

At this point your VIP is publicly addressable:

$ for i in $(seq 1 4) ; do wget -O - http://172.24.4.229/ ; done

Connecting to 172.24.4.229 (172.24.4.229:80)
web-server1
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server1
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA

To demonstration high availability, we’ll go and delete our web_server1 instance to simulate a failure.

$ openstack server delete web_server1

After the health monitor detects the host is not responding it will stop sending requests to web_server1. Now web_server2 is handling all the requests.

$ for i in $(seq 1 4) ; do wget -O - http://172.24.4.229/ ; done

Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA
Connecting to 172.24.4.229 (172.24.4.229:80)
web-server2
-                    100% |************************************| 12 0:00:00 ETA

Note: the first request might take longer to handle. This is because of the timeout before it notices the host is not responding.

Conclusion

As described in this article, it is easy to set up an environment with OpenStack in which you can securely host website, and deny access to your database servers.

In future articles I will talk about High Availability and automation of setting up an environment; for instance with HAProxy, GlusterFS and OpenStack Heat.

If you have any suggestion, please discuss below or send me an email.

Note: the original publication can be found at: OpenStack hands-on-labs

JavaScript Loaders

2014-11-04T00:00:00+08:00

Overview of JavaScript loaders

Trigger PhoneGap Build from GitHub

2014-11-04T00:00:00+08:00

You can remotely trigger a build by using the API

First you need to get your auth token:

$ curl -u me@gbraad.nl-X POST -d "" https://build.phonegap.com/token

Test it and retrieve your appid using the following command:

$ curl https://build.phonegap.com/api/v1/me?auth_token=[token]

After this go to your Github repo, open the admin and go to 'Service hooks'. Add a new WebHook URL (first option) and add yours https://build.phonegap.com/apps/[appid]/build/?auth_token=[token]

Save this and trigger it with 'Test hook'. When you would open: https://build.phonegap.com/apps/[appid]/builds you will probably see a new build.