The Dangers of Skipping

Long time, no write. This will change immediately.

Recently, I have getting myself familiar with OpenStack by way of the book OpenStack Cloud Computing Cookbook. For the most part, it was an easy read and I got my home OpenStack environment and running, albeit with some issues.

The book starts up with asking the reader to setup up a couple of Ubuntu 12.04 using VirtualBox. I decided to be clever and setup KVM instances instead, with fairly mixed results (At some point during the exploration, I decided to started over and went with some spare physical hardware  – just as well, I needed to redo my private lab anyway).

Next, I went and installed the following prerequisites per instruction on one single server:


sudo apt-get -y install rabbitmq-server nova-api nova-objectstore nova-scheduler nova-network nova-compute nova-cert glance qemu unzip.

Which installed OpenStack Essex, which is the default on Ubuntu 12.04. 

Then I setup  pressed parameters for MySQL server. Originally, it was MySQL 5.1

 
cat MYSQL_PRESEED | debconf-set-selections<
mysql-server-5.1 mysql-server/root_password password openstack
mysql-server-5.1 mysql-server/root_password_again password openstack
mysql-server-5.1 mysql-server/start_on_boot boolean true
MYSQL_PRESEED

However, Ubuntu 12.04.02 (which is what I am using) apparently install 5.5 by default, so some quick changes was in order:


cat «MYSQL_PRESEED | debconf-set-selections
mysql-server-5.5 mysql-server/root_password password openstack
mysql-server-5.5 mysql-server/root_password_again password openstack
mysql-server-5.5 mysql-server/start_on_boot boolean true
MYSQL_PRESEED

Then I changed the default config:

 
sudo apt-get update
sudo apt-get -y install mysql-server
sudo sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf

Then reset the default password:

 
MYSQL_PASS=openstack
mysql -uroot -p$MYSQL_PASS -e 'CREATE DATABASE nova;'
mysql -uroot -p$MYSQL_PASS -e "GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%'"
mysql -uroot -p$MYSQL_PASS -e "SET PASSWORD FOR 'nova'@'%' = PASSWORD('$MYSQL_PASS');"

Then I updated /etc/nova/nova.conf with the mySQL password credentials:

 
--sql_connection=mysql://nova:openstack@192.168.15.105/nova

The IP .105 being the IP of the Ubuntu box openstack1.

Then I added the next set of parameters:

 
--use_deprecated_auth
--s3_host=192.168.15.105
--rabbit_host=192.168.15.105
--ec2_host=192.168.15.105
--ec2_dmz_host=192.168.15.105
--public_interface=eth1
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=192.168.15.105:9292
--auto_assign_floating_ip=true
--scheduler_default_filters=AllHostsFilter

(incidentally, I am not getting into detail as the parameters – you can look them up at docs.openstack.org).

Then I sync the configs into the database.

 
sudo nova-manage db sync

Then i setup the network ranges.

 
sudo nova-manage network create vmnet --fixed_range_v4=10.0.0.0/8 --network_size=64 --bridge_interface=eth0
sudo nova-manage floating create --ip_range=192.168.15.0/24

The first set is for administrative access VMs to talk to each other as well to OpenStack. The other set is the public facing IP range (or at least public from a user standpoint).

Then I stopped and started the following services:

nova-compute 

nova-network 

nova-api 

nova-scheduler 

nova-objectstore 

nova-cert

libvirt-bin 

glance-registry 

glance-api

By way of brief explanation, nova-compute creates and destroy the instances, while nova-manage assigns and creates the IPs and VLANs. The nova-api provides access to services by the various application components, nova-scheduler runs the commands submitted by the various services. 

I am not sure about nova-objectstore, but I am sure that nova-vert encrypts the connection between the various services. libvirt-bin, of course, is the wrapper around KVM and provide access to the KVM hypervisor. Finally, glance-registry and glance-api registers and manage the images. 

At this point, I went and created a user, gave it admin access, and then created a project called “cookbook”:

 
sudo nova-manage user admin openstack
sudo nova-manage role add openstack cloudadmin
sudo nova-manage project create cookbook openstack

Then I zip up the cook:

 
sudo nova-manage project zipfile cookbook openstack

Then installed the tools on my Ubuntu) client necessary to manage openstack


sudo apt-get install euca2ools python-novaclient unzip

(later one, I used the Mac OS X version, which I will get to later on).

Copied the zip file from openstack1, unzip it into a directory called openstack, cd to it, then source an environment file so that I got the parameters into the shell.


. novarc

Finally, I generated a key and inserted into the database


nova keypair-add openstack > openstack.pem  

chmod 0600 *.pem

This allows me to setup a SSH key to whatever the default user of the instance I am created, so that I can simply run:


ssh -i openstack.pem username@instance name.

(the book had it was chmod 0600.pem, BTW, which is an obvious typo)

Finally, I was ready to update my image. So I downloaded a cloud version of Ubuntu’s server:


wget http://uec-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-i386.tar.gz

Installed the cloud_util tools:


sudo apt-get install cloud_utils

And then I attempted to upload the image …


cloud-publish-tarball ubuntu-12.04-server-cloudimg-i386.tar.gz images i386

and I ran into problems – kept running of space. I though the destination was the issue, so I re-did the server (at the time, it was a virtual server, not a physical). Eventually, turns out that it is running out space at the source – I had installed the Ubuntu server partitioned with separate file systems for /, /usr, /var, /tmp. By default, it extracts to /tmp, which was too small. So I change the default in the shell to:


TMPDIR=/var/tmp
TEMPDIR=/var/tmp
export TMPDIR ; export TEMPDIR

And afterwards, I was able to upload the image, which I was able to view with:


+———————————————————+——————————————————————————-+————+————+

| ID                                   | Name                                                | Status | Server |

+———————————————————+——————————————————————————-+————+————+

| 6e63d0cd-4e24-4766-ad48-3a01670a607e | images/precise-server-cloudimg-i386-vmlinuz-virtual | ACTIVE |        |

| bedf0e78-c7d4-414e-85fb-291a0ccd851d | images/precise-server-cloudimg-i386.img             | ACTIVE |        |

+———————————————————+——————————————————————————-+————+————+

So I ready to build an instance. 

I added the appropriate access to the ports:


nova secgroup-add-rule default tcp 22 22 0.0.0.0/0 
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

And then I went and “boot” or create my instances:


nova boot myInstance —image 0e2f43a8-e614-48ff-92bd-be0c68da19f4 —flavor 2 —key_name openstack 

And… it didn’t quite work. I mean, I can see the console output with:


nova console-log myInstance

But no IP was assigned, so I couldn’t login. After much reason, I found that the firewall rules was proving DHCP request from going through, so I added:


iptables -A POSTROUTING -t mangle -p udp —dport 68 -j CHECKSUM —checksum-fill

And from that point on, DHCP requests was going through and I was able to login.

At this point, I felt confident and decided to skip ahead and added another node to the group. 

Big mistake. But I am getting ahead of myself.

I went to chapter 11 and per instruction, installed just the following:


sudo apt-get -y installed nova-compute nova-network nova-apit

Copied the nova.conf to the new node, updating the IPs, verified on the original node that the services are listening:


root@openstack1:~# nova-manage service list

2013-04-27 22:38:46 DEBUG nova.utils [req-9dd11667-7967-4221-807f-98ddaf9371b3 None None] backend  from (pid=18080) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:662

Binary           Host                                 Zone             Status     State Updated_At

nova-cert        openstack1                           nova             enabled    :-)   2013-04-28 03:38:45

nova-scheduler   openstack1                           nova             enabled    :-)   2013-04-28 03:38:44

nova-network     openstack1                           nova             enabled    :-)   2013-04-28 03:38:45

nova-compute     openstack1                           nova             enabled    :-)   2013-04-28 03:38:38

nova-compute     openstack2                           nova             enabled    :-)   2013-04-28 03:38:39

nova-network     openstack2                           nova             enabled    :-)   2013-04-26 16:58:30

Feeling pretty sure that it will work as intended, I attempted to create more instances – and I failed. IPs failed to be assigned once again.

First of all, I didn’t setup my switch properly to handle the private network between the new nodes. I forgot my switch can only separate ports into separate broadcast domains using VLAN tagging (which OpenStack uses by default by way of VLAN Manager, which I didn’t pay much attention to – this will become significant shortly in this blog). After a while, I just gave up and plugged a crossover between openstack1 and openstack2 (later one, I did remember how to setup the switch properly and got the packets tagged appropriately). 

So at that point, I was able to create the instances. But then I couldn’t login using my private keys. Reviewing my console log, I found this (example pulled from google):


‘http://169.254.169.254/2009-04-04/meta-data/instance-id’ failed [50/120s]:  url error [timed out]

When you build an instance, it pulls the credentials from this loop back, which in turns is supposed to be routed to the API on the controller (which is on openstack1). I corrected that problem by adding the route:


sudo ip route add 169.254.0.0/16 metric 1000 dev eth1

openstack@openstack2:~$ ip route show

default via 192.168.15.1 dev eth1  metric 100 

169.254.0.0/16 dev eth1  scope link  metric 1000 

192.168.15.0/24 dev eth1  proto kernel  scope link  src 192.168.15.110 

192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 

But the problem still persists. 

Finally, after several long looks at the following pages:

http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/

http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/

And going back and reading at chapter 10 in the book (the chapter I skipped) I uninstalled nova-network. And suddenly, the instances are able to reach the API now.

Remember how the default OpenStack setup was using VLAN Manager? That is useful if you need to separate tenants using IP ranges and VLANs. More importantly, it meant that nova-network is only needed on the controller side (since the controller handles the route and IP. It is only when I need to use the other networking setup (Flat Networking or Flat Networking with DHC –  where I isolate the tenants using Security Group Modes) is nova-network is necessary on the new nodes. Otherwise, the firewalls setup by nova-network was blocking access to the  API by the instances.

So that was resolved and I was able to build instances at will – well mostly. Just tried to create a new instance just now. and I get this:


ERROR: Quota exceeded: code=InstanceLimitExceeded (HTTP 413) (Request-ID: req-82aa560f-0318-4e55-b5bd-98b10d1b9c60)

Heh. Removing one instance now:



stardust:openstack rilindo$ nova delete vmi001

stardust:openstack rilindo$ nova list

+———————————————————+————+————+————————————————+

| ID                                   | Name   | Status | Networks                       |

+———————————————————+————+————+————————————————+

| 7a6a4912-71ce-45e2-8a38-51537a4c7ffb | vmi001 | ACTIVE | vmnet=10.0.0.4, 192.168.15.12  |

| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13  |

| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16  |

| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17  |

| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |

| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |

| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |

| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |

| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |

| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |

+———————————————————+————+————+————————————————+

stardust:openstack rilindo$ nova list

+———————————————————+————+————+————————————————+

| ID                                   | Name   | Status | Networks                       |

+———————————————————+————+————+————————————————+

| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13  |

| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16  |

| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17  |

| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |

| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |

| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |

| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |

| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |

| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |

+———————————————————+————+————+————————————————+

And adding a new one:

stardust:openstack rilindo$ nova boot vmi011 —image bedf0e78-c7d4-414e-85fb-291a0ccd851d —flavor 2 —key_name mykey

+——————————————————-+—————————————————————————————+

| Property                            | Value                                                    |

+——————————————————-+—————————————————————————————+

| status                              | BUILD                                                    |

| updated                             | 2013-04-28T04:02:23Z                                     |

| OS-EXT-STS:task_state               | scheduling                                               |

| OS-EXT-SRV-ATTR:host                | openstack2                                               |

| key_name                            | mykey                                                    |

| image                               | images/precise-server-cloudimg-i386.img                  |

| hostId                              | e1693fb6dfb89d758273a4312096678745f8f568dbdc3fbe279e286b |

| OS-EXT-STS:vm_state                 | building                                                 |

| OS-EXT-SRV-ATTR:instance_name       | instance-00000046                                        |

| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                     |

| flavor                              | m1.small                                                 |

| id                                  | 52cc6451-1c5c-462e-8d06-6f9465ce1a94                     |

| user_id                             | openstack                                                |

| name                                | vmi011                                                   |

| adminPass                           | RVeaJd7j7Kmu                                             |

| tenant_id                           | cookbook                                                 |

| created                             | 2013-04-28T04:02:22Z                                     |

| OS-DCF:diskConfig                   | MANUAL                                                   |

| accessIPv4                          |                                                          |

| accessIPv6                          |                                                          |

| progress                            | 0                                                        |

| OS-EXT-STS:power_state              | 0                                                        |

| metadata                            | {}                                                       |

| config_drive                        |                                                          |

+——————————————————-+—————————————————————————————+

stardust:openstack rilindo$ nova list

+———————————————————+————+————+————————————————+

| ID                                   | Name   | Status | Networks                       |

+———————————————————+————+————+————————————————+

| 2fbaad51-35f4-4ac4-b6a3-d338af5905d8 | vmi002 | ACTIVE | vmnet=10.0.0.7, 192.168.15.13  |

| c70c7522-c8fb-4196-8161-6c6153549729 | vmi003 | ACTIVE | vmnet=10.0.0.8, 192.168.15.16  |

| 9899b0ee-4cac-4588-891b-705a6cc95512 | vmi004 | ACTIVE | vmnet=10.0.0.9, 192.168.15.17  |

| 41db335c-97a2-4335-9e91-7a0a4345b70a | vmi005 | ACTIVE | vmnet=10.0.0.10, 192.168.15.18 |

| 4d2698e2-88ba-42f6-9107-0380d89c3e89 | vmi006 | ACTIVE | vmnet=10.0.0.11, 192.168.15.19 |

| 9cf60a41-1a58-4dbd-a47c-b4f2c42e6117 | vmi007 | ACTIVE | vmnet=10.0.0.12, 192.168.15.24 |

| de257ab1-cf03-4f08-816f-c9e16ce2793a | vmi008 | ACTIVE | vmnet=10.0.0.13, 192.168.15.25 |

| 0c67d3bc-1ca1-4cfe-9653-eb3a5c94350f | vmi009 | ACTIVE | vmnet=10.0.0.14, 192.168.15.26 |

| 9b5601bf-969f-447f-9e2a-8727dd3d45e2 | vmi010 | ACTIVE | vmnet=10.0.0.15, 192.168.15.27 |

| 52cc6451-1c5c-462e-8d06-6f9465ce1a94 | vmi011 | ACTIVE | vmnet=10.0.0.4, 192.168.15.12  |

+———————————————————+————+————+————————————————+

stardust:openstack rilindo$ ssh -i mykey.pem ubuntu@192.168.15.12

The authenticity of host ‘192.168.15.12 (192.168.15.12)’ can’t be established.

RSA key fingerprint is 85:01:59:17:8d:11:4b:7c:60:72:c8:09:be:2d:45:73.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘192.168.15.12’ (RSA) to the list of known hosts.

Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-40-virtual i686)

 * Documentation:  https://help.ubuntu.com/

  System information as of Sun Apr 28 04:17:07 UTC 2013

  System load:  0.0              Processes:           61

  Usage of /:   6.9% of 9.84GB   Users logged in:     0

  Memory usage: 1%               IP address for eth0: 10.0.0.4

  Swap usage:   0%

  Graph this data and manage this system at https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:

    http://www.ubuntu.com/business/services/cloud

  Use Juju to deploy your cloud instances and workloads:

    https://juju.ubuntu.com/#cloud-precise

0 packages can be updated.

0 updates are security updates.

The programs included with the Ubuntu system are free software;

the exact distribution terms for each program are described in the

individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by

applicable law.

To run a command as administrator (user “root”), use “sudo ”.

See “man sudo_root” for details.

ubuntu@vmi011:~$ 

URLs I used as reference:

http://www.mail-archive.com/openstack@lists.launchpad.net/msg20584.html

http://www.mirantis.com/blog/openstack-networking-single-host-flatdhcpmanager/

http://www.mirantis.com/blog/openstack-networking-flatmanager-and-flatdhcpmanager/

http://www.gossamer-threads.com/lists/openstack/dev/17681

http://howtobyexamples.blogspot.com/2011/10/openstack-novaconf-configuration.html

http://blog.tocisoft.com/2011/06/why-ec2-ami-tools-ec2-upload-bundle.html

http://docs.openstack.org/essex/openstack-compute/admin/content/configuring-networking-on-the-compute-node.html

https://answers.launchpad.net/nova/+question/215096

http://docs.openstack.org/essex/openstack-compute/admin/content/network-troubleshooting.html

http://docs.openstack.org/folsom/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html

Note: The book was written for Openstack Essex. Since then, Flosom and now Grizzly was released. Which mean that much of the results I found was for the latter, making things more for me, as I wasn’t sure if the solution was applicable to Essex or not. (For example, nova-network was replaced by Quantum, which apparently does networking a bit diffently).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s