Thursday, October 16, 2014

OpenStack icehouse installation error : nova-api service getting stopped

While trying to install OpenStack icehouse, faced an issue with nova-api service.It was not getting started. The following error was coming up in the Nova-api log

Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iptables-save -c
Exit code: 1

 nova Stdout: ''
2014-10-17 07:21:08.058 27270 TRACE nova Stderr: 'Traceback (most recent call last):\n  File "/usr/bin/nova-rootwrap", line 6, in <module>\n    from oslo.rootwrap.cmd import main\nImportError: No module named rootwrap.cmd\n'

Problem was with one of the oslo.rootwrap module. It was broken

Solution is to upgrade the module using pip

 #pip install oslo.rootwrap --upgrade

Tuesday, September 2, 2014

OpenStack: Restrict instance deletion

In OpenStack, by default users who are members of a  tenant can delete all instances in that tenant, even if it is spinned up by other users. If you want to restrict that, you  need to tweak the nova policy file  ie /etc/nova/policy.json

Add the following lines in the file:

    "admin_or_user":"is_admin:True or user_id:%(user_id)s",

Make the same changes in the /etc/openstack-dashboard/nova_policy.json file also

Now restart the openstack-nova-api service

Now user will be able to delete only those instances spinned up by them. Admin users will be able to delete all instances

Monday, September 1, 2014

OpenStack : Assign floating IP using heat template

Creating Yaml templates that assign floating IPs to your instances being spawned can be a bit tricky.Let us look at a scenario where we need to spin up a VM, assign a floating IP from a pool and make reference to this floating IP in your userdata as well. We will make use of the network ID of the internal and external network, as well as the Subnet ID of the internal network

The logical workflow is as follows:

  •  Create a port resource using internal network and internal subnet IDs
  • Create a floating IP resource , referring to the external network ID
  •  Associate the floating IP to the port
  •   In the server resource being created, associate the port resource
  Now we will see how this can be implemented using both HOT and AWS template formats

Friday, August 29, 2014

OpenStack monitoring: Zabbix Ceilometer proxy installation

Recently a Ceilometer proxy for Zabbix was released by OneSource. This proxy will pull all the instance information from OpenStack and populate it in Zabbix

The source code can be downloaded from here:

The basic prerequisites for the server where the proxy is running is Python and Pika library. Also there should be network connectivity from the proxy machine to your OpenStack installation.

Friday, August 22, 2014

Agentless openstack monitoring using zabbix

Zabbix can be a tough cookie to crack!! And if you are planning to monitor Openstack using Zabbix, there is lot of additional work to be done .More so, if you want to go the agentless way, ie using SNMP

So, here we go.I am using Ubuntu 12.04 OS, both for my Zabbix server as well as openstack nodes

  • First you need to install the following packages using apt-get in the machine being monitored ie the openstack node

Thursday, August 21, 2014

Tech tip: Increase openstack project quota from command line

1. List the keystone tenants and search for the required tenant

keystone tenant-list |grep <tenantname>

 Note the id of the tenant being displayed. You need to use this id in the next command

2. Get quota details of the tenant using the following command

nova-manage project quota <tenantid>

Wednesday, August 13, 2014

Instances goes to paused state in Openstack Havanna


All instances in openstack will be in paused node. You will not be able to create new instances or switch on any of the paused instances


Most often the reason will be lack of disk space in your compute node. By default the instances are created in the /var/lib/nova/instances folder of the compute node. This location is defined by the parameter "instances_path" in nova.conf of the  compute node. If your "/" partition is running out of disk space, then you cannot perform any instance related operations


  • Change the "instances_path" location to a different location. Ideally you could attach an additional disk and mount it to a directory and update the directory path in the "instance_path" variable.
  • Problem arises when you already have a number of instances  in the previous folder. You should move them over to the new location.
  •  Also you should set the group and ownership of the new instances folder to "nova" user, so that the permissions, ownership and group memberships are same as that of the previous folder

Openstack havanna neutron agent-list alive status error

In some scenarios, the openstack neutron-agent status will show as xxx even though you could see he neutron agents services are up and running in the network and compute nodes. Also you could see a fluctuation in the agent status if you try the agent-list command repeatedly.  Confusing, right?

Actually  the problem is not in the actual agent status, but with two default configurations in neutron.conf ie agent_down_time and report_interval. It is the interval during which neutron will check the agent status. There is a bug reported against this issue

As per the details in the bug " report_interval" is how often an agent sends out a heartbeat to the service. The Neutron service responds to these 'report_state' RPC messages by updating the agent's heartbeat DB record. The last heartbeat is then compared to the configured agent_down_time to determine if the agent is up or down"

The neutron agent-list command uses the agent_down_time value to display the status. The default values are set very low, because of which the alive status is shown as down/fluctuating.

Solution: As suggested in the solution for the bug, update the values of agent_down_time and report_interval to 75 and 30 seconds respectively. Since the above mentioned rpc issue with open-vswitch agent in compute is  resolved by this, all the agents will be shown as alive

Friday, July 25, 2014

Ubuntu 12.04 P2V conversion using non-root user

Ubuntu P2V conversion is not as straight forward as other Linux machines with a root user. This is because we use a non-root user by default for managing Ubuntu machines and the root credentials are not known to us. So how do you convert a physical Ubuntu VM to virtual without the root credentials? Here are the steps

PS: please note the steps are for VMware vCenter standalone converter 5.5

1.Edit VMware configuration files converter-agent.xml and converter-worker.xml files present in C:\ProgramData\VMware\VMware vCenter Converter Standalone , update the useSudo flag from false to true

2. Reboot the VMware converter standalone agent service

Monday, July 21, 2014

Tech tip: Create separate routing table for multiple nics in Ubuntu

Scenario: 2 nics in Ubuntu machine, requirement to assign IPs from different VLANs to each of these interfaces, access from outside world to all the assigned IPs.

The situation was a bit complex since the machine was a VM  in ESXi and each of these nics were added to portgroups of two VLANs 200 and 201. The first nic eth0 was assigned a gateway , and was accessible from outside world. The second nic eth1 was assigned IP in 201  VLAN, but was not able to ping to that machine from other machines in a different VLAN


Inorder to solve the issue, we had to add an additional routing table, to select default route for packages which should go out of eth1. The following lines added to the eth1 interface configuration in /etc/networking/interfaces file did the trick

post-up ip route add default via 10.1111.0.1 dev eth1 table 101
post-up ip rule add from lookup 101

Thursday, July 17, 2014

Tech tip: system error 53

Recently we faced a strange issue while connecting a Windows server 2012 R2 to domain, connection was failing with network error.

I was able to ping to the domain, as well as the domain controller. While troubleshooting the issue, I stumbled upon a gem called c:\windows\debug\netsetup.log. It was throwing an error

'NetUseAdd to \\<domain controller>\IPC$ returned 1231

Seems like the machine was having issue connecting to the share. I tried  the "Net view \\sharename" command and got system error 53. Now the root cause was confirmed, the machine was having trouble accessing shared drives

How do we solve it? Actually the solution was pretty simple , you need to install 'client for Microsoft networks" in the network adapter of the server. Somehow it got uninstalled from my VM's adapter and created all this trouble. Installed it back and was able to add the machine to domain!!


Friday, July 4, 2014

Tech tip : How to set http proxy for apt-get in Ubuntu

Steps to set an http proxy for apt-get permanently

1. Create a new file apt.conf in /etc/apt  . Add the following line in it

Acquire::http::proxy "http://<proxyip>:<port>/";

 2. Edit the file named environment in /etc .Add the following line to it.


If you want to add the proxy temporarily for a session, you can use the command

export http_proxy=http://yourproxyaddress:proxyport

PS: if you want to add proxy exception for local addresses in your network, add the following command to /etc/environment

no_proxy = localhost,<hostname>

Friday, April 4, 2014

Create Windows stack using Heat Orchestration Template in Openstack

The blog explains the process of creating a basic Windows stack in Openstack using Heat Orchestration Template. The hypervisor being used is VMware ESXi5.5.

Image preparation to upload in glance

  • Lets start with creating a Windows 2012 R2 VM in ESXi. In this server, download and install the cloudbase-init package for windows .The beta version is available at this link:

Follow the steps in this link for installation :

  • Once installation is completed, edit the 'setup.exe" registry key at HKLocal machine/SYSTEM/SETUP/STATUS/ChildCompletion and change the value from 1 to 3. This is to avoid a system restart exception when the image boots up for the first time in openstack
  • If you want to do any custom configurations in the windows machine, like open a specific firewall port, enable ping ,rdp can do it at this point
  •  Run Syprep and shutdown the VM
 C:\Windows\System32\sysprep\sysprep.exe /generalize /oobe /shutdown

  • Use a VMware standalone convertor ,select the prepared VM as source and convert it to a VM suitable for Vmware workstation 10.0.x. When the conversion process is completed, you will get a vmdk and .vmx file at the destination
  • Using winscp or any other similar tools, copy the converted vmdk to your openstack glance  server. 
  • Create an image from this vmdk using the following command

glance image-create --name <image name>--disk-format=vmdk --container-format=bare --is-public=true --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property vmware_ostype="windows8Server64Guest" < Openstack_win2012.vmdk

Sample yaml template

The template given below uses heat orchestrator to spin up an instance with the image that we created and install IIS in it

heat_template_version: 2013-05-23

description: >
  Basic windows+IIS installation


    type: string
    description : Name of a KeyPair to enable access to the instance
    type: string
    description: Instance type for Windows server
    default: m1.small
      - allowed_values: [m1.small, m1.medium, m1.large]
        description: instance_type must be one of m1.small, m1.medium or m1.large
    type: string
    description: ID of the image to use for the Windows erver
   type: string
   description: windows feature to be installed

    type: OS::Nova::Server
      image: { get_param: image_id }
      flavor: { get_param: instance_type }
      key_name: { get_param: key_name }
          template: |

            Install-WindowsFeature -Name feature -IncludeManagementTools
            feature: { get_param: windows_feature }


 Get password of the instance

Once the instance is up and running, you can use the keypair used while spinning up the stack to retrieve the password. In your openstance machine, run the following command

nova get-password <instance-id> <private key>

instance - id: This is the Id of instance created by the stack , can be obtained from horizon dashboard
privaye key : This is the pem file you downloaded while creating the keypair. Please note that you should copy it over to openstack machine using winscp before running the get-password command


The VMware specific configuration options for creating glance image
The Vmware_ostype derived from the enumerator VirtualMachineGuestOsIdentifier in the API reference:

Tuesday, February 11, 2014

How to create new Local SR in XenServer

In this blog, I will explain the process of creating an additional local SR in XenServer. In this scenario, there is an additional hard disk present in the server which I plan to add as a local SR in addition to the existing Local storage

First you need to identify which disk is being currently used for your existing local SR, so that it is not accidentally deleted

1)SSH to your server using putty

type the command xsconsole

In the console that is displayed go to Disks & Storage repositories -> Current Storage Repositories

You need to locate your current local repository and view the properties on the right pane

Note down the scsi ID

2)Type the following command

#ll /dev/disk/by-id

Note down which device the ID maps to. In this case it is sda3.So that is your present local SR

Also you need to note the SCSI id of the disk that you are planning to use for the new SR. In my case it is sdb

3)Next step is to create the new SR. You need run the following command and note down the "uuid"

#xe host-list

4) Create new SR using the following command

# xe sr-create content-type=user device-config:device=/dev/disk/by-id/<scsi-xxxxxxxxxxxxxxxxxxxxxxxxx> host-uuid=<host-uuid> name-label=”Local Storage 2” shared=false type=lvm

you need to replace the scsi id and host uuid

Now you should be able to view the new Local SR in the XenCenter!!

Friday, February 7, 2014

Install nested KVM in VMware ESXi 5.1

In this blog, I will explain the steps required to run a nested KVM hypervisor on  Vmware ESXi. The installation of KVM is done on Ubuntu 13.10(64 bit).

Note: It is assumed that you have already installed your Ubuntu 13.10 VM in ESXi, and hence we will not look into the Ubuntu installation part.

1) Upgrade VM Hardware version to 9.

In my ESXi server, the default VM hardware version was 8. So I had to shutdown my VM and upgrade the Hardware version to 9 to get the KVM hypervisor working. You can right click the VM and select the Upgrade hardware option to do this.

2)In the ESXi host In /etc/vmware edit the 'config' file and add the following setting

vhv.enable = "TRUE"

3)Edit the VM settings and go to VM settings > Options > CPU/MMU
Virtualization .
Select the Intel EPT option

4) Go to Options->CPUID mask> Advanced-> Level 1, add the following CPU mask level

ECX  ---- ---- ---- ---- ---- ---- --H- ----

5) Open the vmx file of the Ubuntu VM and add the following setting

monitor.virtual_exec = “hardware”
hypervisor.cpuid.v0 = “FALSE”

KVM installation steps

Once the above configurations are done in ESXi and the VM, hardware virtualization would be supported in the VM. Inorder to confirm that, run the following command in the Ubuntu VM.
egrep -c '(vmx|svm)' /proc/cpuinfo

You should get output as 1, which means hardware  virtualization is supported. Also you should see hvm flags in the output when you run the following command

cat /sys/hypervisor/properties/capabilities 
Or else you could also run the following command
The output should be as follows:
INFO: /dev/kvm exists

KVM acceleration can be used
 If the above command succeeds, proceed with the remaining steps below

1)Install KVM

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils

2)Add user to libvirtd group

$ sudo adduser <username> libvirtd
username should be the same as the admin user that you are logged in with(the one created during Ubuntu installation)

3)Set permissions

The sock file permission should be as follows
$ sudo ls -la /var/run/libvirt/libvirt-sock

srwxrwx--- 1 root libvirtd 0 2010-08-24 14:54 /var/run/libvirt/libvirt-sock

The /dev/kvm permission should be as follows

 $ ls -l /dev/kvm

crw-rw----+ 1 root libvirtd 10, 232 Jul  8 22:04 /dev/kvm

sometimes, the device group would be root, in that case change it to libvirtd

sudo chown root:libvirtd /dev/kvm

Log off and log in for the changes to take effect

4)Install GUI

In the default Ubuntu installation GUI is not installed. You could choose to install it followed by
if you plan to use the graphical user interface virt-manager to manage the VMs in KVM

First install the Ubuntu GUI

sudo apt-get install --no-install-recommends ubuntu-desktop

This will get a bare minimum installation of the GUI, that would serve our purpose of using Virt-manager

Reboot the machine after installation, and you can login to the GUI from the VM console

Now, Install the virt-manager using the following command

sudo apt-get install virt-manager

5)Access Virt-manager to create/manage VMs in KVM

Once you login to GUI, the virt manager can  be found by browing /Usr/share/applications  and select Virt-manager

If all turns out well, you should be able to see the Virt-manager connected to localhost(QEMU). Now you can start creating your VMs!!!

Monday, January 27, 2014

Timeout error during OpenStack installation

While trying to install OpenStack using RDO, following the instructions in the link here , you might face the following timeout error

Applying <IP address>_prescript.pp   [ ERROR ]

ERROR : Error appeared during Puppet run: <IP address>_prescript.pp
Error: Command exceeded timeout

The reason for this error, in all probabilities would be the delay in your network, which could be causing the netns component packages to be timed out You can increase the timeout by editing the following file


Add a timeout line in the file

    if $::netns_support != "true" {
        exec { "netns_dependecy_install":
            path => "/usr/bin/",
            command => "yum update -y kernel iputils iproute",
            timeout => 1900,

Now you can retry the installation. It is advisable to use the answer file that is created during the initial installation, otherwise the reinstall would fail

Sample command for doing the reinstall using answer file is

packstack --answer-file=/root/packstack-answers-20140125-195230.txt

replace the text file name with the name of your answer file that was created during the initial installation

Thursday, January 9, 2014

XenCenter troubleshooting

Issue 1:

While trying to add a new host to a Pool in XenCenter, I was getting error "internal connection failed no route to host "


It was quite simple actually, but since I couldnt find it mentioned in any knowledge bases or articles, I am noting it down here. The machine where XenCenter was installed couldnt resolve the Xen host name. The machines were not added to domains and hence no DNS entries were there. So I had to add a host entry to the XenCenter machine to help with the name resolution

Issue 2:

Enabling AD authentication via XenCenter fails with error "clock skew detected with active directory server"


This happens because of the time difference between XenServer and AD server


Update NTP server list of the XenServer and point it to the AD server of the domain. Steps below

From the XenServer console, go to 'Network and management interface"->Network time(NTP)

Select option "Remove all NTP servers" to remove the default NTP server configuration

Then select option "Add an NTP server"

Enter the name or IP address of the NTP server

This will update the NTP configuration. On the same menu, click on 'Enable NTP time synchronization" to complete the configuration


Wednesday, January 8, 2014

Xen learnings

This week, I was trying to get my head around a new Hypervisor(new for me, obiviously), ie XenServer. Though it is pretty much same as ESXi and is free as well, there are some striking differences as well.The observations are based on the free version of Citrix XenServer version 6.2.0

- While ESXi needs a paid vCenter to manage multiple hosts, you can use the free XenCenter software to manage multiple Xen hosts

- Latest version of Xen server doesnt have the equalent of DRS or DPM. There was a feature named workload balancing, which was strangely discontinued in version 6.2.0 stating reason that there are no takers for it.

- It does offer a High Availability of VMs using pool based clustering of hosts

-XenMotion is the equalent of VM live migration, but it is restricted to one VM at a time

-XenCenter doesnt have a web client like vCenter

-There was a tool named XenConvert used for physical to virtual conversion, but it is retired as well.

-There is an option named Dynamic Memory Control(DMC), which can be used for dynamic allocation of memory for VMs. We can set a maximum and minimum memory options for VMs which will be used by XenServer to manage memory crunch situations

-Thin provisioning is supported  for local storages only

-Distributed vSwitch controller appliance  is available for centralized management of networks in XenCenter. However, this too is being depreciated in v6.2.0

PS: One interesting point to note is that the configuration limits document of xenServer is very small when compared to VMware and it doesnt have much details mentioned. For example, VMware specified the maximum number of vCPUs that we can create per physical processor core. For v5.1 its 25 and for v5.5 its 32. However Xen doesnt give you a hardcoded value for that. When we contacted Xen support regarding the same, they mentioned that there is no limit!!! Obiviously, that means you have to keep performance in mind while deciding on the number of vCPUs

Here is a good comparison matrix between various hypervisors available in market