rezib / cloubed Goto Github PK

In addition to resources status in libvirt, cloubed status should also print useful information in the output. Esp. auto-allocated spice port for domains.

Cloubed CLI cleanup action currently destroy everything previously created in Libvirt. But it should also remove files that have been generated by templates.

When running the following Python script:

VM  =  "mngt0"

try:
     print("generating template file")
     cloubed.gen_file(domain_name  =  VM,  template_name  =  "ipxe")
     cloubed.gen_file(domain_name  =  VM,  template_name  =  "kickstart")
     print("booting vm {vm}".format(vm=VM))
     cloubed.boot_vm(domain_name  =  VM,
                     bootdev  =  "network",
                     overwrite_disks  =  True,
                     overwrite_networks  =  True)
     print("waiting event shutdown on vm {vm}".format(vm=VM))
     cloubed.wait_event(VM,  "STOPPED",  "SHUTDOWN")
     print("booting vm {vm}".format(vm=VM))
     cloubed.boot_vm(VM) 

I systematically obtain the following error at the end at the second call to boot_vm():

generating template file
booting vm mngt0
waiting event shutdown on vm mngt0
10.5.0.105 - - [27/Aug/2013 10:18:20] "GET /http/rhel.ks HTTP/1.1" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/host/mngt0/ HTTP/1.0" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] code 404, message File not found
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /robots.txt HTTP/1.0" 404 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/host/mngt0/ssh_host_dsa_key HTTP/1.0" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/host/mngt0/ssh_host_dsa_key.pub HTTP/1.0" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/host/mngt0/ssh_host_rsa_key HTTP/1.0" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/host/mngt0/ssh_host_rsa_key.pub HTTP/1.0" 200 -
10.5.0.10 - - [27/Aug/2013 10:24:21] "GET /http/ssh/id_rsa.pub HTTP/1.0" 200 -
booting vm mngt0
Exception in thread ClouBedHTTPServer:
Traceback (most recent call last):
   File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
     self.run()
   File "/usr/lib/python2.7/threading.py", line 763, in run
     self.__target(*self.__args, **self.__kwargs)
   File "/usr/local/lib/python2.7/dist-packages/cloubed/HTTPServer.py", line 58, in threaded_server
     self._handler)
   File "/usr/lib/python2.7/SocketServer.py", line 419, in __init__
     self.server_bind()
   File "/usr/lib/python2.7/SocketServer.py", line 430, in server_bind
     self.socket.bind(self.server_address)
   File "/usr/lib/python2.7/socket.py", line 224, in meth
     return getattr(self._sock,name)(*args)
error: [Errno 98] Address already in use

When booting a VM on a Intel Nehalem CPU, Cloubed fails with this error:

generating template file
booting vm mngt0
libvir: QEMU error : internal error guest CPU is not compatible with host CPU
ERROR:root:libvirt error: internal error guest CPU is not compatible with host CPU
internal error guest CPU is not compatible with host CPU

Done. 

The manpage should be generated using Sphinx, so this also linked with #12.

This may help a lot in testing and diagnosing some issues with libvirt.

When no graphics is set on domain in YAML file, Cloubed default to spice. But spice is not supported by old versions of Libvirt still used in production. Therefore, Cloubed should check the version of Libvirt before setting spice, and fallback to vnc when appropriate.

Sphinx is probably the best candidate for this matter.

The storage volumes could be defined in domains section under disks sub-section. This way, the storagevolumes is optional.

When running the following code:

from cloubed import cloubed
     cloubed.boot_vm("mngt0")

It sometimes produces this message:

Exception in thread ClouBedHTTPServer (most likely raised during interpreter shutdown):Error in sys.excepthook:
Traceback (most recent call last):
Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 54, in apport_excepthook
   File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
   File "/usr/lib/python2.7/threading.py", line 763, in run
   File "/usr/local/lib/python2.7/dist-packages/cloubed/HTTPServer.py", line 67, in threaded_server
   File "/usr/lib/python2.7/SocketServer.py", line 235, in serve_forever
 <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'select'

See the example traceback below:

Traceback (most recent call last):
  File "bin/build-base", line 88, in <module>
    build_base(domain)
  File "bin/build-base", line 59, in build_base
    cloubed.gen_file(domain_name=domain, template_name="preseed")
  File "/usr/lib/python2.6/dist-packages/cloubed/__init__.py", line 37, in gen_file
    cloubed.gen_file(domain_name, template_name)
  File "/usr/lib/python2.6/dist-packages/cloubed/Cloubed.py", line 293, in gen_file
    domain_template.render(templates_dict)
  File "/usr/lib/python2.6/dist-packages/cloubed/DomainTemplate.py", line 53, in render
    template_str = ExtTemplate(open(self._source_file, 'r').read()) \
IOError: [Errno 2] No such file or directory: 'templates/preseed-jessy.cfg'

This should be handled more gracefully.

When a domain is booted, it could be convenient to print the allocated spice or vnc port opened for incoming connection to the graphical console.

It is possible to specify a domain name in Libvirt network XML description. This is then used by the DHCP server to give a domain name to clients. Refer to Libvirt documentation for more information.

This would be nice if this domain name could be also set in Cloubed YAML file.

It should be possible to setup specific NUMA topologies on domains CPUs in YAML file.

When running a Python script based on cloubed, the following error happens sometimes:

generating template file
booting vm mngt0
libvir:  error : internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied
ERROR:root:libvirt error: internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied

internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied

It would be nice to have colorized output for cloubed CLI when having a TTY.

Instead of current network address/netmask format:

ip_host: 10.1.0.1
netmask: 255.255.255.0

The address could be defined with neater CIDR syntax:

address: 10.1.0.1/24

The storagepools section should be optional with a default storage pool named pool in pool/ sub-directory.

I am using Centos 7,

XX|⇒ python install.py
generating template file
DEBUG : new RW VirtController
INFO : initializing storage pool pool
INFO : initializing storage volume server
INFO : initializing network net
INFO : initializing domain vmtest
DEBUG : generated mac 00:16:3e:03:e2:b1 for netif on domain vmtest connected to network net
DEBUG : registering host vmtest in network net
booting vm vmtest
DEBUG : disks to overwrite for domain vmtest: ['server']
INFO : found storage pool danie1m:centos:pool with the same path
INFO : found storage pool danie1m:centos:pool with the same path
INFO : found storage pool danie1m:centos:pool with the same path
INFO : deleting storage volume danie1m:centos:server.qcow2
INFO : found storage pool toto:centos:pool with the same path
DEBUG : networks to recreate for domain vmtest: ['net']
INFO : destroying network net
libvirt: Domain Config error : unknown OS type hvm
unknown OS type hvm
DEBUG : clean exit

The created domain is Wrong ? (i.e in domain.py)
xml domain :

...
<os>
     <type>hvm</type>
     <boot dev='network' />
</os>
...

I'm not able to print the xml domain to debug! Is it possible with cloubed ?

See this blog post generally considered as a reference:
http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html

When running a script with cloubed library, this error raised when calling boot_vm():

generating template file
booting vm mngt0
libvir: QEMU error : Cannot find QEMU binary /usr/bin/kvm: No such file
or directory
ERROR:root:libvirt error: Cannot find QEMU binary /usr/bin/kvm: No such
file or directory
Cannot find QEMU binary /usr/bin/kvm: No such file or directory

Done. 

The distribution is RHEL 6.3 with the following packages:

libvirt-0.9.10-21.bl6.x86_64
libvirt-client-0.9.10-21.bl6.x86_64
libvirt-python-0.9.10-21.bl6.x86_64
qemu-kvm-0.12.1.2-2.295.bl6_3.2.x86_64

Here is the [truncated] output of virsh capabilities:

<capabilities>
  [...]

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine>rhel6.3.0</machine>
      <machine canonical='rhel6.3.0'>pc</machine>
      <machine>rhel6.2.0</machine>
      <machine>rhel6.1.0</machine>
      <machine>rhel6.0.0</machine>
      <machine>rhel5.5.0</machine>
      <machine>rhel5.4.4</machine>
      <machine>rhel5.4.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities> 

It looks like cloubed looks for KVM executable under path /usr/bin/kvm whereas
libvirt finds it under path /usr/libexec/qemu-kvm.

Ex:

boot_domain() -> boot()
wait_event() -> wait()

And so on. All previous functions must still work though, a deprecation message will be printed.

The script should have an option to print the list of all available template variables based on the content of the YAML file. This list should significantly help users writing template files.

Users should be able to poweroff/halt/reboot domains using fake ACPI.

Some Cloubed classes are easy to tests without the need of mocks, especially configuration and script arguments parsing. A first set of unit tests has to be written in order to automatically test these classes.

It would be cool to be able to make cloubed wait until specific TCP ports open on the domains (ex: 22/SSH).

When a network is created in Libvirt but another existing network has conflicting IP settings, libvirt (and therefore cloubed) fails totally. It would be better if cloubed was able to detect such conflicting network in order to properly warn users.

This will allow installing domains without network connection, which could be useful in some cases.

It would be great to be able to update network settings (ex: add static IP assignement in DHCP) without the need to destroy and recreate it (with all domains attached to it).

When running a Python script based on cloubed, the following error happens sometimes:

generating template file
booting vm mngt0
libvir:  error : internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied
ERROR:root:libvirt error: internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied

internal error Child process (/usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/net.conf) unexpected exit status 3:
dnsmasq: TFTP directory /path/to/tftp/dir inaccessible: Permission denied

What does this error mean?

Instead of current PXE TFTP file path definition format:

pxe:
  tftp_dir: tftp
  boot_file: boot.ipxe

The path could be defined with only one parameter:

pxe: tftp/boot.pxe

This would be neater.

New functions should be added to Cloubed library to get the list of defined resources (domains, networks, etc) in the YAML file.

This feature will allow the domains to access part of the host filesystem easily (without NFS or such other things).

This class should help in "mocking" this external library in unit tests.

When a cloubed based Python script is run but no cloubed.yaml file could be found in current directory, cloubed fails miserably with the following error:

Traceback (most recent call last):
  File "install.py", line 9, in <module>
    cloubed.gen_file(domain_name = VM, template_name = "ipxe")
  File "/usr/local/lib/python2.7/dist-packages/cloubed/cloubed.py", line 35, in gen_file
    cloubed = Cloubed()
  File "/usr/local/lib/python2.7/dist-packages/cloubed/Cloubed.py", line 51, in __call__
    .__call__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cloubed/Cloubed.py", line 84, in __init__
    self._conf = Configuration(configuration_filename)
  File "/usr/local/lib/python2.7/dist-packages/cloubed/Configuration.py", line 35, in __init__
    yaml_file = open(self._file_path)
IOError: [Errno 2] No such file or directory: '/home/.../cloubed.yaml'

I'm working with cloubed, and I need 1 or almost 2 great new features (killer features...) :

• possibility to create disk snapshot (as a complete system restore point)
• the posibility to destroy the virtual machine (and virtual bridge associated)

Sometimes, os.getlogin() returns weird errors surch as file not found.

See python issue 584566 for more details.

The solution is to use pwd.getpwuid(os.geteuid())[0] instead.

The following python script uses cloubed:

VM = "mngt0"

try:
    print("generating template file")
    cloubed.gen_file(domain_name = VM, template_name = "ipxe")
    cloubed.gen_file(domain_name = VM, template_name = "kickstart")
    print("booting vm {vm}".format(vm=VM))
    cloubed.boot_vm(domain_name = VM,
                    bootdev = "network",
                    overwrite_disks = True,
                    overwrite_networks = True)
    print("waiting event shutdown on vm {vm}".format(vm=VM))
    cloubed.wait_event(VM, "STOPPED", "SHUTDOWN")
    print("booting vm {vm}".format(vm=VM))
    cloubed.boot_vm(VM)

But when it is run with debug output enable, it stays stuck in wait_event() for ever:

generating template file
DEBUG  : initialized event manager
INFO   : initializing storage pool bullx-pool
INFO   : initializing storage volume bullx-vol-mngt
INFO   : initializing network bullx-net
INFO   : initializing domain mngt0
booting vm mngt0
INFO   : destroying storage pool bullx-pool
INFO   : deleting storage volume bullx-vol-mngt.qcow2
INFO   : destroying network bullx-net
INFO   : domain mngt0: created
waiting event shutdown on vm mngt0
DEBUG  : domain mngt0: entering in wait_for_event loop waiting for event STOPPED>STOPPED_SHUTDOWN

Nothing else happens then, the script must be stopped manually with ^c.

The environment is an Ubuntu 13.04 with the following packages:

$ dpkg -l | grep libvirt
ii  libvirt-bin 1.0.2-0ubuntu11.13.04.2      amd64        programs for the libvirt library
ii  libvirt0 1.0.2-0ubuntu11.13.04.2         amd64        library for interfacing with different virtualization systems
ii  libvirtodbc0 6.1.6+repack-0ubuntu2       amd64        high-performance database - ODBC libraries
ii  python-libvirt 1.0.2-0ubuntu11.13.04.2   amd64        libvirt Python bindings

When running cloubed, asks for ebtables if it's not installed:

booting domain base
DEBUG : disks to overwrite for domain base: ['base']
INFO : found storage pool ag06347s:slurmappliance:pool with the same path
INFO : found storage pool ag06347s:slurmappliance:pool with the same path
INFO : found storage pool ag06347s:slurmappliance:pool with the same path
INFO : deleting storage volume ag06347s:slurmappliance:base.qcow2
INFO : found storage pool ag06347s:slurmappliance:pool with the same path
DEBUG : networks to recreate for domain base: ['net']
libvirt: Firewall error : direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
direct firewall backend requested, but /sbin/ebtables is not available: No such file or directory
DEBUG : clean exit

Installed ebtables and reloading libvirtd fixed the issue.

The internal HTTP server is always enable by default in cloubed wait action. This is not a sane default, the user should explicitly enable it when needed.

All functions of the cloubed library should be moved from cloubed.py to __init__.py. This way, importing the cloubed library should be as simple as:

import cloubed

Instead of the current:

from cloubed import cloubed

When there is only one storage pool defined, the storagepool parameter in storage volumes should be optional since it is necessarily this one storage pool.

Currently, one must use the following template excerpt to get access to internal HTTP server:

http://${network.net.ip_host}:5432/http/vmlinuz

Cloubed should provide an implicitely defined template variable to avoid setting TCP port explicitely in template files, like this:

${network.net.http_server}/http/vmlinuz

Currently, cloubed cleanup action basically delete everything in the testbed. It would be much better if the user could control what should be deleted (ex: domains but not the volumes).

Cloubed requires Python argparse library to manage script parameters but it is missing in the install_requires list of setup.py.