[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Freeipa-devel] [PATCHES] [WIP] 0337-0343 YAML test configuration



Hello Tomáš! I'm planning a little Christmas present for you. Instead of a surprise I'm Releasing early :)
Apply patches or: git pull http://github.com/encukou/freeipa yaml-config


These patches add YAML/JSON configuration for tests.

YAML/JSON is completely optional, the existing envvar style continues to work. The ipa-test-config tool can convert between the two (which is not just for show, export will be needed for debugging and unit tests).
If you choose to use YAML, you need PyYAML (python-yaml) installed.


Example:

$ MASTER=master.ipa.test REPLICA=repl.ipa.test TESTHOST_XYZ_env1=xyz.ipa.test BEAKERMASTER1_IP_env1=1.2.3.4 BEAKERREPLICA1_IP_env1=1.2.3.5 BEAKERXYZ1_IP_env1=1.2.3.6 ipa-test-config --global --yaml | tee /tmp/testconf.yaml
ad_admin_name: Administrator
ad_admin_password: Secret123
admin_name: admin
admin_password: Secret123
debug: false
dirman_dn: cn=Directory Manager
dirman_password: Secret123
dns_forwarder: 8.8.8.8
domains:
- hosts:
    master:
      external_hostname: master.ipa.test
      ip: 1.2.3.4
      role: master
    repl:
      external_hostname: repl.ipa.test
      ip: 1.2.3.5
      role: replica
    xyz:
      external_hostname: xyz.ipa.test
      ip: 1.2.3.6
      role: xyz
  name: ipa.test
  type: IPA
ipv6: false
nis_domain: ipatest
ntp_server: 2.pool.ntp.org
root_password: null
root_ssh_key_filename: ~/.ssh/id_rsa
test_dir: /root/ipatests

$ IPATEST_YAML_CONFIG=/tmp/testconf.yaml ipa-run-tests ...



What's left is to update the design and write tests. I'll get to it eventually, but now I'll probably be busy for a few days. If you'd like to do them as part of review, tests could be in the format:

from ipatests.test_integration import config
conf = config.Config.from_env($environment)
assert conf.to_dict() == {$result}

conf = config.Config.from_dict($inputdict)
assert conf.to_dict() == {$result}


https://fedorahosted.org/freeipa/ticket/3938

--
Petr³
From 7b2694aa002b2154daed4063b4a8c3adbe2e11f2 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Wed, 11 Dec 2013 18:15:48 +0100
Subject: [PATCH] test_integration.config: Fix crash in to_env when no replica
 is defined

---
 ipatests/test_integration/config.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index b8c5fdc7f9ce1877e34491964418a8d806168e73..1c2b6a50dd49ad9dd14a9824ce5b62f63e80a8fb 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -29,6 +29,8 @@
 from ipapython.ipa_log_manager import log_mgr
 from ipatests.test_integration.host import BaseHost, Host
 
+TESTHOST_PREFIX = 'TESTHOST_'
+
 
 class Config(object):
     def __init__(self, **kwargs):
@@ -181,8 +183,11 @@ def to_env(self, simple=True):
             for role in domain.roles:
                 hosts = domain.hosts_by_role(role)
 
+                prefix = ('' if role in domain.static_roles
+                          else TESTHOST_PREFIX)
+
                 hostnames = ' '.join(h.hostname for h in hosts)
-                env['%s%s' % (role.upper(), domain._env)] = hostnames
+                env['%s%s%s' % (prefix, role.upper(), domain._env)] = hostnames
 
                 ext_hostnames = ' '.join(h.external_hostname for h in hosts)
                 env['BEAKER%s%s' % (role.upper(), domain._env)] = ext_hostnames
@@ -209,9 +214,10 @@ def to_env(self, simple=True):
                     env['MASTER'] = default_domain.master.hostname
                     env['BEAKERMASTER'] = default_domain.master.external_hostname
                     env['MASTERIP'] = default_domain.master.ip
-                env['SLAVE'] = env['REPLICA'] = env['REPLICA_env1']
-                env['BEAKERSLAVE'] = env['BEAKERREPLICA_env1']
-                env['SLAVEIP'] = env['BEAKERREPLICA_IP_env1']
+                if default_domain.replicas:
+                    env['SLAVE'] = env['REPLICA'] = env['REPLICA_env1']
+                    env['BEAKERSLAVE'] = env['BEAKERREPLICA_env1']
+                    env['SLAVEIP'] = env['BEAKERREPLICA_IP_env1']
                 if default_domain.clients:
                     client = default_domain.clients[0]
                     env['CLIENT'] = client.hostname
-- 
1.8.3.1

From 27a6aa40ab90ac527638513635b583cd333614e3 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Wed, 11 Dec 2013 18:31:10 +0100
Subject: [PATCH] test_integration.config: Do not save the input environment

Using the input environment saved in self._session_env
outside of the config loading meant that methods of
configuration other than environment variables wouldn't
be possible.

Restructure the roles/extra_roles to not depend on _session_env.

Part of the work for: https://fedorahosted.org/freeipa/ticket/3938
---
 ipatests/test_integration/config.py | 53 ++++++++++++++-----------------------
 ipatests/test_integration/host.py   |  3 ++-
 2 files changed, 22 insertions(+), 34 deletions(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index 1c2b6a50dd49ad9dd14a9824ce5b62f63e80a8fb..75fb59657983380b496a958d1f62d8e46a1e2b4a 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -197,7 +197,8 @@ def to_env(self, simple=True):
 
                 for i, host in enumerate(hosts, start=1):
                     suffix = '%s%s' % (role.upper(), i)
-                    prefix = 'TESTHOST_' if role in domain.extra_roles else ''
+                    prefix = ('' if role in domain.static_roles
+                              else TESTHOST_PREFIX)
 
                     ext_hostname = host.external_hostname
                     env['%s%s%s' % (prefix, suffix,
@@ -292,12 +293,9 @@ def __init__(self, config, name, index, domain_type):
         self.realm = self.name.upper()
         self.basedn = DN(*(('dc', p) for p in name.split('.')))
 
-        self._extra_roles = tuple()  # Serves as a cache for the domain roles
-        self._session_env = None
-
     @property
     def roles(self):
-        return self.static_roles + self.extra_roles
+        return sorted(set(host.role for host in self.hosts))
 
     @property
     def static_roles(self):
@@ -309,31 +307,21 @@ def static_roles(self):
 
     @property
     def extra_roles(self):
-        if self._extra_roles:
-            return self._extra_roles
-
-        roles = ()
-
-        # Extra roles can be defined via env variables of form TESTHOST_key_envX
-        for variable in self._session_env:
-            if variable.startswith('TESTHOST'):
-
-                variable_split = variable.split('_')
-
-                defines_extra_role = (
-                    variable.endswith(self._env) and
-                    # at least 3 parts, as in TESTHOST_key_env1
-                    len(variable_split) > 2 and
-                    # prohibit redefining roles
-                    variable_split[-2].lower() not in roles
-                    )
-
-                if defines_extra_role:
-                    key = '_'.join(variable_split[1:-1])
-                    roles += (key.lower(),)
-
-        self._extra_roles = roles
-        return roles
+        return [role for role in self.roles if role not in self.static_roles]
+
+    def _roles_from_env(self, env):
+        for role in self.static_roles:
+            yield role
+
+        # Extra roles are defined via env variables of form TESTHOST_key_envX
+        seen = set()
+        for var in env:
+            if var.startswith(TESTHOST_PREFIX) and var.endswith(self._env):
+                variable_split = var.split('_')
+                role_name = '_'.join(variable_split[1:-1])
+                if role_name and role_name not in seen:
+                    yield role_name.lower()
+                    seen.add(role_name)
 
     @classmethod
     def from_env(cls, env, config, index, domain_type):
@@ -349,10 +337,9 @@ def from_env(cls, env, config, index, domain_type):
         master_env = '%s_env%s' % (master_role, index)
         hostname, dot, domain_name = env[master_env].partition('.')
         self = cls(config, domain_name, index, domain_type)
-        self._session_env = env
 
-        for role in self.roles:
-            prefix = 'TESTHOST_' if role in self.extra_roles else ''
+        for role in self._roles_from_env(env):
+            prefix = '' if role in self.static_roles else TESTHOST_PREFIX
             value = env.get('%s%s%s' % (prefix, role.upper(), self._env), '')
 
             for index, hostname in enumerate(value.split(), start=1):
diff --git a/ipatests/test_integration/host.py b/ipatests/test_integration/host.py
index 507e19ed62b3d0a76e6e2ff6286fd83f17a68627..a35602a973c1e1fc7d969a06302b0e9d2979bb0f 100644
--- a/ipatests/test_integration/host.py
+++ b/ipatests/test_integration/host.py
@@ -128,7 +128,8 @@ def to_env(self, **kwargs):
         env['MYBEAKERHOSTNAME'] = self.external_hostname
         env['MYIP'] = self.ip
 
-        prefix = 'TESTHOST_' if self.role in self.domain.extra_roles else ''
+        prefix = ('' if self.role in self.domain.static_roles
+                  else TESTHOST_PREFIX)
         env['MYROLE'] = '%s%s%s' % (prefix, role, self.domain._env)
         env['MYENV'] = str(self.domain.index)
 
-- 
1.8.3.1

From 64642fc77cab976f2d4f1bbdc6397352786710bb Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Wed, 11 Dec 2013 19:06:31 +0100
Subject: [PATCH] test_integration.config: Use a more declarative approach to
 test-wide settings

The list of options was duplicated too many times. Consolidate.

Part of the work for: https://fedorahosted.org/freeipa/ticket/3938
---
 ipatests/test_integration/config.py | 105 +++++++++++++++++-------------------
 1 file changed, 48 insertions(+), 57 deletions(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index 75fb59657983380b496a958d1f62d8e46a1e2b4a..cf61d83c3230a148aa0cb0c994d6eac56bfdd7c2 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -32,29 +32,57 @@
 TESTHOST_PREFIX = 'TESTHOST_'
 
 
+_SettingInfo = collections.namedtuple('Setting', 'name var_name default')
+_setting_infos = (
+    # Directory on which test-specific files will be stored,
+    _SettingInfo('test_dir', 'IPATEST_DIR', '/root/ipatests'),
+
+    # File with root's private RSA key for SSH (default: ~/.ssh/id_rsa)
+    _SettingInfo('root_ssh_key_filename', 'IPA_ROOT_SSH_KEY', None),
+
+    # SSH password for root (used if root_ssh_key_filename is not set)
+    _SettingInfo('root_password', 'IPA_ROOT_SSH_PASSWORD', None),
+
+    _SettingInfo('admin_name', 'ADMINID', 'admin'),
+    _SettingInfo('admin_password', 'ADMINPW', 'Secret123'),
+    _SettingInfo('dirman_dn', 'ROOTDN', 'cn=Directory Manager'),
+    _SettingInfo('dirman_password', 'ROOTDNPWD', None),
+
+    # 8.8.8.8 is probably the best-known public DNS
+    _SettingInfo('dns_forwarder', 'DNSFORWARD', '8.8.8.8'),
+    _SettingInfo('nis_domain', 'NISDOMAIN', 'ipatest'),
+    _SettingInfo('ntp_server', 'NTPSERVER', None),
+    _SettingInfo('ad_admin_name', 'ADADMINID', 'Administrator'),
+    _SettingInfo('ad_admin_password', 'ADADMINPW', 'Secret123'),
+
+    _SettingInfo('ipv6', 'IPv6SETUP', False),
+    _SettingInfo('debug', 'IPADEBUG', False),
+)
+
+
 class Config(object):
     def __init__(self, **kwargs):
         self.log = log_mgr.get_logger(self)
 
         admin_password = kwargs.get('admin_password') or 'Secret123'
 
+        # This unfortunately duplicates information in _setting_infos,
+        # but is left here for the sake of static analysis.
         self.test_dir = kwargs.get('test_dir', '/root/ipatests')
-        self.root_password = kwargs.get('root_password')
         self.root_ssh_key_filename = kwargs.get('root_ssh_key_filename')
-        self.ipv6 = bool(kwargs.get('ipv6', False))
-        self.debug = bool(kwargs.get('debug', False))
+        self.root_password = kwargs.get('root_password')
         self.admin_name = kwargs.get('admin_name') or 'admin'
         self.admin_password = admin_password
         self.dirman_dn = DN(kwargs.get('dirman_dn') or 'cn=Directory Manager')
         self.dirman_password = kwargs.get('dirman_password') or admin_password
-        self.admin_name = kwargs.get('admin_name') or 'admin'
-        # 8.8.8.8 is probably the best-known public DNS
         self.dns_forwarder = kwargs.get('dns_forwarder') or '8.8.8.8'
         self.nis_domain = kwargs.get('nis_domain') or 'ipatest'
         self.ntp_server = kwargs.get('ntp_server') or (
             '%s.pool.ntp.org' % random.randint(0, 3))
         self.ad_admin_name = kwargs.get('ad_admin_name') or 'Administrator'
         self.ad_admin_password = kwargs.get('ad_admin_password') or 'Secret123'
+        self.ipv6 = bool(kwargs.get('ipv6', False))
+        self.debug = bool(kwargs.get('debug', False))
 
         if not self.root_password and not self.root_ssh_key_filename:
             self.root_ssh_key_filename = '~/.ssh/id_rsa'
@@ -71,25 +99,7 @@ def from_env(cls, env):
 
         Input variables:
 
-        DOMAIN: the domain to install in
-        IPATEST_DIR: Directory on which test-specific files will be stored,
-            by default /root/ipatests
-        IPv6SETUP: "TRUE" if setting up with IPv6
-        IPADEBUG: non-empty if debugging is turned on
-        IPA_ROOT_SSH_KEY: File with root's private RSA key for SSH
-            (default: ~/.ssh/id_rsa)
-        IPA_ROOT_SSH_PASSWORD: SSH password for root
-            (used if IPA_ROOT_SSH_KEY is not set)
-
-        ADMINID: Administrator username
-        ADMINPW: Administrator password
-        ROOTDN: Directory Manager DN
-        ROOTDNPWD: Directory Manager password
-        ADADMINID: Active Directory Administrator username
-        ADADMINPW: Active Directory Administrator password
-        DNSFORWARD: DNS forwarder
-        NISDOMAIN
-        NTPSERVER
+        See _setting_infos for test-wide settings
 
         MASTER_env1: FQDN of the master
         REPLICA_env1: space-separated FQDNs of the replicas
@@ -115,21 +125,14 @@ def from_env(cls, env):
         """
         env_normalize(env)
 
-        self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests',
-                   ipv6=(env.get('IPv6SETUP') == 'TRUE'),
-                   debug=env.get('IPADEBUG'),
-                   root_password=env.get('IPA_ROOT_SSH_PASSWORD'),
-                   root_ssh_key_filename=env.get('IPA_ROOT_SSH_KEY'),
-                   admin_name=env.get('ADMINID'),
-                   admin_password=env.get('ADMINPW'),
-                   dirman_dn=env.get('ROOTDN'),
-                   dirman_password=env.get('ROOTDNPWD'),
-                   dns_forwarder=env.get('DNSFORWARD'),
-                   nis_domain=env.get('NISDOMAIN'),
-                   ntp_server=env.get('NTPSERVER'),
-                   ad_admin_name=env.get('ADADMINID'),
-                   ad_admin_password=env.get('ADADMINPW'),
-                   )
+        kwargs = {s.name: env.get(s.var_name, s.default)
+                  for s in _setting_infos}
+
+        # $IPv6SETUP needs to be 'TRUE' to enable ipv6
+        if isinstance(kwargs['ipv6'], basestring):
+            kwargs['ipv6'] = (kwargs['ipv6'].upper() == 'TRUE')
+
+        self = cls(**kwargs)
 
         # Either IPA master or AD can define a domain
 
@@ -156,24 +159,12 @@ def to_env(self, simple=True):
             # Older Python versions
             env = {}
 
-        env['IPATEST_DIR'] = self.test_dir
-        env['IPv6SETUP'] = 'TRUE' if self.ipv6 else ''
-        env['IPADEBUG'] = 'TRUE' if self.debug else ''
-        env['IPA_ROOT_SSH_PASSWORD'] = self.root_password or ''
-        env['IPA_ROOT_SSH_KEY'] = self.root_ssh_key_filename or ''
-
-        env['ADMINID'] = self.admin_name
-        env['ADMINPW'] = self.admin_password
-
-        env['ROOTDN'] = str(self.dirman_dn)
-        env['ROOTDNPWD'] = self.dirman_password
-
-        env['ADADMINID'] = self.ad_admin_name
-        env['ADADMINPW'] = self.ad_admin_password
-
-        env['DNSFORWARD'] = self.dns_forwarder
-        env['NISDOMAIN'] = self.nis_domain
-        env['NTPSERVER'] = self.ntp_server
+        for setting in _setting_infos:
+            value = getattr(self, setting.name)
+            if value is None:
+                env[setting.var_name] = ''
+            else:
+                env[setting.var_name] = str(value)
 
         for domain in self.domains:
             env['DOMAIN%s' % domain._env] = domain.name
-- 
1.8.3.1

From cf2ea27e7c95a18ed2be4fc566952b3fbf4844f0 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Wed, 11 Dec 2013 19:33:30 +0100
Subject: [PATCH] test_integration.config: Do not store the index in Domain and
 Host objects

The index is a detail of the environment variable method of
configuration, it should only be used there.
---
 ipatests/test_integration/config.py | 45 +++++++++++++++++++------------------
 ipatests/test_integration/host.py   | 21 +++++++++--------
 2 files changed, 35 insertions(+), 31 deletions(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index cf61d83c3230a148aa0cb0c994d6eac56bfdd7c2..ee5c2eb5f1a223d824accc99e57858a1ba21a80a 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -167,9 +167,10 @@ def to_env(self, simple=True):
                 env[setting.var_name] = str(value)
 
         for domain in self.domains:
-            env['DOMAIN%s' % domain._env] = domain.name
-            env['RELM%s' % domain._env] = domain.realm
-            env['BASEDN%s' % domain._env] = str(domain.basedn)
+            env_suffix = '_env%s' % (self.domains.index(domain) + 1)
+            env['DOMAIN%s' % env_suffix] = domain.name
+            env['RELM%s' % env_suffix] = domain.realm
+            env['BASEDN%s' % env_suffix] = str(domain.basedn)
 
             for role in domain.roles:
                 hosts = domain.hosts_by_role(role)
@@ -178,13 +179,13 @@ def to_env(self, simple=True):
                           else TESTHOST_PREFIX)
 
                 hostnames = ' '.join(h.hostname for h in hosts)
-                env['%s%s%s' % (prefix, role.upper(), domain._env)] = hostnames
+                env['%s%s%s' % (prefix, role.upper(), env_suffix)] = hostnames
 
                 ext_hostnames = ' '.join(h.external_hostname for h in hosts)
-                env['BEAKER%s%s' % (role.upper(), domain._env)] = ext_hostnames
+                env['BEAKER%s%s' % (role.upper(), env_suffix)] = ext_hostnames
 
                 ips = ' '.join(h.ip for h in hosts)
-                env['BEAKER%s_IP%s' % (role.upper(), domain._env)] = ips
+                env['BEAKER%s_IP%s' % (role.upper(), env_suffix)] = ips
 
                 for i, host in enumerate(hosts, start=1):
                     suffix = '%s%s' % (role.upper(), i)
@@ -193,9 +194,9 @@ def to_env(self, simple=True):
 
                     ext_hostname = host.external_hostname
                     env['%s%s%s' % (prefix, suffix,
-                                    domain._env)] = host.hostname
-                    env['BEAKER%s%s' % (suffix, domain._env)] = ext_hostname
-                    env['BEAKER%s_IP%s' % (suffix, domain._env)] = host.ip
+                                    env_suffix)] = host.hostname
+                    env['BEAKER%s%s' % (suffix, env_suffix)] = ext_hostname
+                    env['BEAKER%s_IP%s' % (suffix, env_suffix)] = host.ip
 
         if simple:
             # Simple Vars for simplicity and backwards compatibility with older
@@ -270,16 +271,13 @@ def extend(name, name2):
 
 class Domain(object):
     """Configuration for an IPA / AD domain"""
-    def __init__(self, config, name, index, domain_type):
+    def __init__(self, config, name, domain_type):
         self.log = log_mgr.get_logger(self)
         self.type = domain_type
 
         self.config = config
         self.name = name
         self.hosts = []
-        self.index = index
-
-        self._env = '_env%s' % index
 
         self.realm = self.name.upper()
         self.basedn = DN(*(('dc', p) for p in name.split('.')))
@@ -300,14 +298,14 @@ def static_roles(self):
     def extra_roles(self):
         return [role for role in self.roles if role not in self.static_roles]
 
-    def _roles_from_env(self, env):
+    def _roles_from_env(self, env, env_suffix):
         for role in self.static_roles:
             yield role
 
         # Extra roles are defined via env variables of form TESTHOST_key_envX
         seen = set()
         for var in env:
-            if var.startswith(TESTHOST_PREFIX) and var.endswith(self._env):
+            if var.startswith(TESTHOST_PREFIX) and var.endswith(env_suffix):
                 variable_split = var.split('_')
                 role_name = '_'.join(variable_split[1:-1])
                 if role_name and role_name not in seen:
@@ -325,20 +323,23 @@ def from_env(cls, env, config, index, domain_type):
         else:
             master_role = 'AD'
 
-        master_env = '%s_env%s' % (master_role, index)
+        env_suffix = '_env%s' % index
+
+        master_env = '%s%s' % (master_role, env_suffix)
         hostname, dot, domain_name = env[master_env].partition('.')
-        self = cls(config, domain_name, index, domain_type)
+        self = cls(config, domain_name, domain_type)
 
-        for role in self._roles_from_env(env):
+        for role in self._roles_from_env(env, env_suffix):
             prefix = '' if role in self.static_roles else TESTHOST_PREFIX
-            value = env.get('%s%s%s' % (prefix, role.upper(), self._env), '')
+            value = env.get('%s%s%s' % (prefix, role.upper(), env_suffix), '')
 
-            for index, hostname in enumerate(value.split(), start=1):
-                host = BaseHost.from_env(env, self, hostname, role, index)
+            for host_index, hostname in enumerate(value.split(), start=1):
+                host = BaseHost.from_env(env, self, hostname, role,
+                                         host_index, index)
                 self.hosts.append(host)
 
         if not self.hosts:
-            raise ValueError('No hosts defined for %s' % self._env)
+            raise ValueError('No hosts defined for %s' % env_suffix)
 
         return self
 
diff --git a/ipatests/test_integration/host.py b/ipatests/test_integration/host.py
index a35602a973c1e1fc7d969a06302b0e9d2979bb0f..58f1fe4d9b1c1f99bdca03170c3fb16af9e7a089 100644
--- a/ipatests/test_integration/host.py
+++ b/ipatests/test_integration/host.py
@@ -32,11 +32,10 @@ class BaseHost(object):
     """Representation of a remote IPA host"""
     transport_class = None
 
-    def __init__(self, domain, hostname, role, index, ip=None,
+    def __init__(self, domain, hostname, role, ip=None,
                  external_hostname=None):
         self.domain = domain
         self.role = role
-        self.index = index
 
         shortname, dot, ext_domain = hostname.partition('.')
         self.shortname = shortname
@@ -94,11 +93,11 @@ def remove_log_collector(self, collector):
         self.log_collectors.remove(collector)
 
     @classmethod
-    def from_env(cls, env, domain, hostname, role, index):
+    def from_env(cls, env, domain, hostname, role, index, domain_index):
         ip = env.get('BEAKER%s%s_IP_env%s' %
-                        (role.upper(), index, domain.index), None)
+                        (role.upper(), index, domain_index), None)
         external_hostname = env.get(
-            'BEAKER%s%s_env%s' % (role.upper(), index, domain.index), None)
+            'BEAKER%s%s_env%s' % (role.upper(), index, domain_index), None)
 
         # We need to determine the type of the host, this depends on the domain
         # type, as we assume all Unix machines are in the Unix domain and
@@ -109,7 +108,7 @@ def from_env(cls, env, domain, hostname, role, index):
         else:
             cls = Host
 
-        self = cls(domain, hostname, role, index, ip, external_hostname)
+        self = cls(domain, hostname, role, ip, external_hostname)
         return self
 
     @property
@@ -120,9 +119,12 @@ def to_env(self, **kwargs):
         """Return environment variables specific to this host"""
         env = self.domain.to_env(**kwargs)
 
+        index = self.domain.hosts.index(self) + 1
+        domain_index = self.config.domains.index(self.domain) + 1
+
         role = self.role.upper()
         if self.role != 'master':
-            role += str(self.index)
+            role += str(index)
 
         env['MYHOSTNAME'] = self.hostname
         env['MYBEAKERHOSTNAME'] = self.external_hostname
@@ -130,8 +132,9 @@ def to_env(self, **kwargs):
 
         prefix = ('' if self.role in self.domain.static_roles
                   else TESTHOST_PREFIX)
-        env['MYROLE'] = '%s%s%s' % (prefix, role, self.domain._env)
-        env['MYENV'] = str(self.domain.index)
+        env_suffix = '_env%s' % domain_index
+        env['MYROLE'] = '%s%s%s' % (prefix, role, env_suffix)
+        env['MYENV'] = str(domain_index)
 
         return env
 
-- 
1.8.3.1

From c58cda529deab0c2f69f760e8a7802cbca7a97aa Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Wed, 11 Dec 2013 19:36:46 +0100
Subject: [PATCH] test_integration.config: Load/store from/to dicts

Part of the work for: https://fedorahosted.org/freeipa/ticket/3938
---
 ipatests/test_integration/config.py | 59 ++++++++++++++++++++++++++++++++++++-
 ipatests/test_integration/host.py   | 24 +++++++++++++--
 ipatests/test_integration/util.py   |  7 +++++
 3 files changed, 87 insertions(+), 3 deletions(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index ee5c2eb5f1a223d824accc99e57858a1ba21a80a..c19d1e84dd00b856ac5dfb5e7e0e3c2590174cc0 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -27,7 +27,7 @@
 from ipapython import ipautil
 from ipapython.dn import DN
 from ipapython.ipa_log_manager import log_mgr
-from ipatests.test_integration.host import BaseHost, Host
+from ipatests.test_integration.util import check_config_dict_empty
 
 TESTHOST_PREFIX = 'TESTHOST_'
 
@@ -94,6 +94,27 @@ def ad_domains(self):
         return filter(lambda d: d.type == 'AD', self.domains)
 
     @classmethod
+    def from_dict(cls, dct):
+        kwargs = {s.name: dct.pop(s.name, s.default) for s in _setting_infos}
+        self = cls(**kwargs)
+
+        for domain_dict in dct.pop('domains'):
+            self.domains.append(Domain.from_dict(domain_dict, self))
+
+        check_config_dict_empty(dct, 'config')
+
+        return self
+
+    def to_dict(self):
+        dct = {'domains': [d.to_dict() for d in self.domains]}
+        for setting in _setting_infos:
+            value = getattr(self, setting.name)
+            if isinstance(value, DN):
+                value = str(value)
+            dct[setting.name] = value
+        return dct
+
+    @classmethod
     def from_env(cls, env):
         """Create a test config from environment variables
 
@@ -313,7 +334,43 @@ def _roles_from_env(self, env, env_suffix):
                     seen.add(role_name)
 
     @classmethod
+    def from_dict(cls, dct, config):
+        from ipatests.test_integration.host import BaseHost
+
+        domain_type = dct.pop('type')
+        assert domain_type in ('IPA', 'AD')
+        domain_name = dct.pop('name')
+        hosts_dict = dct.pop('hosts')
+        self = cls(config, domain_name, domain_type)
+
+        for hostname, host_dict in hosts_dict.iteritems():
+            if '.' not in hostname:
+                hostname = '.'.join((hostname, domain_name))
+            default_role = 'master' if not self.hosts else 'replica'
+            role = host_dict.pop('role', default_role).lower()
+            host = BaseHost.from_dict(host_dict, self, hostname, role)
+            self.hosts.append(host)
+
+        check_config_dict_empty(dct, 'domain %s' % domain_name)
+
+        return self
+
+    def to_dict(self):
+        hosts = {}
+        for host in self.hosts:
+            name = host.hostname
+            if name.endswith('.' + self.name):
+                name = name[:-len(self.name) - 1]
+            hosts[name] = host.to_dict()
+        return {
+            'type': self.type,
+            'name': self.name,
+            'hosts': hosts,
+        }
+
+    @classmethod
     def from_env(cls, env, config, index, domain_type):
+        from ipatests.test_integration.host import BaseHost
 
         # Roles available in the domain depend on the type of the domain
         # Unix machines are added only to the IPA domains, Windows machines
diff --git a/ipatests/test_integration/host.py b/ipatests/test_integration/host.py
index 58f1fe4d9b1c1f99bdca03170c3fb16af9e7a089..f16bc548105fab6a0ed5e6b1003490c78586344b 100644
--- a/ipatests/test_integration/host.py
+++ b/ipatests/test_integration/host.py
@@ -26,6 +26,7 @@
 from ipapython import ipautil
 from ipapython.ipa_log_manager import log_mgr
 from ipatests.test_integration import transport
+from ipatests.test_integration.util import check_config_dict_empty
 
 
 class BaseHost(object):
@@ -99,6 +100,10 @@ def from_env(cls, env, domain, hostname, role, index, domain_index):
         external_hostname = env.get(
             'BEAKER%s%s_env%s' % (role.upper(), index, domain_index), None)
 
+        return cls._make_host(domain, hostname, role, ip, external_hostname)
+
+    @staticmethod
+    def _make_host(domain, hostname, role, ip, external_hostname):
         # We need to determine the type of the host, this depends on the domain
         # type, as we assume all Unix machines are in the Unix domain and
         # all Windows machine in a AD domain
@@ -108,8 +113,23 @@ def from_env(cls, env, domain, hostname, role, index, domain_index):
         else:
             cls = Host
 
-        self = cls(domain, hostname, role, ip, external_hostname)
-        return self
+        return cls(domain, hostname, role, ip, external_hostname)
+
+    @classmethod
+    def from_dict(cls, dct, domain, hostname, role):
+        ip = dct.pop('ip', None)
+        external_hostname = dct.pop('external_hostname', None)
+
+        check_config_dict_empty(dct, 'host %s' % hostname)
+
+        return cls._make_host(domain, hostname, role, ip, external_hostname)
+
+    def to_dict(self):
+        return {
+            'ip': self.ip,
+            'role': self.role,
+            'external_hostname': self.external_hostname,
+        }
 
     @property
     def config(self):
diff --git a/ipatests/test_integration/util.py b/ipatests/test_integration/util.py
index 1a1bb3fcc923c9f2721f0a4c1cb7a1ba2ccc2dd8..b5463b7cb64df08daabb2ca68bd411440764c627 100644
--- a/ipatests/test_integration/util.py
+++ b/ipatests/test_integration/util.py
@@ -20,6 +20,13 @@
 import time
 
 
+def check_config_dict_empty(dct, name):
+    """Ensure that no keys are left in a configuration dict"""
+    if dct:
+        raise ValueError('Extra keys in confuguration for %s: %s' %
+                         (name, ', '.join(dct)))
+
+
 def run_repeatedly(host, command, assert_zero_rc=True, test=None,
                 timeout=30, **kwargs):
     """
-- 
1.8.3.1

From 8d8e5ef90d66127a07ee4c81d587352304287f82 Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Thu, 12 Dec 2013 10:14:25 +0100
Subject: [PATCH] test_integration.config: Add environment variables for
 JSON/YAML

Part of the work for: https://fedorahosted.org/freeipa/ticket/3938
---
 ipatests/test_integration/config.py | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/ipatests/test_integration/config.py b/ipatests/test_integration/config.py
index c19d1e84dd00b856ac5dfb5e7e0e3c2590174cc0..b3fe4fb51b254ecd236b8e326b9b3d1efd4de187 100644
--- a/ipatests/test_integration/config.py
+++ b/ipatests/test_integration/config.py
@@ -23,6 +23,7 @@
 import os
 import collections
 import random
+import json
 
 from ipapython import ipautil
 from ipapython.dn import DN
@@ -118,7 +119,12 @@ def to_dict(self):
     def from_env(cls, env):
         """Create a test config from environment variables
 
-        Input variables:
+        If IPATEST_YAML_CONFIG or IPATEST_JSON_CONFIG is set,
+        configuration is read from the named file.
+        For YAML, the PyYAML (python-yaml) library needs to be installed.
+
+        Otherwise, configuration is read from various curiously
+        named environment variables:
 
         See _setting_infos for test-wide settings
 
@@ -144,6 +150,17 @@ def from_env(cls, env):
 
         Also see env_normalize() for alternate variable names
         """
+        if 'IPATEST_YAML_CONFIG' in env:
+            import yaml
+            with open(env['IPATEST_YAML_CONFIG']) as file:
+                data = yaml.safe_load(file)
+            return cls.from_dict(data)
+
+        if 'IPATEST_JSON_CONFIG' in env:
+            with open(env['IPATEST_JSON_CONFIG']) as file:
+                data = json.load(file)
+            return cls.from_dict(data)
+
         env_normalize(env)
 
         kwargs = {s.name: env.get(s.var_name, s.default)
-- 
1.8.3.1

From 81ad524759853dfba5eb644a59c0e4a723ddb93c Mon Sep 17 00:00:00 2001
From: Petr Viktorin <pviktori redhat com>
Date: Thu, 12 Dec 2013 10:19:56 +0100
Subject: [PATCH] ipa-test-config: Add --json and --yaml output options

Also update the man page.

Part of the work for: https://fedorahosted.org/freeipa/ticket/3938
---
 ipatests/ipa-test-config       | 17 ++++++++++++++++-
 ipatests/man/ipa-test-config.1 | 25 ++++++++++++++++++++++++-
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/ipatests/ipa-test-config b/ipatests/ipa-test-config
index fbaf3d57a4c7395afcf1ff81d3b29502563305cc..ea6d2ce5c31b78169387510c7c76e3f90decee21 100755
--- a/ipatests/ipa-test-config
+++ b/ipatests/ipa-test-config
@@ -22,6 +22,7 @@
 import sys
 import os
 import argparse
+import json
 
 from ipalib.constants import FQDN
 from ipatests.test_integration import config
@@ -59,6 +60,14 @@ def main(argv):
                         help='Do not print Simple Vars '
                              '(normally included backwards-compatibility)')
 
+    parser.add_argument('--yaml', action='store_const', dest='output_format',
+                        const='yaml',
+                        help='Output configuration in YAML format')
+
+    parser.add_argument('--json', action='store_const', dest='output_format',
+                        const='json',
+                        help='Output configuration in JSON format')
+
     args = parser.parse_args(argv)
 
     hostsargs = [bool(args.host), bool(args.master), bool(args.replica),
@@ -77,7 +86,13 @@ def main(argv):
 
     conf = config.Config.from_env(os.environ)
 
-    return config.env_to_script(get_object(conf, args).to_env(**kwargs))
+    if args.output_format == 'json':
+        return json.dumps(conf.to_dict(), indent=2)
+    elif args.output_format == 'yaml':
+        import yaml
+        return yaml.safe_dump(conf.to_dict(), default_flow_style=False)
+    else:
+        return config.env_to_script(get_object(conf, args).to_env(**kwargs))
 
 
 def get_object(conf, args):
diff --git a/ipatests/man/ipa-test-config.1 b/ipatests/man/ipa-test-config.1
index 317bd40d86a1e49c73f7e5d529617f40fc2ae564..320d1fe1fb22ee676da51b866870a39d7ccbbc06 100644
--- a/ipatests/man/ipa-test-config.1
+++ b/ipatests/man/ipa-test-config.1
@@ -30,7 +30,7 @@ The FreeIPA integration test suite is configured by setting environment
 variables.
 The ipa\-run\-tests command reads these variables and prints detailed
 configuration for shell-based scripts to standard output.
-The output of ipa\-run\-tests consists of export statements that can be
+The default output of ipa\-run\-tests consists of export statements that can be
 sourced by Bash.
 
 If run without arguments, it prints out configuration specific to the local
@@ -68,10 +68,33 @@ Output configuration for the host with the given role.
 \fB\-\-no\-simple\fR
 Do not output Simple Vars.
 These are normally included for backwards compatibility.
+.TP
+\fB\-\-yaml\fR
+Output configuration in YAML format instead of Bash script.
+This requires the PyYAML library to be installed.
+.TP
+\fB\-\-json\fR
+Output configuration in JSON format instead of Bash script.
 
 .SH "ENVIRONMENT VARIABLES"
 
 .TP
+File\-based configuration:
+
+.TP
+\fB$IPATEST_YAML_CONFIG\fR
+    Specifies a file that contains configuration in YAML format,
+    as given by \fBipa\-test\-config \-\-global \-\-yaml\fR.
+    If given, the other environment variables are ignored.
+    This requires the PyYAML library to be installed.
+
+.TP
+\fB$IPATEST_JSON_CONFIG\fR
+    Specifies a file that contains configuration in JSON format,
+    as given by \fBipa\-test\-config \-\-global \-\-json\fR.
+    If given, the other environment variables are ignored.
+
+.TP
 Domain configuration:
     Domain is implicitly defined by _envX suffix of the environment variables,
     if either AD_envX or MASTER_envX is defined.
-- 
1.8.3.1


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]