Python regex to match a valid IPv4 address

 

Python regex to match a valid IPv4 address

(?:(?:25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])

Checking existence of an attribute value within a dictionary in Ansible

When you have to search within a list of dictionaries for the existence of a certain attribute value, you can use the 'selectattr' filter.

You specify the attribute you are interested in, the values this attribute is checked against, and after you convert to a list, you get the length. If the length is greater than zero, your list contains at least one item matching the criteria.


downlink_ports|selectattr('type','in','dslam,isam,lightspan')|list|length > 0

References:
https://jinja.palletsprojects.com/en/master/templates/#list-of-builtin-tests
http://www.oznetnerd.com/jinja2-selectattr-filter/

Formatting numbers in your Ansible/Jinja templates

Quite often you may have to use an index somewhere in your templates. I think the most common scenarion is to track the loop index within a for loop in Jinja2 or a loop in Ansible.

One such scenario is shown below, where I'm trying to produce the configuration for the uplink ports of a router. Notice that I'm using the index variable to differentiate the configuration sections of each port and store it in a different file.

  - name: Generate uplink ports config 
    template:
      src: configtemplates/{{ ansible_network_os }}/uplink_ports_cfg
      dest: deviceconfigs/{{ inventory_hostname }}/{{ index }}_uplink_ports_cfg
    delegate_to: localhost
    changed_when: false
    loop: "{{ uplink_ports }}"
    loop_control:
      index_var: index
    when: uplink_ports is defined

The filenames produced by this task are in the following format
   0_downlink_ports_cfg
   1_downlink_ports_cfg
   2_downlink_ports_cfg
   ..
   8_downlink_ports_cfg
   9_downlink_ports_cfg
  10_downlink_ports_cfg

After producing the config sections I'll use the 'assemble' function in Ansible, to merge all the sections in a common file. I expect the merging to follow the index sequence, which is the default behavior of the 'assemble' module, based on filename sorting.

And everything works well, as long as the index is smaller than 10. If you exceed 10, then the sorting function of 'assemble' will mess up, by merging the 10th section before the 2nd.

In this case you can start indexing with a two digit number and thank god there's an easy way to do that. Just replace  '{{ index }}' with '{{ "%02d"|format(index) }}'.

The filenames produced in this case are in the following format and 'assemble' merges correctly.

  00_downlink_ports_cfg
  01_downlink_ports_cfg
  02_downlink_ports_cfg
   ..
  08_downlink_ports_cfg
  09_downlink_ports_cfg
  10_downlink_ports_cfg

Checking duplicate interfaces or addresses in Ansible

Consider the scenario that you have the following variable consisting of information about the uplink ports of a router.

uplink_ports:
    - {port: "Te0/0/26", ip: 10.10.10.11/31, peer_name: neighbor1, peer_port: Gi0/0/2 }
    - {port: "Te0/0/27", ip: 10.10.10.13/31, peer_name: neighbor2, peer_port: Gi0/0/2 }

Before proceeding to config generation and application on the router, it's a good idea to check for duplicates. Very often, usually when we copy paste, we forget to change all the parameters and this may result in unexpected failures on the network.

One of the approaches is to check for duplicates using the 'assert' module. You actually ask Ansible to check certain conditions and report back, either with a success or a fail message.

  - name: Check uplink ports for duplicates
    assert:
      that: uplink_ports|map(attribute='port')|list|length  == uplink_ports|map(attribute='port')|list|unique|length
      fail_msg: "Duplicates exist in your uplink ports variable. Please revise."
    delegate_to: localhost
    changed_when: false

The tricky part here is the condition you specify to the function. As you see we compare
"uplink_ports|map(attribute='port')|list|length" to "uplink_ports|map(attribute='port')|list|unique|length", but what does it mean?

• map(attribute='port')|list ==> Will produce a list of items including only the 'port' key of our variable
• unique ==> Will remove all the duplicates from the previous list
• length ==> Will calculate the length of the list

So, we compare the length of the list to the length of the same list after having removed the duplicates. This means if our variable had duplicates in the first place, the length of the lists won't match. If no duplicates existed, the length would be the same before and after the 'unique' operation.

You can actually assert multiple conditions at once. If you want to check both the 'port' and the 'ip' keys for duplicates you can do the following.

  - name: Check uplink ports for duplicates

    assert:
      that: 
       - uplink_ports|map(attribute='port')|list|length  == uplink_ports|map(attribute='port')|list|unique|length
       - uplink_ports|map(attribute='ip')|list|length  == uplink_ports|map(attribute='ip')|list|unique|length
      fail_msg: "Duplicates exist in your uplink ports variable. Please revise."
    delegate_to: localhost
    changed_when: false

Working with dynamic inventories in Ansible using PHP (part 2)

Following the discussion in part 1, here follows the PHP code that will return the json structure required by Ansible.  It's quite complex and probably there are other better ways to do it, but that's my way.. It is optimized so that only one loop over each table will populate the structure as needed.

The database behind the scenes is PostgreSQL and you can see the structure below. Don't focus on the exact SQL query strings, as they are based on my specific data model. Depending on your model you will have to write your own queries. In the end, irrespectively of your data model,  you need two table structures, one with Group/Level associations and another with Group/Hostname associations. In my case I don't need to include any 'vars' section, so it's not shown below.

Concerning the group membership table, it's quite straight forward. The group table however, requires some special attention. You need to carefully set the grouplevel of each group, keeping in mind the 'modulo 100' and 'modulo' 20 rules. Each group with grouplevel multiple of 100 will be a major group. Each group with grouplevel multiple of 20 will be a subgroup within the major group. It may sound complex, but you actually do it once and you don't need to change frequently.

Group table groupname grouplevel south 100 crete 103 athens 120 lab 121 islands 140 mikonos 141 rodos 142 north 200 thessaloniki 201 os 300 ios 301 iosxr 302 nxos 303 junos 304 function 700 metro 702 core 703 datacenter 720 spine 721 leaf 722

Group membership table groupname hostname crete switch2 lab router1 thessaloniki switch1 ios router1 nxos switch1 metro router1 spine switch1

Below you can see the exact PHP code along with some comments to help you understand how it works. I have tried to remove some non-critical parts to make the code more readable. Normally you should do some error checking on several parts..

<?php
  // Create connection to database  $conn = pg_connect($conn_string);

  // Query the group table that allows building of groups and children  
  $sql = "SELECT groupname, grouplevel FROM YOUR_TABLE ORDER BY grouplevel ASC";
  $groups = pg_query($sql);

  // Query the group membership table that allows building of hosts in groups
  $sql = "SELECT groupname, hostname FROM YOUR_TABLE ORDER BY grouplevel ASC";
  $groupmembers = pg_query($sql);

  // Set some helper variables
  $response = array();
  $parentgroup = "";
  $subgroup = ""; 
  $subgrouplevel = 0;

  // We loop once over the group list. We create the respective arrays and 
  // identify the children of each group based on the grouplevel hierarchy.
  // The hierarchy is based on a modulo 100 function for major groups and 
  // modulo 20 for subgroups. Very important to keep in mind that the group
  // list is sorted based on grouplevel.
  // To avoid conflicts with subgroups we restrict the subgroup range to +20 from 
  // the subgroup level 
  // Example of the hierarchy we achieve for groups residing in the range 200-299: 
  // 200 ( 201, 202, 203, 220, 240), 220 ( 221, 222), 240 ( 241, 242) 

  while ($row = pg_fetch_array($groups, null, PGSQL_ASSOC)) {
      $group = trim($row['groupname']); 
      $level = trim($row['grouplevel']);
      if (!array_key_exists($group,$response)) $response[$group] = array();
      // if true we have identified a major group. just store the name 
      if ($level % 100 == 0) $parentgroup = $group;
      else 
      {
        // if true we have identified a subgroup. store the name and the level and continue
        if ($level % 20 == 0) {$subgroup = $group; $subgrouplevel = $level;}

        // at the next loop if we are within the subgroup limits 
        // we set current group as subgroup child
        if ($level > $subgrouplevel && $level < ($subgrouplevel + 20)) 
        {
          if (!array_key_exists('children',$response[$subgroup])) $response[$subgroup]['children'] = array();  
          array_push($response[$subgroup]['children'],$group);
        }
        else // otherwise we set current group as parentgroup child
        {
          if (!array_key_exists('children',$response[$parentgroup])) $response[$parentgroup]['children'] = array();  
          array_push($response[$parentgroup]['children'],$group);
        }
      }
  }
  // We have finished setting children for each group. Time to deal with hosts
  // We loop over the group membership list. We identify the hosts and set them 
  // to their respective groups
  while ($row = pg_fetch_array($groupmembers, null, PGSQL_ASSOC)) {
    $group = trim($row['groupname']); 
    $host = trim($row['hostname']);
    if (!array_key_exists('hosts',$response[$group])) $response[$group]['hosts'] = array();
    array_push($response[$group]['hosts'],$host);
  }
  // release resources and close connection to database
  pg_free_result($groupmembers);
  pg_free_result($groups);
  pg_close($conn);
  
  // encode the array as json send it back
  echo json_encode($response);

?>

Executing the code you would get something like the following json

{"south":{"children":["crete","athens","islands"]},"crete":{"hosts":["switch2"]},"athens":{"children":["lab"]},"lab":{"hosts":["router1"]},"islands":{"children":["mikonos","rodos"]},"mikonos":[],"rodos":[],"north":{"children":["thessaloniki"]},"thessaloniki":{"hosts":["switch1"]},"os":{"children":["ios","iosxr","nxos","junos"]},"ios":{"hosts":["router1"]},"iosxr":[],"nxos":{"hosts":["switch1"]},"junos":[],"function":{"children":["metro","core","datacenter"]},"metro":{"hosts":["router1"]},"core":[],"datacenter":{"children":["spine","leaf"]},"spine":{"hosts":["switch1"]},"leaf":[]}

This json structure is acceptable by Ansible and works quite well as you can see below. I use the dynamic inventory and ask Ansible to return the groups associated with a specific host.

ansible-playbook -i ./get_inventory.php --limit router1 get_host_groups.yml 

PLAY [all] *********************************************************************************************************

TASK [show group associations for the host(s)] *********************************************************************************************************
ok: [router1 -> localhost] => 
  msg:
  - athens
  - function
  - ios
  - lab
  - metro
  - os
  - south

PLAY RECAP **********************************************************************************************************
router1                    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

As you see, inheritance works well and although I have set only 'lab', 'ios' and 'metro' groups for my host, it is also associated with the parent groups, not explicitly specified in the membership table.

Working with dynamic inventories in Ansible using PHP (part 1)

As you probably know, Ansible has a great group inheritance mechanism. If you build the inventory file carefully, the benefits of inheritance are significant.

If you have a host belonging to group 'childgroup' and this group is a child of 'parentgroup' the variables set in the 'parentgroup' are inherited by the 'childgroup'. This way you avoid setting variables in multiple places, which is considered a best practice when writing playbooks.

Having to maintain a text file with few groups and hosts works quite well, but when you want to scale, you probably want to keep a database with your groups, hosts and the respective membership. The choice of database is a matter of personal preference, in my case I chose PostgreSQL.

There are several ways to do that and Ansible accepts several types of dynamic inventories. Instead of giving a filename as inventory in the command line, you specify an executable file written in any language you like, that returns json encoded data. I'm more familiar with PHP so I decided to use it instead of Python or another language. The command line looks like the following

ansible-playbook -i get_inventory.php my_sample_playbook.yml 

The data that is returned by the script must be in the format shown below and is documented at Developing dynamic inventory

{
    "group001": {
        "hosts": ["host001", "host002"],
        "vars": {
            "var1": true
        },
        "children": ["group002"]
    },
    "group002": {
        "hosts": ["host003","host004"],
        "vars": {
            "var2": 500
        },
        "children":[]
    }

}

Keep in mind that you don't have to return the 'vars' and 'children' sections if you don't actually utilize them. Ansible will accept the data structure, without any complain, even with just the 'hosts' section.

Well.. it's quite easy to say it.. but not so easy to develop such a script that will return this kind of structure.. We'll see that in part 2 of this story!

Creating access-list wildcard masks for Cisco in Ansible

Working with access lists for Cisco IOS in Ansible is almost a nightmare by itself. This is due to the fact that you need to handle the exact position of each entry and you may have to remove the complete access list before you do anything.

One more thing to take into account is handling of wildcard bits. If you have defined your variables in CIDR notation you need to calculate the wildcard (or don't-care) bits before actually using them.

Just recently I found out there is a filter in Jinja2 that does exactly this calculation. It's an option in the ipaddr filter called 'hostmask'. It seems this filter is not so popular and I found very few references online, nevertheless it works quite well!

{{ mycidrvariable | ipaddr('hostmask') }}

For example if you apply this filter on '10.10.8.16/28' you will get '0.0.0.15'