Saturday, December 10, 2022

Python regex to match a valid IPv4 address

 


Python regex to match a valid IPv4 address


(?:(?:25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1?[0-9]?[0-9])



Friday, November 22, 2019

Checking existence of an attribute value within a dictionary in Ansible


When you have to search within a list of dictionaries for the existence of a certain attribute value, you can use the 'selectattr' filter.

You specify the attribute you are interested in, the values this attribute is checked against, and after you convert to a list, you get the length. If the length is greater than zero, your list contains at least one item matching the criteria.


downlink_ports|selectattr('type','in','dslam,isam,lightspan')|list|length > 0

References:
https://jinja.palletsprojects.com/en/master/templates/#list-of-builtin-tests
http://www.oznetnerd.com/jinja2-selectattr-filter/

Saturday, August 31, 2019

Formatting numbers in your Ansible/Jinja templates


Quite often you may have to use an index somewhere in your templates. I think the most common scenarion is to track the loop index within a for loop in Jinja2 or a loop in Ansible.

One such scenario is shown below, where I'm trying to produce the configuration for the uplink ports of a router. Notice that I'm using the index variable to differentiate the configuration sections of each port and store it in a different file.

  - name: Generate uplink ports config 
    template:
      src: configtemplates/{{ ansible_network_os }}/uplink_ports_cfg
      dest: deviceconfigs/{{ inventory_hostname }}/{{ index }}_uplink_ports_cfg
    delegate_to: localhost
    changed_when: false
    loop: "{{ uplink_ports }}"
    loop_control:
      index_var: index
    when: uplink_ports is defined

The filenames produced by this task are in the following format
   0_downlink_ports_cfg
   1_downlink_ports_cfg
   2_downlink_ports_cfg
   ..
   8_downlink_ports_cfg
   9_downlink_ports_cfg
  10_downlink_ports_cfg

After producing the config sections I'll use the 'assemble' function in Ansible, to merge all the sections in a common file. I expect the merging to follow the index sequence, which is the default behavior of the 'assemble' module, based on filename sorting.

And everything works well, as long as the index is smaller than 10. If you exceed 10, then the sorting function of 'assemble' will mess up, by merging the 10th section before the 2nd.

In this case you can start indexing with a two digit number and thank god there's an easy way to do that. Just replace  '{{ index }}' with '{{ "%02d"|format(index) }}'.

The filenames produced in this case are in the following format and 'assemble' merges correctly.

  00_downlink_ports_cfg
  01_downlink_ports_cfg
  02_downlink_ports_cfg
   ..
  08_downlink_ports_cfg
  09_downlink_ports_cfg
  10_downlink_ports_cfg

Checking duplicate interfaces or addresses in Ansible

Consider the scenario that you have the following variable consisting of information about the uplink ports of a router.

uplink_ports:
    - {port: "Te0/0/26", ip: 10.10.10.11/31, peer_name: neighbor1, peer_port: Gi0/0/2 }
    - {port: "Te0/0/27", ip: 10.10.10.13/31, peer_name: neighbor2, peer_port: Gi0/0/2 }

Before proceeding to config generation and application on the router, it's a good idea to check for duplicates. Very often, usually when we copy paste, we forget to change all the parameters and this may result in unexpected failures on the network.

One of the approaches is to check for duplicates using the 'assert' module. You actually ask Ansible to check certain conditions and report back, either with a success or a fail message.

  - name: Check uplink ports for duplicates
    assert:
      that: uplink_ports|map(attribute='port')|list|length  == uplink_ports|map(attribute='port')|list|unique|length
      fail_msg: "Duplicates exist in your uplink ports variable. Please revise."
    delegate_to: localhost
    changed_when: false

The tricky part here is the condition you specify to the function. As you see we compare
"uplink_ports|map(attribute='port')|list|length" to "uplink_ports|map(attribute='port')|list|unique|length", but what does it mean?


  • map(attribute='port')|list ==> Will produce a list of items including only the 'port' key of our variable
  • unique ==> Will remove all the duplicates from the previous list
  • length ==> Will calculate the length of the list
So, we compare the length of the list to the length of the same list after having removed the duplicates. This means if our variable had duplicates in the first place, the length of the lists won't match. If no duplicates existed, the length would be the same before and after the 'unique' operation.

You can actually assert multiple conditions at once. If you want to check both the 'port' and the 'ip' keys for duplicates you can do the following.


  - name: Check uplink ports for duplicates
    assert:
      that: 
       - uplink_ports|map(attribute='port')|list|length  == uplink_ports|map(attribute='port')|list|unique|length
       - uplink_ports|map(attribute='ip')|list|length  == uplink_ports|map(attribute='ip')|list|unique|length
      fail_msg: "Duplicates exist in your uplink ports variable. Please revise."
    delegate_to: localhost
    changed_when: false

Wednesday, August 28, 2019

Working with dynamic inventories in Ansible using PHP (part 2)


Following the discussion in part 1, here follows the PHP code that will return the json structure required by Ansible.  It's quite complex and probably there are other better ways to do it, but that's my way.. It is optimized so that only one loop over each table will populate the structure as needed.

The database behind the scenes is PostgreSQL and you can see the structure below. Don't focus on the exact SQL query strings, as they are based on my specific data model. Depending on your model you will have to write your own queries. In the end, irrespectively of your data model,  you need two table structures, one with Group/Level associations and another with Group/Hostname associations. In my case I don't need to include any 'vars' section, so it's not shown below.

Concerning the group membership table, it's quite straight forward. The group table however, requires some special attention. You need to carefully set the grouplevel of each group, keeping in mind the 'modulo 100' and 'modulo' 20 rules. Each group with grouplevel multiple of 100 will be a major group. Each group with grouplevel multiple of 20 will be a subgroup within the major group. It may sound complex, but you actually do it once and you don't need to change frequently.

Group table
groupnamegrouplevel
south               
100
crete               
103
athens              
120
lab                 
121
islands              
140
mikonos              
141
rodos              
142
north               
200
thessaloniki        
201
os                  
300
ios                 
301
iosxr               
302
nxos                
303
junos               
304
function            
700
metro               
702
core                
703
datacenter          
720
spine               
721
leaf                
722

Group membership table
groupnamehostname
crete               
switch2              
lab                 
router1        
thessaloniki        
switch1              
ios                 
router1         
nxos                
switch1              
metro               
router1        
spine               
switch1              












Below you can see the exact PHP code along with some comments to help you understand how it works. I have tried to remove some non-critical parts to make the code more readable. Normally you should do some error checking on several parts..

<?php
  // Create connection to database  $conn pg_connect($conn_string);

  // Query the group table that allows building of groups and children  
  $sql "SELECT groupname, grouplevel FROM YOUR_TABLE ORDER BY grouplevel ASC";
  $groups pg_query($sql);

  // Query the group membership table that allows building of hosts in groups
  $sql "SELECT groupname, hostname FROM YOUR_TABLE ORDER BY grouplevel ASC";
  $groupmembers pg_query($sql);

  // Set some helper variables
  $response = array();
  $parentgroup "";
  $subgroup ""; 
  $subgrouplevel 0;

  // We loop once over the group list. We create the respective arrays and 
  // identify the children of each group based on the grouplevel hierarchy.
  // The hierarchy is based on a modulo 100 function for major groups and 
  // modulo 20 for subgroups. Very important to keep in mind that the group
  // list is sorted based on grouplevel.
  // To avoid conflicts with subgroups we restrict the subgroup range to +20 from 
  // the subgroup level 
  // Example of the hierarchy we achieve for groups residing in the range 200-299: 
  // 200 ( 201, 202, 203, 220, 240), 220 ( 221, 222), 240 ( 241, 242) 

  while ($row pg_fetch_array($groupsnullPGSQL_ASSOC)) {
      $group trim($row['groupname']); 
      $level trim($row['grouplevel']);
      if (!array_key_exists($group,$response)) $response[$group] = array();
      // if true we have identified a major group. just store the name 
      if ($level 100 == 0$parentgroup $group;
      else 
      {
        // if true we have identified a subgroup. store the name and the level and continue
        if ($level 20 == 0) {$subgroup $group$subgrouplevel $level;}

        // at the next loop if we are within the subgroup limits 
        // we set current group as subgroup child
        if ($level $subgrouplevel && $level < ($subgrouplevel 20)) 
        {
          if (!array_key_exists('children',$response[$subgroup])) $response[$subgroup]['children'] = array();  
          array_push($response[$subgroup]['children'],$group);
        }
        else // otherwise we set current group as parentgroup child
        {
          if (!array_key_exists('children',$response[$parentgroup])) $response[$parentgroup]['children'] = array();  
          array_push($response[$parentgroup]['children'],$group);
        }
      }
  }
  // We have finished setting children for each group. Time to deal with hosts
  // We loop over the group membership list. We identify the hosts and set them 
  // to their respective groups
  while ($row pg_fetch_array($groupmembersnullPGSQL_ASSOC)) {
    $group trim($row['groupname']); 
    $host trim($row['hostname']);
    if (!array_key_exists('hosts',$response[$group])) $response[$group]['hosts'] = array();
    array_push($response[$group]['hosts'],$host);
  }
  // release resources and close connection to database
  pg_free_result($groupmembers);
  pg_free_result($groups);
  pg_close($conn);
  
  // encode the array as json send it back
  echo json_encode($response);

?>

Executing the code you would get something like the following json

{"south":{"children":["crete","athens","islands"]},"crete":{"hosts":["switch2"]},"athens":{"children":["lab"]},"lab":{"hosts":["router1"]},"islands":{"children":["mikonos","rodos"]},"mikonos":[],"rodos":[],"north":{"children":["thessaloniki"]},"thessaloniki":{"hosts":["switch1"]},"os":{"children":["ios","iosxr","nxos","junos"]},"ios":{"hosts":["router1"]},"iosxr":[],"nxos":{"hosts":["switch1"]},"junos":[],"function":{"children":["metro","core","datacenter"]},"metro":{"hosts":["router1"]},"core":[],"datacenter":{"children":["spine","leaf"]},"spine":{"hosts":["switch1"]},"leaf":[]}

This json structure is acceptable by Ansible and works quite well as you can see below. I use the dynamic inventory and ask Ansible to return the groups associated with a specific host.

ansible-playbook -i ./get_inventory.php --limit router1 get_host_groups.yml 

PLAY [all] *********************************************************************************************************

TASK [show group associations for the host(s)] *********************************************************************************************************
ok: [router1 -> localhost] => 
  msg:
  - athens
  - function
  - ios
  - lab
  - metro
  - os
  - south

PLAY RECAP **********************************************************************************************************
router1                    : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

As you see, inheritance works well and although I have set only 'lab', 'ios' and 'metro' groups for my host, it is also associated with the parent groups, not explicitly specified in the membership table.

Tuesday, August 27, 2019

Working with dynamic inventories in Ansible using PHP (part 1)

As you probably know, Ansible has a great group inheritance mechanism. If you build the inventory file carefully, the benefits of inheritance are significant.

If you have a host belonging to group 'childgroup' and this group is a child of 'parentgroup' the variables set in the 'parentgroup' are inherited by the 'childgroup'. This way you avoid setting variables in multiple places, which is considered a best practice when writing playbooks.

Having to maintain a text file with few groups and hosts works quite well, but when you want to scale, you probably want to keep a database with your groups, hosts and the respective membership. The choice of database is a matter of personal preference, in my case I chose PostgreSQL.

There are several ways to do that and Ansible accepts several types of dynamic inventories. Instead of giving a filename as inventory in the command line, you specify an executable file written in any language you like, that returns json encoded data. I'm more familiar with PHP so I decided to use it instead of Python or another language. The command line looks like the following

ansible-playbook -i get_inventory.php my_sample_playbook.yml 

The data that is returned by the script must be in the format shown below and is documented at Developing dynamic inventory
{
    "group001": {
        "hosts": ["host001", "host002"],
        "vars": {
            "var1": true
        },
        "children": ["group002"]
    },
    "group002": {
        "hosts": ["host003","host004"],
        "vars": {
            "var2": 500
        },
        "children":[]
    }

}
Keep in mind that you don't have to return the 'vars' and 'children' sections if you don't actually utilize them. Ansible will accept the data structure, without any complain, even with just the 'hosts' section.

Well.. it's quite easy to say it.. but not so easy to develop such a script that will return this kind of structure.. We'll see that in part 2 of this story!






Saturday, August 24, 2019

Creating access-list wildcard masks for Cisco in Ansible

Working with access lists for Cisco IOS in Ansible is almost a nightmare by itself. This is due to the fact that you need to handle the exact position of each entry and you may have to remove the complete access list before you do anything.

One more thing to take into account is handling of wildcard bits. If you have defined your variables in CIDR notation you need to calculate the wildcard (or don't-care) bits before actually using them.

Just recently I found out there is a filter in Jinja2 that does exactly this calculation. It's an option in the ipaddr filter called 'hostmask'. It seems this filter is not so popular and I found very few references online, nevertheless it works quite well!

{{ mycidrvariable | ipaddr('hostmask') }}

For example if you apply this filter on '10.10.8.16/28' you will get '0.0.0.15'