The netbox.netbox col­lec­tion from An­si­ble Gal­axy pro­vides sev­eral mod­ules to up­date NetBox objects:

- name: create a device in NetBox
  netbox_device:
    netbox_url: http://netbox.local
    netbox_token: s3cret
    data:
      name: to3-p14.sfo1.example.com
      device_type: QFX5110-48S
      device_role: Compute Switch
      site: SFO1

How­ever, if NetBox is not your source of truth, you may want to en­sure it stays in sync with your con­fig­u­ra­tion man­age­ment data­base¹ by re­mov­ing out­dated de­vices or IP ad­dresses. While it should be pos­si­ble to glue to­gether a play­book with a query, a loop, and some fil­ter­ing to delete un­wanted el­e­ments, it feels clunky, in­ef­fi­cient, and an abuse of YAML as a pro­gram­ming lan­guage. A spe­cific An­si­ble mod­ule solves this is­sue and is likely more flexible.

Code#

The mod­ule has the fol­low­ing sig­na­ture and it syncs NetBox with the con­tent of the pro­vided YAML file:

netbox_sync:
  source: netbox.yaml
  api: https://netbox.example.com
  token: s3cret

The syn­chro­nized ob­jects are:

• sites
• man­u­facturers
• de­vice types
• de­vice roles
• devices
• IP addresses

In our en­vi­ron­ment, the YAML file is gen­er­ated from our con­fig­u­ra­tion man­age­ment data­base and con­tains a set of de­vices and a list of IP ad­dresses:

devices:
  ad2-p6.sfo1.example.com:
     datacenter: sfo1
     manufacturer: Cisco
     model: Catalyst 2960G-48TC-L
     role: net_tor_oob_switch
  to1-p6.sfo1.example.com:
     datacenter: sfo1
     manufacturer: Juniper
     model: QFX5110-48S
     role: net_tor_gpu_switch
# […]
ips:
  - device: ad2-p6.example.com
    ip: 172.31.115.18/21
    interface: oob
  - device: to1-p6.example.com
    ip: 172.31.115.33/21
    interface: oob
  - device: to1-p6.example.com
    ip: 172.31.254.33/32
    interface: lo0.0
# […]

The net­work team is not the sole ten­ant in NetBox. While adding new ob­jects or mod­i­fy­ing ex­ist­ing ones should be rel­a­tively safe, delet­ing un­wanted ob­jects can be risky. The mod­ule only deletes ob­jects it did cre­ate or mod­ify. To iden­tify them, it marks them with a spe­cific tag, cmdb. Most ob­jects in NetBox ac­cept tags.

Module definition#

Start­ing from the skele­ton de­scribed in the pre­vi­ous ar­ti­cle, we de­fine the module:

module_args = dict(
    source=dict(type='path', required=True),
    api=dict(type='str', required=True),
    token=dict(type='str', required=True, no_log=True),
    max_workers=dict(type='int', required=False, default=10)
)

result = dict(
    changed=False
)

module = AnsibleModule(
    argument_spec=module_args,
    supports_check_mode=True
)

It con­tains ad­di­tional op­tional ar­gu­ments defin­ing the num­ber of work­ers to talk to NetBox and query the ex­ist­ing ob­jects in par­al­lel to speed up the ex­ecution.

Abstracting synchronization#

We need to syn­chro­nize dif­fer­ent ob­ject types, but once we have a list of ob­jects we want in NetBox, the grunt work is al­ways the same:

• check if the ob­jects al­ready exist,
• re­trieve them and put them in a form suit­able for com­parison,
• re­trieve the ex­tra ob­jects we don’t want anymore,
• com­pare the two sets, and
• add miss­ing ob­jects, up­date ex­ist­ing ones, delete ex­tra ones.

We code these be­hav­iors into a Synchronizer ab­stract class. For each kind of ob­ject, a con­crete class is built with the ap­pro­pri­ate class at­tributes to tune its be­hav­ior and a wanted() method to pro­vide the ob­jects we want.

I am not ex­plain­ing the ab­stract class code here. Have a look at the source if you want.

Synchronizing tags and tenants#

As a starter, here is how we de­fine the class syn­chro­niz­ing the tags:

class SyncTags(Synchronizer):
    app = "extras"
    table = "tags"
    key = "name"

    def wanted(self):
        return {"cmdb": dict(
            slug="cmdb",
            color="8bc34a",
            description="synced by network CMDB")}

The app and table at­tributes de­fine the NetBox ob­jects we want to ma­nip­u­late. The key at­tribute is used to de­ter­mine how to look up ex­ist­ing ob­jects. In this ex­am­ple, we want to look up tags us­ing their names.

The wanted() method is ex­pected to re­turn a dic­tio­nary map­ping ob­ject keys to the list of wanted at­tributes. Here, the keys are tag names and we cre­ate only one tag, cmdb, with the pro­vided slug, color, and de­scrip­tion. This is the tag we will use to mark the ob­jects we cre­ate or modify.

If the tag does not ex­ist, it is cre­ated. If it ex­ists, the pro­vided at­tributes are up­dated. Other at­tributes are left un­touched.

We also want to cre­ate a spe­cific ten­ant for ob­jects ac­cept­ing such an at­tribute (de­vices and IP ad­dresses):

class SyncTenants(Synchronizer):
    app = "tenancy"
    table = "tenants"
    key = "name"

    def wanted(self):
        return {"Network": dict(slug="network",
                                description="Network team")}

Synchronizing sites#

We also need to syn­chro­nize the list of sites. This time, the wanted() method uses the in­for­ma­tion pro­vided in the YAML file: it walks the de­vices and builds a set of dat­a­cen­ter names.

class SyncSites(Synchronizer):

    app = "dcim"
    table = "sites"
    key = "name"
    only_on_create = ("status", "slug")

    def wanted(self):
        result = set(details["datacenter"]
                     for details in self.source['devices'].values()
                     if "datacenter" in details)
        return {k: dict(slug=k,
                        status="planned")
                for k in result}

Thanks to the use of the only_on_create at­tribute, the spec­i­fied at­tributes are not up­dated if they are dif­fer­ent. The goal of this syn­chro­nizer is mostly to col­lect the ref­er­ences to the dif­fer­ent sites for other objects.

>>> pprint(SyncSites(**sync_args).wanted())
{'sfo1': {'slug': 'sfo1', 'status': 'planned'},
 'chi1': {'slug': 'chi1', 'status': 'planned'},
 'nyc1': {'slug': 'nyc1', 'status': 'planned'}}

Synchronizing manufacturers, device types, and device roles#

The syn­chro­niza­tion of man­u­fac­tur­ers is pretty sim­i­lar, ex­cept we do not use the only_on_create at­tribute:

class SyncManufacturers(Synchronizer):

    app = "dcim"
    table = "manufacturers"
    key = "name"

    def wanted(self):
        result = set(details["manufacturer"]
                     for details in self.source['devices'].values()
                     if "manufacturer" in details)
        return {k: {"slug": slugify(k)}
                for k in result}

Re­gard­ing the de­vice types, we use the foreign at­tribute link­ing a NetBox at­tribute to the syn­chro­nizer han­dling it.

class SyncDeviceTypes(Synchronizer):

    app = "dcim"
    table = "device_types"
    key = "model"
    foreign = {"manufacturer": SyncManufacturers}

    def wanted(self):
        result = set((details["manufacturer"], details["model"])
                     for details in self.source['devices'].values()
                     if "model" in details)
        return {k[1]: dict(manufacturer=k[0],
                           slug=slugify(k[1]))
                for k in result}

The wanted() method refers to the man­u­fac­turer us­ing its key at­tribute. In this case, this is the man­u­fac­turer name.

>>> pprint(SyncManufacturers(**sync_args).wanted())
{'Cisco': {'slug': 'cisco'},
 'Dell': {'slug': 'dell'},
 'Juniper': {'slug': 'juniper'}}
>>> pprint(SyncDeviceTypes(**sync_args).wanted())
{'ASR 9001': {'manufacturer': 'Cisco', 'slug': 'asr-9001'},
 'Catalyst 2960G-48TC-L': {'manufacturer': 'Cisco',
                           'slug': 'catalyst-2960g-48tc-l'},
 'MX10003': {'manufacturer': 'Juniper', 'slug': 'mx10003'},
 'QFX10002-36Q': {'manufacturer': 'Juniper', 'slug': 'qfx10002-36q'},
 'QFX10002-72Q': {'manufacturer': 'Juniper', 'slug': 'qfx10002-72q'},
 'QFX5110-32Q': {'manufacturer': 'Juniper', 'slug': 'qfx5110-32q'},
 'QFX5110-48S': {'manufacturer': 'Juniper', 'slug': 'qfx5110-48s'},
 'QFX5200-32C': {'manufacturer': 'Juniper', 'slug': 'qfx5200-32c'},
 'S4048-ON': {'manufacturer': 'Dell', 'slug': 's4048-on'},
 'S6010-ON': {'manufacturer': 'Dell', 'slug': 's6010-on'}}

The de­vice roles are de­fined like this:

class SyncDeviceRoles(Synchronizer):

    app = "dcim"
    table = "device_roles"
    key = "name"

    def wanted(self):
        result = set(details["role"]
                     for details in self.source['devices'].values()
                     if "role" in details)
        return {k: dict(slug=slugify(k),
                        color="8bc34a")
                for k in result}

Synchronizing devices#

A de­vice is mostly a name with ref­er­ences to a role, a model, a dat­a­cen­ter and a ten­ant. These ref­er­ences are de­clared as for­eign keys us­ing the syn­chro­niz­ers de­fined pre­viously.

class SyncDevices(Synchronizer):
    app = "dcim"
    table = "devices"
    key = "name"
    foreign = {"device_role": SyncDeviceRoles,
               "device_type": SyncDeviceTypes,
               "site": SyncSites,
               "tenant": SyncTenants}
    remove_unused = 10

    def wanted(self):
        return {name: dict(device_role=details["role"],
                           device_type=details["model"],
                           site=details["datacenter"],
                           tenant="Network")
                for name, details in self.source['devices'].items()
                if {"datacenter", "model", "role"} <= set(details.keys())}

The remove_unused at­tribute is a safety im­ple­mented to fail if we have to delete more than 10 de­vices: this may be the in­di­ca­tion there is a bug some­where, un­less one of your dat­a­cen­ters sud­denly caught fire.

>>> pprint(SyncDevices(**sync_args).wanted())
{'ad2-p6.sfo1.example.com': {'device_role': 'net_tor_oob_switch',
                             'device_type': 'Catalyst 2960G-48TC-L',
                             'site': 'sfo1',
                             'tenant': 'Network'},
 'to1-p6.sfo1.example.com': {'device_role': 'net_tor_gpu_switch',
                             'device_type': 'QFX5110-48S',
                             'site': 'sfo1',
                             'tenant': 'Network'},
[…]

Synchronizing IP addresses#

The last step is to syn­chro­nize IP ad­dresses. We do not at­tach them to a de­vice.² In­stead, we spec­ify the de­vice names in the de­scrip­tion of the IP address:

class SyncIPs(Synchronizer):
    app = "ipam"
    table = "ip-addresses"
    key = "address"
    foreign = {"tenant": SyncTenants}
    remove_unused = 1000

    def wanted(self):
        wanted = {}
        for details in self.source['ips']:
            if details['ip'] in wanted:
                wanted[details['ip']]['description'] = \
                    f"{details['device']} (and others)"
            else:
                wanted[details['ip']] = dict(
                    tenant="Network",
                    status="active",
                    dns_name="",        # information is present in DNS
                    description=f"{details['device']}: {details['interface']}",
                    role=None,
                    vrf=None)
        return wanted

There is a slight dif­fi­culty: NetBox al­lows du­pli­cate IP ad­dresses, so a sim­ple lookup is not enough. In case of mul­ti­ple matches, we choose the best by pre­fer­ring those tagged with cmdb, then those al­ready at­tached to an in­terface:

def get(self, key):
    """Grab IP address from NetBox."""
    # There may be duplicate. We need to grab the "best."
    results = super(Synchronizer, self).get(key)
    if len(results) == 0:
        return None
    if len(results) == 1:
        return results[0]
    scores = [0]*len(results)
    for idx, result in enumerate(results):
        if "cmdb" in result.tags:
            scores[idx] += 10
        if result.interface is not None:
            scores[idx] += 5
    return sorted(zip(scores, results),
                  reverse=True, key=lambda k: k[0])[0][1]

Getting the current and wanted states#

Each syn­chro­nizer is ini­tial­ized with a ref­er­ence to the An­si­ble mod­ule, a ref­er­ence to a pynet­box’s API ob­ject, the data con­tained in the pro­vided YAML file and two empty dic­tio­nar­ies for the cur­rent and ex­pected states:

source = yaml.safe_load(open(module.params['source']))
netbox = pynetbox.api(module.params['api'],
                      token=module.params['token'])

sync_args = dict(
    module=module,
    netbox=netbox,
    source=source,
    before={},
    after={}
)
synchronizers = [synchronizer(**sync_args) for synchronizer in [
    SyncTags,
    SyncTenants,
    SyncSites,
    SyncManufacturers,
    SyncDeviceTypes,
    SyncDeviceRoles,
    SyncDevices,
    SyncIPs
]]

Each syn­chro­nizer has a prepare() method whose goal is to com­pute the cur­rent and wanted states. It re­turns True in case of a dif­ference:

# Check what needs to be synchronized
try:
    for synchronizer in synchronizers:
        result['changed'] |= synchronizer.prepare()
except AnsibleError as e:
    result['msg'] = e.message
    module.fail_json(**result)

Applying changes#

Back to the skele­ton de­scribed in the pre­vi­ous ar­ti­cle, the last step is to ap­ply the changes if there is a dif­fer­ence be­tween these states. Each syn­chro­nizer reg­is­ters the cur­rent and wanted states in sync_args["before"][table] and sync_args["after"][table] where table is the name of the ta­ble for a given NetBox ob­ject type. The diff ob­ject is a bit elab­o­rate as it is built ta­ble by ta­ble. This en­ables An­si­ble to dis­play the name of each ta­ble be­fore the diff rep­re­sentation:

# Compute the diff
if module._diff and result['changed']:
    result['diff'] = [
        dict(
            before_header=table,
            after_header=table,
            before=yaml.safe_dump(sync_args["before"][table]),
            after=yaml.safe_dump(sync_args["after"][table]))
        for table in sync_args["after"]
        if sync_args["before"][table] != sync_args["after"][table]
    ]

# Stop here if check mode is enabled or if no change
if module.check_mode or not result['changed']:
    module.exit_json(**result)

Each syn­chro­nizer also ex­poses a synchronize() method to ap­ply changes and a cleanup() method to delete un­wanted ob­jects. Or­der is im­por­tant due to the re­la­tion be­tween the objects.

# Synchronize
for synchronizer in synchronizers:
    synchronizer.synchronize()
for synchronizer in synchronizers[::-1]:
    synchronizer.cleanup()
module.exit_json(**result)

The com­plete code is avail­able on GitHub. Com­pared to us­ing netbox.netbox col­lec­tion, the logic is writ­ten in Python in­stead of try­ing to glue An­si­ble tasks to­gether. I be­lieve this is both more flex­i­ble and eas­ier to read, no­tably when try­ing to delete out­dated ob­jects. While I did not test it, it should also be faster.

An al­ter­na­tive would have been to re­use code from the netbox.netbox col­lec­tion, as it con­tains sim­i­lar prim­i­tives. Un­for­tu­nately, I didn’t think of it un­til now. 😶

1. https://netbox-ansible-collection.readthedocs.io/en/latest/
2. https://galaxy.ansible.com
3. https://netbox.readthedocs.io/en/stable/
4. https://vincent.bernat.ch/en/blog/2020-custom-ansible-module
5. https://vincent.bernat.ch/en/blog/2020-syncing-mysql-tables-ansible
6. https://github.com/vincentbernat/ansible-custom-module-examples/blob/main/netbox_sync.py
7. https://www.attrs.org/en/stable/index.html
8. https://pynetbox.readthedocs.io/en/latest/
9. https://www.redhat.com/en/blog/using-netbox-for-ansible-source-of-truth