The first step when au­tomat­ing a net­work is to build the source of truth. A source of truth is a repos­i­tory of data that pro­vides the in­tended state: the list of de­vices, the IP ad­dresses, the net­work pro­to­cols set­tings, the time servers, etc. A pop­u­lar choice is NetBox. Its doc­u­men­ta­tion high­lights its us­age as a source of truth:

NetBox in­tends to rep­re­sent the de­sired state of a net­work ver­sus its op­er­a­tional state. As such, au­to­mated im­port of live net­work state is strongly dis­cour­aged. All data cre­ated in NetBox should first be vet­ted by a hu­man to en­sure its in­tegrity. NetBox can then be used to pop­u­late mon­i­tor­ing and pro­vi­sion­ing sys­tems with a high de­gree of con­fidence.

When in­tro­duc­ing Jerikan, a com­mon feed­back we got was: “you should use NetBox for this.” In­deed, Jerikan’s source of truth is a bunch of YAML files ver­sioned with Git.

Why Git?#

If we look at how things are done with servers and ser­vices, in a dat­a­cen­ter or in the cloud, we are likely to find users of Ter­raform, a tool turn­ing declar­a­tive con­fig­u­ra­tion files into in­fra­struc­ture. Declar­a­tive con­fig­u­ra­tion man­age­ment tools like Salt, Pup­pet,¹ or An­si­ble take care of server con­fig­u­ra­tion. NixOS is an al­ter­na­tive: it com­bines pack­age man­age­ment and con­fig­u­ra­tion man­age­ment with a func­tional lan­guage to build vir­tual ma­chines and con­tain­ers. When us­ing a Ku­ber­netes clus­ter, peo­ple use Kus­tomize or Helm, two other declar­a­tive con­fig­u­ra­tion man­age­ment tools. Taped to­gether, these tools im­ple­ment the in­fra­struc­ture as code paradigm.

In­fra­struc­ture as code is an ap­proach to in­fra­struc­ture au­to­ma­tion based on prac­tices from soft­ware de­vel­op­ment. It em­pha­sizes con­sis­tent, re­peat­able rou­tines for pro­vi­sion­ing and chang­ing sys­tems and their con­fig­u­ra­tion. You make changes to code, then use au­to­ma­tion to test and ap­ply those changes to your systems.

― Kief Mor­ris, In­fra­struc­ture as Code, O’Reilly.

A ver­sion con­trol sys­tem is a cen­tral tool for in­fra­struc­ture as code. The usual can­di­date is Git with a source code man­age­ment sys­tem like GitLab or GitHub. You get:

Trace­abil­ity and vis­i­bil­ity

Git keeps a log of all changes: what, who, why, and when. With a bit of dis­ci­pline, each change is ex­plained and self-con­tained. It be­comes part of the in­fra­struc­ture doc­u­men­ta­tion. When the sup­port team com­plains about a de­graded ex­pe­ri­ence for some cus­tomers over the last two months or so, you quickly dis­cover this may be re­lated to a change to an in­com­ing pol­icy in New York.

Rolling back

If a change is de­fec­tive, it can be re­verted quickly, safely, and with­out much ef­fort, even if other changes hap­pened in the mean­time. The pol­icy change at the ori­gin of the prob­lem spanned over three routers. Re­vert­ing this spe­cific change and de­ploy­ing the con­fig­u­ra­tion let you solve the sit­u­a­tion un­til you find a better fix.

Branch­ing, re­view­ing, merg­ing

When work­ing on a new fea­ture or refac­tor­ing some part of the in­fra­struc­ture, a team mem­ber cre­ates a branch and works on their change with­out in­ter­fer­ing with the work of other mem­bers. Once the branch is ready, a pull re­quest is cre­ated and the change is ready to be re­viewed by the other team mem­bers be­fore merg­ing. You dis­cover the is­sue was re­lated to di­vert­ing traf­fic through an IX where one ISP was con­nected with­out enough ca­pac­ity. You pro­pose and dis­cuss a fix that in­cludes a change of the schema and the tem­plates used to de­clare poli­cies to be able to han­dle this case.

Con­tin­u­ous in­te­gra­tion

For each change, au­to­mated tests are trig­gered. They can de­tect prob­lems and give more de­tails on the ef­fect of a change. Branches can be de­ployed to a test in­fra­struc­ture where re­gres­sion tests are ex­e­cuted. The re­sults can be syn­the­sized as a com­ment in the pull re­quest to help the re­view. You check your pro­posed change does not mod­ify the other ex­ist­ing policies.

Why not NetBox?#

NetBox does not share these fea­tures. It is a data­base with a REST and a GraphQL API. Trace­abil­ity is lim­ited: changes are not grouped into a trans­ac­tion and they are not doc­u­mented. You can­not fork the data­base. Usu­ally, there is one stag­ing data­base to test mod­i­fi­ca­tions be­fore ap­ply­ing them to the pro­duc­tion data­base. It does not scale well and re­views are dif­fi­cult. Ap­ply­ing the same change to the pro­duc­tion data­base can be haz­ardous. Rolling back a change is non-trivial.

More­over, NetBox is not usu­ally the sin­gle source of truth. It con­tains your hard­ware in­ven­tory, the IP ad­dresses, and some topol­ogy in­for­ma­tion. How­ever, this is not the place you put au­tho­rized SSH keys, sys­log servers, or the BGP con­fig­u­ra­tion. If you also use An­si­ble, this in­for­ma­tion ends in its in­ven­tory. The source of truth is there­fore frag­mented be­tween sev­eral tools with dif­fer­ent work­flows. Since NetBox 2.7, you can ap­pend ad­di­tional data with con­fig­u­ra­tion con­texts. This mit­i­gates this point. The data is ar­ranged hi­er­ar­chi­cally but the hi­er­ar­chy can­not be cus­tom­ized.² Nau­to­bot can man­age con­fig­u­ra­tion con­texts in a Git repos­i­tory, while still al­low­ing the use of the API to fetch them. You get some ad­di­tional perks, thanks to Git, but the re­main­ing data is still in a data­base with a dif­fer­ent lifecycle.

Lastly, the schema used by NetBox may not fit your needs and you can­not tweak it. For ex­am­ple, you may have a rule to com­pute the IPv6 ad­dress from the IPv4 ad­dress for dual-stack in­ter­faces. Such a re­la­tion­ship can­not be eas­ily ex­pressed and en­forced in NetBox. When chang­ing the IPv4 ad­dress, you may for­get the IPv6 ad­dress. The source of truth should only con­tain the IPv4 ad­dress but you also want the IPv6 ad­dress in NetBox be­cause this is your IPAM and you need it to up­date your DNS entries.

Why not Git?#

There are some lim­i­ta­tions when putting your source of truth in Git:

1. If you want to ex­pose a web in­ter­face to al­low an ex­ter­nal team to re­quest a change, it is more dif­fi­cult to do it with Git than with a data­base. Out-of-the-box, NetBox pro­vides a nice web in­ter­face and a per­mis­sion sys­tem. You can also write your own web in­ter­face and in­ter­act with NetBox through its API.
2. YAML files are more dif­fi­cult to query in dif­fer­ent ways. For ex­am­ple, look­ing for a free IP ad­dress is com­plex if they are scat­tered in mul­ti­ple places.

In my opin­ion, in most cases, you are bet­ter off putting the source of truth in Git in­stead of NetBox. You get a lot of perks by do­ing that and you can still use NetBox as a read-only view, us­able by other tools. We do that with an An­si­ble mod­ule. In the re­main­ing cases, Git could still fit the bill. Read-only ac­cess con­trol can be done through sub­mod­ules. Pull re­quests can re­strict write ac­cess: a bot can check the changes only mod­ify al­lowed files be­fore auto-merg­ing. This still re­quires some Git knowl­edge, but many teams are now com­fort­able us­ing Git, thanks to its ubiquity.