Adding and deleting maas subnets and vlans in parallel from a csv file

maas-mass-vlan+sn-add

#Consider we have the following csv and we want to use it to create subnets and vlans in MAAS using the cli

VLAN (name+tag),cidr,subnet name
v501,10.183.128.0/24,fa-POD5-v501
v503,10.183.129.0/24,fa-POD5-v503
v505,10.183.130.0/24,fa-POD5-v505
v507,10.183.131.0/24,fa-POD5-v507
v509,10.183.132.0/24,fa-POD5-v509
v511,10.183.133.0/24,fa-POD5-v511
v513,10.183.134.0/24,fa-POD5-v513
v515,10.183.135.0/24,fa-POD5-v515
v517,10.183.136.0/24,fa-POD5-v517
v519,10.183.137.0/24,fa-POD5-v519
v521,10.183.138.0/24,fa-POD5-v521
v523,10.183.139.0/24,fa-POD5-v523
v525,10.183.140.0/24,fa-POD5-v525
v527,10.183.141.0/24,fa-POD5-v527
v529,10.183.142.0/24,fa-POD5-v529
v531,10.183.143.0/24,fa-POD5-v531
v533,10.183.144.0/24,fa-POD5-v533
v535,10.183.145.0/24,fa-POD5-v535
v537,10.183.146.0/24,fa-POD5-v537
v539,10.183.147.0/24,fa-POD5-v539
v541,10.183.148.0/24,fa-POD5-v541
v543,10.183.149.0/24,fa-POD5-v543
v545,10.183.150.0/24,fa-POD5-v545
v547,10.183.151.0/24,fa-POD5-v547
v549,10.183.152.0/24,fa-POD5-v549
v551,10.183.153.0/24,fa-POD5-v551
v553,10.183.154.0/24,fa-POD5-v553
v555,10.183.155.0/24,fa-POD5-v555
v557,10.183.156.0/24,fa-POD5-v557
v559,10.183.157.0/24,fa-POD5-v559
v561,10.183.158.0/24,fa-POD5-v561
v563,10.183.159.0/24,fa-POD5-v563
v565,10.183.160.0/24,fa-POD5-v565
v567,10.183.161.0/24,fa-POD5-v567
v569,10.183.162.0/24,fa-POD5-v569
v571,10.183.163.0/24,fa-POD5-v571

### Creation ###

# The followng command will add vlans, subnets, and each subnet's gateway to MAAS in one command based on the above csv file

awk -F, 'NR>1 {print $1,$2,$3}' subnet.csv|\
xargs \
-n3 \
-P0 \
bash -c 'VLAN_ID=$(maas admin vlans create 0 name=$0 vid=${0##*v}|jq -r .id) && maas admin subnets create cidr=$1 name=$2 vlan=$VLAN_ID vid=${0##*v} gateway_ip=${1/.0\/24/.1}'

# Commamd Notes:
# awk:
# -F, tells awk to separate fields based on a comma
# NR>1 tells awk to skip the first line (the header in the csv file)
# The output is piped to xargs for further processing

# xargs
# When passing commands to another shell within xargs the first argument is $0, the second is $1, etc.
# - - this is different from normal bash where $0 is the entire command line
# -n3 tells xargs there are three arguments per command line
# -P0 tells xargs to run as many command lines in parallel as possible (e.g. -P5 would tell it to run 5 in parallel, wait until those 5 complete, then run next 5,etc)
# bash -c tells xargs to launch a new bash shell process for each command line
# - The commands are wrapped in single quotes to prevent the variables from being pulled from the current environment
# Each shell is going to run at least one, but hopefully, two commands
# - the "&&" tells bash only to run if the previous command exited cleanly (e.g. $? = 0 ) 

# The first command: VLAN_ID=$(maas admin vlans create 0 name=$0 vid=${0##*v}|jq -r .id)
# - is setup as a parameter whose value set from the command's output being piped through jq to parse only the new vlan's MAAS vlan id
# - - It's important to note that the MAAS vlan id is different that the vlan tag which is known in MAAS as the "vid"
# - adds a vlan to the default fabric-0 (MAAS fabric_id 0).
# - - vids (vlan tags) need to be unique per fabric
# - - Since all the vids being added are unique, we can use the same fabric for all vlans and subnets
# - The vlan is named using the first argument
# - The vid (vlan tag) is derived from from the first argument using ${0##*v} which prints everthing after the "v"  

# The second command: 
# - Adds a subnet to maas with the desired name (3rd argument) (xargs $2)
# - the cidr (second argument)(xargs $1)
# - the appropriate MAAS vlan ID (captured from the first command)
# - the vlan tag (derived from the first argument (xargs $0) using bash parameter expansion ${0##*v} which drops the "v")
# - the gateway_ip (derived from the second argument (xargs $1) with ${1/.0\/24/.1} which replaces ".0/24" with .1    



### Deletion ###

# We can also delete all the subnets and vlans based on the above csv file.
# To do is as a one liner is a little tricky since we are feeding a command information from two different sources
# - 1) via a pipe (|)
# - 2) via a "here string" (<<<)
# - And we have to protect of the single quotes of the jq when nested inside the single quotes of the bash -c 

export SUBNETS=$(maas admin subnets read)
(awk -F, 'NR>1 {print $2}' subnet.csv|\
xargs \
-n1 \
-P0 \
bash -c 'jq -r '"'"'.[]|select(.cidr=="'"'"'${0}'"'"'")|"\(.id) \(.vlan.fabric_id) \(.vlan.vid)"'"'"' <<< ${SUBNETS}')|\
xargs \
-n3 \
-P0 \
bash -c 'maas admin subnet delete $0 && maas admin vlan delete $1 $2'

# Command notes:
# export:
# - To reduce the number of calls to MAAS, first we will dump all the subnet information from MAAS and save as a parameter
# - The command is exported so it available to our child shells that will be called via xargs

# The awk and first args commands are enclosed in parentheses to ensure they execute together, the output is then piped to xargs for final processing

# awk:
# -F, tells awk to separate fields based on a comma
# NR>1 tells awk to skip the first line (the header in the csv file)
# {print $2} - Unlike during creation, only need the subnet cidr is needed.
# - We can derive all the other information by parsing the ${SUBNET} variable and using the cidr as the element of the parse
# The results are piped to xargs for further processing

# xargs (1)
# When passing commands to another shell within xargs the first argument is $0, the second is $1, etc.
# - - this is different from normal bash where $0 is the entire command line
# -n1 tells xargs there are three arguments per command line
# -P0 tells xargs to run as many command lines in parallel as possible (e.g. -P5 would tell it to run 5 in parallel, wait until those 5 complete, then run next 5,etc)
# bash -c tells xargs to launch a new bash shell process for each command line
# The commands are wrapped in single quotes to prevent the variables from being pulled from the current environment
# - - extra care is used to protect the single quotes around the jq command as this command is using xargs + 

#The first command: jq -r '"'"'.[]|select(.cidr=="'"'"'${0}'"'"'")|"\(.id) \(.vlan.fabric_id) \(.vlan.vid)"'"'"'
# - Sets the cidr value to the first argument passed to xargs (xarg $0) from awk (awk's $2 or the second column in the csv)
# - - This value is used as selection critera for the information we need to delete the subnet and vlans from a MAAS fabric
# - - the select command limits the other json keys to only those that match the cidr
# - With the cidr set, we can extract the MAAS subnet id, the fabric id that vlan rides on, the vid (vlan tag) from the ${SUBNET} variable
# - Using <<< prevents jq from spawning another subshell (we've already spawned one via xargs)
# - - It also prevents multiple calls to lookup the subnet on a per-cidr basisi
# - - The ${SUBNET} variable is available to the xargs spawned shell because it's a parent child of the where we ran the export command
# Output is then piped to xargs (again) for final processing

#xargs (2)
# -n3 tells xargs there are three arguments per command line
# -P0 tells xargs to run as many command lines in parallel as possible (e.g. -P5 would tell it to run 5 in parallel, wait until those 5 complete, then run next 5,etc)
# bash -c tells xargs to launch a new bash shell process for each command line
# - The commands are wrapped in single quotes to prevent the variables from being pulled from the current environment
# Each shell is going to run at least one, but hopefully, two commands
# - the "&&" tells bash only to run if the previous command exited cleanly (e.g. $? = 0 ) 

# First command: maas admin subnet delete $0
# Deletes the maas subnet, this is because we cannot delete a vlan that has subnets defined
# - To delete a subnet, we need the MAAS subnet id
# - - This was derived from the jq parsing of the ${SUBNETS} variable using the cidr from the csv and passed to xargs as the first argument (xargs $0)

# Second command: maas admin vlan delete $1 $2
# Delete the MAAS vlan
# - This command only runs if the first was successful
# - To delete a vlan, we need the MAAS fabric id and the vid (vlan tag) assocated with the MAAS vlan ID for the subnet.  
# - - These were derived from the jq parsing of the ${SUBNETS} variable using the cidr from the csv and passed to xargs as the second (xargs $1) and third arguments (xargs $2)

Video of above in action:
https://vimeo.com/thinguy/maas-tricks-vlans

Would be awesome to do this with Ansible! :-D
I'm trying it right now... Thank you for it!