first simple python jinja2 writer to control config files of a bunch of openwrt WL APs

ReadMe.md

I want to write a simple python 2.7 jinja2 writer that writes out text config files for WL AccessPoints

I have the following structure in my directory ~/Documents/PyCharmProjects:

config-data.csv
mk-wl-ap-configs.py
shadow.jinja
network.jinja
wireless,jinja

and I ultimatley I want it to write the jinja2 rendered output to subdirs named after the AccessPoints

ap1:
    shadow
    network
    wireless
    
ap2:
    shadow
    network
    wireless

config-data.csv

mk-wl-ap-configs_v1.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os
#
# First things first, we need to import the csv module
import csv

csvFile = 'config-data.csv'
configdata = {}

print('Reading file %s' % csvFile)
configdata=csv.DictReader(open(csvFile, 'rU'), delimiter=',')

print "\nHere is the config-data dictionary "
print configdata

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))

# some debug lines
#print "\nWe are in directory "
#print THIS_DIR

def print_transform():
    # Create the jinja2 environment.
    j2_env = Environment(loader=FileSystemLoader(THIS_DIR))
    print j2_env.get_template('network.jinja').render()

if __name__ == '__main__':
    print_transform()

mk-wl-ap-configs_v2.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

# and for debugging , I like my pprint functions
import pprint

# let's initialize my variables
csvFile = 'config-data.csv'
j2_env = {}
configs = {}
config = {}
data = {}
template = {}
item = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))

# open my jinja template config files (*.jinja)
configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as f:
    data = csv.DictReader(f, delimiter=',')

# render the templates to output files in their own subdirectory 
for config in configs:
    template = j2_env.get_template('%s.jinja' % config)
    for item in data:
        with open('%s-%s' % (config, item['AP_Name'])) as f:
            f.write(template.render(item))

mk-wl-ap-configs_v3.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

# and for debugging , I like my pprint functions
import pprint

# let's initialize my variables
csvFile = 'config-data.csv'
j2_env = {}
configs = {}
config = {}
data = {}
template = {}
item = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))

# open my jinja template config files (*.jinja)
configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as tf:
    data = csv.DictReader(tf, delimiter=',')

    # render the templates to output files in their own subdirectory 
    for config in configs:
        template = j2_env.get_template('%s.jinja' % config)
        for item in data:
            with open('%s-%s' % (config, item['AP_Name'])) as rf:
                rf.write(template.render(item))

mk-wl-ap-configs_v4.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

# and for debugging , I like my pprint functions
import pprint

# let's initialize my variables
csvFile = 'config-data.csv'
j2_env = {}
configs = {}
config = {}
data = {}
template = {}
item = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))

# open my jinja template config files (*.jinja)
configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as tf:
    data = csv.DictReader(tf, delimiter=',')

    # render the templates to output files in their own subdirectory 
    for config in configs:
        template = j2_env.get_template('%s.jinja' % config)
        for item in data:
            with open('%s/etc/config/%s' % (item['AP_Name'], config), 'w') as rf:
                rf.write(template.render(item))

mk-wl-ap-configs_v5.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

# and for debugging , I like my pprint functions
import pprint

# let's initialize my variables
csvFile = 'config-data.csv'
j2_env = {}
configs = {}
config = {}
data = {}
template = {}
item = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))

# open my jinja template config files (*.jinja)
configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as tf:
    data = list(csv.DictReader(tf, delimiter=','))

    # render the templates to output files in their own subdirectory 
    for config in configs:
        template = j2_env.get_template('%s.jinja' % config)
        for item in data:
            with open('%s/etc/config/%s' % (item['AP_Name'], config), 'w') as rf:
                rf.write(template.render(item))

mk-wl-ap-configs_v6.py

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

# and for debugging , I like my pprint functions
import pprint

# let's initialize my variables
csvFile = 'config-data.csv'
j2_env = {}
configs = {}
config = {}
data = {}
template = {}
item = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))

# open my jinja template config files (*.jinja)
configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as tf:
    data = list(csv.DictReader(tf, delimiter=','))

    # render the templates to output files in their own subdirectory 
    for config in configs:
        template = j2_env.get_template('%s.jinja' % config)
        for item in data:
            if config == "shadow":
                with open('%s/etc/%s' % (item['AP_Name'], config), 'w') as rf:
                    rf.write(template.render(item))            
            else:
                with open('%s/etc/config/%s' % (item['AP_Name'], config), 'w') as rf:
                    rf.write(template.render(item))

network.jinja

config globals 'globals'
	option ula_prefix '{{ AP_Name.ULA_PREFIX }}'

config interface 'lan'
	option force_link '1'
	option type 'bridge'
	option proto 'static'
	option ipaddr '{{ AP_Name.LAN_IPADDR }}'
	option netmask '{{ AP_Name.LAN_NETMASK }}'
	option gateway '{{ AP_Name.LAN_GATEWAY }}'
	option broadcast '{{ AP_Name.LAN_BROADCAST }}'
	option dns '{{ AP_Name.LAN_DNS }}'
	option _orig_ifname 'eth1 wlan0'
	option _orig_bridge 'true'
	option ifname 'eth0.1 eth0.128 eth1 eth1.1 eth1.128'

config interface 'wan'
	option _orig_ifname 'eth0'
	option _orig_bridge 'false'
	option proto 'static'
	option ipaddr '{{ AP_Name.WAN_IPADDR }}'
	option netmask '{{ AP_Name.WAN_NETMASK }}'
	option gateway '{{ AP_Name.WAN_GATEWAY }}'
	option broadcast '{{ AP_Name.WAN_BROADCAST }}'
	option dns '{{ AP_Name.WAN_DNS }}'
	option type 'bridge'
	option ifname 'eth0 eth0.1 eth0.96 eth1.1 eth1.96'

shadow.jinja

root:{{ enc_password }}:16491:0:99999:7:::
daemon:*:0:0:99999:7:::
ftp:*:0:0:99999:7:::
network:*:0:0:99999:7:::
nobody:*:0:0:99999:7:::

wireless.jinja

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option ssid '{{ AP_Name.AP_Prefix }}-office'
	option encryption 'psk2+ccmp'
	option hidden '1'
	option network 'lan'
	option key 'redacted out ;-)'

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option encryption 'psk2+ccmp'
	option ssid '{{ AP_Name.AP_Prefix }}-servers'
	option hidden '1'
	option network 'wan'
	option disabled '1'
	option key 'redacted out ;-)'

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option encryption 'psk2+ccmp'
	option ssid '{{ AP_Name.AP_Prefix }}-residents'
	option key 'redacted out ;-)'
	option network 'residents'

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option encryption 'psk2+ccmp'
	option ssid '{{ AP_Name.AP_Prefix }}-office-guests'
	option network 'officeguests'
	option key 'redacted out ;-)'

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option encryption 'psk2+ccmp'
	option key 'redacted out ;-)'
	option network 'accomguests'
	option ssid '{{ AP_Name.AP_Prefix }}-accom-guests'

How do I write my python jinja processor so I have my csv data in the right python dictionary (config-data read from my csv file), so I can iterate over the config -data dictionary and write out the jinja rendered resulting files to subdirs?

When I run my mk-wl-ap-configs.py code it tells me that AP_Name is not defined as it tries rendering the very first value in the first file.

jinja2.exceptions.UndefinedError: 'AP_Name' is undefined

full trace back

python mk-wl-ap-config.py 
Reading file config-data.csv

Here is the config-data dictionary 
<csv.DictReader instance at 0x103f3e710>
Traceback (most recent call last):
  File "mk-wl-ap-configs.py", line 34, in <module>
    print_transform()
  File "mk-wl-ap-config.py", line 31, in print_transform
    print j2_env.get_template('shadow.jinja').render()
  File "/Library/Python/2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Library/Python/2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "~/Documents/PycharmProjects/mk-wl-ap-configs/shadow.jinja", line 1, in top-level template code
    root:{{ AP_Name.ULA_PREFIX }}:16491:0:99999:7:::
  File "/Library/Python/2.7/site-packages/jinja2/environment.py", line 397, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'AP_Name' is undefined

@ondrag from Freenet IRC helped me further. He told me to simply place configdata in the render() function. I did, and

now I get:

Reading file config-data.csv

Traceback (most recent call last):
  File "~/Documents/PycharmProjects/mk-wl-ap-configs/mk-wl-ap-configs.py", line 34, in <module>
    print_transform()
  File "~/Documents/PycharmProjects/mk-wl-ap-configs/mk-wl-ap-configs.py", line 31, in print_transform
    print j2_env.get_template('shadow.jinja').render(configdata)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jinja2/environment.py", line 964, in render
    vars = dict(*args, **kwargs)
ValueError: dictionary update sequence element #0 has length 15; 2 is required

Thanks to @ondrag from FreeNet IRC I now have it all coming out to the console:
My (really his ;-)) new code looks like this:

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

csvFile = 'config-data.csv'
configdata = {}
configdatas = {}
shadowdatas = {}
shadowdata = {}
networkdatas = {}
networkdata = {}
wirelessdatas = {}
wirelessdata = {}
template = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))


print('Reading file %s' % csvFile)
shadowdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('shadow.jinja')
for shadowdata in shadowdatas:
  print template.render(shadowdata)


print('Reading file %s' % csvFile)
networkdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('network.jinja')
for networkdata in networkdatas:
  print template.render(networkdata)


print('Reading file %s' % csvFile)
wirelessdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('wireless.jinja')
for wirelessdata in wirelessdatas:
  print template.render(wirelessdata)

Well, in the end you probably want something like this: (not tested)

configs = ['shadow', 'network', 'wireless']
with open(csvFile, 'rU') as f:
    data = csv.DictReader(f, delimiter=',')

for config in configs:
    template = j2_env.get_template('%s.jinja' % config)
    for item in data:
        with open('%s-%s' % (config, item['AP_Name'])) as f:
            f.write(template.render(item))

Python can be very brief. :-)

OK, thanks to @ondrag on Freenet IRC, (maybe @garncarz ?) i now have code that runs "through" without errors.

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

csvFile = 'config-data.csv'
configdata = {}
configdatas = {}
shadowdatas = {}
shadowdata = {}
networkdatas = {}
networkdata = {}
wirelessdatas = {}
wirelessdata = {}
template = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))


print('Reading file %s' % csvFile)
shadowdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('shadow.jinja')
for shadowdata in shadowdatas:
  #print list(shadowdata)
  #print dict(shadowdata)
  #print dict(shadowdata)
  #print template.render(shadowdata)


print('Reading file %s' % csvFile)
networkdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('network.jinja')
for networkdata in networkdatas:
  #print list(networkdata)
  #print dict(networkdata)
  #print template.render(networkdata)


print('Reading file %s' % csvFile)
wirelessdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('wireless.jinja')
for wirelessdata in wirelessdatas:
  #print list(wirelessdata)
  #print dict(wirelessdata)
  #print template.render(wirelessdata)

But now I need to split the whole kabuddle up and write the three jinja rendered output files into 3 separate outputfiles under twelve separate subdirs (one per WL AP)

OK, now I got a pretty good pretty printer view into my loops. But I can't for the life of me, catch the value of the first key in the current loop of the dictionaory being processed?

#!/usr/bin/env/python
# 
# import os to get fundamentals
import os

#
# First things first, we need to import the csv module
import csv

import pprint

csvFile = 'config-data.csv'
shadowdatas = {}
shadowdata = {}
networkdatas = {}
networkdata = {}
wirelessdatas = {}
wirelessdata = {}
template = {}

from jinja2 import Environment, FileSystemLoader

# Capture our current directory
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR))


print('Reading file %s' % csvFile)
shadowdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('shadow.jinja')
for shadowdata in shadowdatas:
  print "\npprint.pprint(dict(shadowdata))"
  pprint.pprint(dict(shadowdata))
  pprint(shadowdata['AP_Name'])
  print template.render(shadowdata)


print('Reading file %s' % csvFile)
networkdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('network.jinja')
for networkdata in networkdatas:
  print "\npprint.pprint(dict(networkdata))"
  pprint.pprint(dict(networkdata))
  pprint(networkdata['AP_Name'])
  print template.render(networkdata)


print('Reading file %s' % csvFile)
wirelessdatas=csv.DictReader(open(csvFile, 'rU'), delimiter=',')
template = j2_env.get_template('wireless.jinja')
for wirelessdata in wirelessdatas:
  print "\npprint.pprint(dict(wirelessdata))"
  pprint.pprint(dict(wirelessdata))  
  pprint(wirelessdata['AP_Name'])
  print template.render(wirelessdata)

so my pprint lines 34 44 54 all fail the same way. but how can I get the value of 'Ap_Name' of the current line in the current dictionary loop into a variabel? So I can then create the right subdirectory for placing the rendered jinja output into.

@garncarz man I am SLOW. You did it! And I didn't even notice the work you put in. Thank you so much! I can learn and glean so much from this. This is soo different from doing it in perl.
But I a getting too tired now. but it now only complains about an I/O operation happening on a closed file. Hope that a strong coffee in the morning will help me see what needs tweaking.

mk-wl-ap-configsv2.py 
Traceback (most recent call last):
  File "mk-wl-ap-configsv2.py", line 33, in <module>
    for item in data:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 107, in next
    self.fieldnames
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 90, in fieldnames
    self._fieldnames = self.reader.next()
ValueError: I/O operation on closed file

OK, now I have a new python file called "mk-wl-ap-configs_v2.py" (thanks @garncarz)
but when I run it, it gives me an "ValueError: I/O operation on closed file" error?

mk-wl-ap-configs_v2.py 
Traceback (most recent call last):
  File "mk-wl-ap-configsv2.py", line 36, in <module>
    for item in data:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 107, in next
    self.fieldnames
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 90, in fieldnames
    self._fieldnames = self.reader.next()
ValueError: I/O operation on closed file

This is getting so close now. It is now writing the output into the desired subdir structure, And it is now perfectly rendering the 1st out of 3 templates. No idea why it won't do the 2nd and 3rd one?

the current code in use is "mk-wl-ap-configs_v4.py"

Why does it only do the 1st loop on the 'shadow' and not the one on 'network' and 'wireless' templates as well?

@zahlman from Freenet IRC solved that one for me. ;-) all it took was putting the csv.DictReader output into a list, (see "mk-wl-ap-configs_v5.py" above) so I can iterate over it again and again. The way it was the dictionary was consumed after first run through anbd the 'network' and 'wireless' renderer had no config-data to render with. So it did actually run through three times, jsut didn;t render anything after first run

And most likely final adjustment now.

Latest and greatest is: "mk-wl-ap-configs_v6.py"

It now puts the jinja2 rendered 'shadow' output file into the place openwrt expects and needs it.

The internet is a great place.
I would have never managed to finish this over a weekend without the guys and gals on Freenet IRC #python

Thanks a lot.