ebtables issue in CORE

ebtables_issue.txt

Problem: at ~300 nodes (many multi-homed) and ~1200 interconnects, CORE began generating ebtables issues which appeard in the CORE logs.

Solution (unresolved):

added --concurrent option to ebtables call string in:
/usr/local/lib/python2.7/dist-packages/core/netns/vnet.py

saved original vnet.py
deleted vnet.pyc



core log:
##added --concurrent to both ( --atomic-file    and --atomic-save)

Unknown argument: '--concurrent --atomic-file'.
exception: b.457.48 (457) ebtables command error: atomic-save (['/sbin/ebtables', '--concurrent --atomic-file', '/tmp/pycore.ebtables.atomic', '--concurrent --atomic-save'])
Command '['/sbin/ebtables', '--concurrent --atomic-file', '/tmp/pycore.ebtables.atomic', '--concurrent --atomic-save']' returned non-zero exit status 255


##added --concurrent only to ( --atomic-save)

Unknown argument: '--concurrent --atomic-save'.
exception: b.1600.142 (1600) ebtables command error: atomic-save (['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--concurrent --atomic-save'])
Command '['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--concurrent --atomic-save']' returned non-zero exit status 255

##added --concurrent only BEHIND ( --atomic-save)

Unknown argument: '--atomic-save --concurrent'.
exception: b.1301.27 (1301) ebtables command error: atomic-save (['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-save --concurrent'])
Command '['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-save --concurrent']' returned non-zero exit status 255


##removed all --concurent instances, few errors:

Unable to update the kernel. Two possible causes:
1. Multiple ebtables programs were executing simultaneously. The ebtables
   userspace tool doesn't by default support multiple ebtables programs running
   concurrently. The ebtables option --concurrent or a tool like flock can be
   used to support concurrent scripts that update the ebtables kernel tables.
2. The kernel doesn't support a certain ebtables extension, consider
   recompiling your kernel or insmod the extension.
.
exception: b.1301.119 (1301) ebtables command error: atomic-commit (['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-commit'])
Command '['/sbin/ebtables', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-commit']' returned non-zero exit status 255


## added '--concurrent, as separate parameter

Unable to update the kernel. Two possible causes:
1. Multiple ebtables programs were executing simultaneously. The ebtables
   userspace tool doesn't by default support multiple ebtables programs running
   concurrently. The ebtables option --concurrent or a tool like flock can be
   used to support concurrent scripts that update the ebtables kernel tables.
2. The kernel doesn't support a certain ebtables extension, consider
   recompiling your kernel or insmod the extension.
.
exception: b.345.30 (345) ebtables command error: atomic-commit (['/sbin/ebtables', '--concurrent', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-commit'])
Command '['/sbin/ebtables', '--concurrent', '--atomic-file', '/tmp/pycore.ebtables.atomic', '--atomic-commit']' returned non-zero exit status 255