scram-venv

#!/bin/bash -e

# check for CMSSW environment
if [ -z "$CMSSW_BASE" ]; then
	echo '$CMSSW_BASE not set'
	exit 1
fi

# check for python3 in actual CMSSW area
if ! (cd $CMSSW_BASE && scram tool info python3 >& /dev/null); then
	echo "$CMSSW_VERSION does not provide python3"
	exit 1
fi

# get info from existing python
PYPATH=$(which python3)
PYHOME=$(dirname $PYPATH)
PYVERSION=$(python3 --version | cut -d' ' -f2)
PYVERSHORT=$(echo $PYVERSION | cut -d'.' -f1-2)
PYVERSHORTER=$(echo $PYVERSION | cut -d'.' -f1)
PYNAME=python$PYVERSHORT

# create venv directories
VENVDIR=$CMSSW_BASE/venv/$SCRAM_ARCH
DIRS=(
$VENVDIR \
$VENVDIR/bin \
$VENVDIR/include \
$VENVDIR/lib/$PYNAME/site-packages \
)
for DIR in ${DIRS[@]}; do
	mkdir -p $DIR
done
(cd $VENVDIR; ln -s lib lib64)

# create venv config
cat << EOF > $VENVDIR/pyvenv.cfg
home = $PYHOME
include-system-site-packages = true
version = $PYVERSION
EOF

# link venv executables
EXES=(
python$PYVERSHORTER \
python$PYVERSHORT \
)
for EXE in ${EXES[@]}; do
	ln -s $PYPATH $VENVDIR/bin/$EXE
done

# set up scram hook infrastructure if missing
HOOKDIR=$CMSSW_BASE/config/SCRAM/hooks/runtime
HOOKFILE=${HOOKDIR}-hook
if [ ! -f "$HOOKFILE" ]; then
	mkdir -p $HOOKDIR
	cat << 'EOF' > $HOOKFILE
#!/bin/bash
SCRIPT_DIR=$(dirname $0)
if [ -e ${SCRIPT_DIR}/runtime ] ; then
  for tool in $(find ${SCRIPT_DIR}/runtime -type f | sort) ; do
    [ -x $tool ] && $tool
  done
fi
EOF
	chmod +x $HOOKFILE
fi

# install hook for venv
HOOK=${HOOKDIR}/py3-venv
cat << 'EOF' > $HOOK
#!/bin/bash
echo "RUNTIME:path:prepend:PATH=${LOCALTOP}/venv/$SCRAM_ARCH/bin"
EOF
chmod +x $HOOK

# instructions to activate venv hook and update environment
echo "scram-venv succeeded! please call 'cmsenv' now"

@kpedro88 , this looks fine. Can you please do the following modificaations

• please do check if $CMSSW_BASE is set otherwise fail early
• one can set multiple archs in dev area, e.g. el8_amd64_gcc11, el8_amd64_gcc12. Or on shared FS e.g AFS, one can have el8 and el9 for x86_64 and non-x86_64 in same dev area. So how about using VENVDIR=$CMSSW_BASE/venv/$SCRAM_ARCH and making proper changes in config/SCRAM/hooks/runtime/py3-venv too?
• Note that after running this once, the python command changed to python3 which I think is not right. I would suggest to only create symlinks for python$PYVERSHORTER and python$PYVERSHORT
• do we need CMSSW_BASE=${LOCALTOP} in py3-venv ? I think echo "RUNTIME:path:prepend:PATH=${LOCALTOP}/venv/${SCRAM_ARCH}/bin" should be enough.

this look generic enough to go in to cms-common which will allow it be to used by any cmssw release. Can you please open a PR for https://github.com/cms-sw/cms-common

@smuzaffar thanks for these comments, I've updated the gist accordingly.

I was actually wondering if we could build this into scram project, so the venv dir and associated setup are automatically provided whenever anyone checks out CMSSW? scram-venv could be provided for compatibility to upgrade existing areas that were created before this change. Let me know what you think of this idea and then I'll proceed w/ PR(s).

updated gist again with some more error checking and user-friendlier messages. works as far back as 10_2_X now. 10_1_X kind of works, but pip3 (once installed) gives /usr/bin/python3: symbol lookup error: /usr/bin/python3: undefined symbol: _Py_LegacyLocaleDetected, not clear why.

@kpedro88 , doing it at scram project time means every one gets an extra

• bin directory in PATH
• lib directory in LD_LIBRARY_PATH
• py directory in PYTHONPATH

which though is not bad but should be avoid for all the grid jobs. I would prefer that a user run it exeplicitly if he/she needs to install any package.

By the way, why it needs symlinks for python3?

@smuzaffar the symlink is needed for the venv to be recognized: PEP 405:

If a pyvenv.cfg file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form key = value. If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.

The home area needs to be writable, so we need to link to the cvmfs (read-only) Python executable in the local working area.

Only PATH that changes here; LD_LIBRARY_PATH and PYTHON3PATH do not need to be updated for this to work. So it's actually less overhead than adding a new external package...

However, after thinking about it more, I agree that it's a good idea to start as an optional feature in cms-common so that users can test it and find any edge cases (and we avoid any interference with production in the meantime). This also gives CRAB time to update, etc. Maybe in the future it can be propagated to scram project.

PR opened at cms-sw/cms-common#6

@kpedro88 @smuzaffar This is great! Would be good to have some documentation on how to use it, e.g. somewhere at https://cms-sw.github.io/, now that it's merged.

agree @clelange . @kpedro88 can you please open a PR for https://github.com/cms-sw/cms-sw.github.io/tree/code ?

Will do - after my CHEP plenary today...