matthewfeickert

FROM mambaorg/micromamba:1.5.9-bookworm-slim

USER mambauser

SHELL [ "/bin/bash", "-c" ]

# Use C.UTF-8 locale to avoid issues with unicode encoding
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8

matthewfeickert

# build products
output/
example
# pixi environments
.pixi
*.egg-info

matthewfeickert

Minimal PyTorch CUDA environment with pixi

Based off of the pixi docs example

Steps

0. Install pixi
   • curl -fsSL https://pixi.sh/install.sh | bash
1. pixi run detect-gpu

• If this is your first time doing anything in the environment pixi will run pixi install for you first.

matthewfeickert

Use VS Code to run notebooks on remote servers in environments built on top of LCG views

(Note that this whole thing is a bit cursed as LCG views are a terrible way to distribute Python environments)

Setup

Get a working VS Code server on the remote

Requires:

matthewfeickert

Use GMail's SMTP servers to use CERN email account as an alias

1. Enable [forwarding of emails from Outlook][outlook-forwarding] to GMail

   1. In Outlook select Settings > Mail > Forwarding.
   2. Select "Enable forwarding".
   3. Enter your GMail address in the box under "Forward my email to:".
   4. (Optional, but recommended) Select "Keep a copy of forwarded messages"

[![outlook-view][outlook-view-gist]][outlook-forwarding]

matthewfeickert

Unexpected crashing of VS Code when opening Python files

Today (2023-04-03) VS Code is crashing whenever a Python file is opened in the mixed language repository https://gitlab.cern.ch/gstark/pycolumnarprototype

This behavior has not been observed until today. The machine uptime has been minimal.

$ uptime
 17:57:25 up 14:11,  1 user,  load average: 2.04, 2.15, 2.04

matthewfeickert

Bootstrap example

Bootstrap example to get around bugs with panda and rucio on CVMFS and PYTHONPATH pollution as described in the March 2023 PATHelp email thread "[ROOT undefined symbol error when setting up ROOT and Rucio together after setupATLAS][PATHelp-thread]".

Example

The PYTHONPATH is polluted with Python 3.6, so make an environment with the last pip and pandas releases that were compatible with Python 3.6.

The boostrap.sh Bash script sets up a Python virtual environment that has a version of pip that won't break with Python 3.6 which is further used in example.sh.

matthewfeickert

Failure of not using a virtual environment in installPip

By not using a virtual environment installPip makes it unable to interact with the virtual environment manually as one can not reliably manually install or uninstall packages.

Example: Unable to uninstall numpy

$ setupATLAS
...
$ asetup AnalysisBase,22.2.110

matthewfeickert

20223-03-15 technical response to JIRA ticket ATLAS ASGATLASG-2477: python packages for atlasexternals

As a warning, this is going to be a long response as it is going to need to discuss the technical details of how the way LCG views and the current way ATLAS externals works is fundamentally at odds with the core concepts of Python virtual environments. To make this more digestible in a JIRA ticket thread and to avoid repeating myself (somewhat) I am going to assume that anyone who is reading this will read the README I've already written for cvmfs-venv which explains this at a high level. Also, as I find JIRA's lack of Markdown to make reading next to anything on it pretty terrible I've put the full contents of the technical parts that follow in this GitHub Gist in case you want to read it

matthewfeickert

Creating a Jupyter kernel for Python virtual environment on UChicago Analysis Facility

• Reproduction of core content of ATLAS Talk Topic: Preferred way of creating a Jupyter kernel for Python virtual environment (ATLAS Internal, sorry)
• Corresponding YouTube explainer video: Creating an IPython kernel for your venv on UChicago AF Jupyter Lab (no captions, sorry)

For robustness and reproduciblity, I would like to use a Python virtual environment when working with ml_platform:latest containers (easy enough) but I would also like to have IPython kernels that I make in that virtual environment be recognized by Jupyter Lab's launcher. I believe that the easiest way to do this is to just use the --user flag when creating an IPython kernel, but I wanted to know if there is an alternative / create this as an example Issue for others to reference.

Here's an exam