Thoughts on Package Management for CircuitPython

CircuitPython_Packages.MD

Package Management for CircuitPython

Nothing formal written up on this yet, but I think we're seeing the download the zip file approach start to become brittle. Already there are two different bundles. My initial thought is why not have a cpip (or modify circup) and use PyPi. One thing I like about this is that it teaches new programmers things they need to learn if they also do Python programming. I've not done any real research into the technical parts of this yet, but in an ideal world, a fork of pip or some modular components of pip could be used to drive the core logic of version checking and install and the CP specific parts that deal with the board would be the additional functionality.

Interesting comments in pip/_internal/cli/main.py

# Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.

# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.

# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.

# The following example shows how to use runpy to invoke pip in that
# case:
#
#     sys.argv = ["pip", your, args, here]
#     runpy.run_module("pip", run_name="__main__")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.

It seems that you can also use pip for installing into CP-s lib directory: https://stackoverflow.com/a/2916320/261181

Thanks for that @aivarannamaa! The next answer down mentions the --target option for pip, which I also was not aware of. It works. Unfortunately, all the PyPi Adafruit_CircuitPyton_* packages are tied to the Blinka\CPython experience and they don't install the MPY versions. So still more exploration to do on how to get the MPY and CPython versions both working.

Example using --target option from pip

pip install  --target /Volumes/CIRCUITPY/lib/ adafruit-circuitpython-pm25
Collecting adafruit-circuitpython-pm25
  Downloading adafruit-circuitpython-pm25-2.1.0.tar.gz (20 kB)
Collecting Adafruit-Blinka
  Downloading Adafruit-Blinka-5.10.0.tar.gz (113 kB)
     |████████████████████████████████| 113 kB 483 kB/s
Collecting Adafruit-PlatformDetect>=2.18.1
  Downloading Adafruit-PlatformDetect-2.25.0.tar.gz (29 kB)
Collecting Adafruit-PureIO>=1.1.7
  Using cached Adafruit_PureIO-1.1.8.tar.gz (26 kB)
Collecting pyftdi>=0.40.0
  Downloading pyftdi-0.52.9-py3-none-any.whl (139 kB)
     |████████████████████████████████| 139 kB 2.0 MB/s
Collecting pyserial>=3.0
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting pyusb>=1.0.0
  Using cached pyusb-1.1.0.tar.gz (78 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting adafruit-circuitpython-busdevice
  Downloading adafruit-circuitpython-busdevice-5.0.3.tar.gz (22 kB)
Building wheels for collected packages: adafruit-circuitpython-pm25, Adafruit-Blinka, Adafruit-PlatformDetect, Adafruit-PureIO, pyusb, adafruit-circuitpython-busdevice
  Building wheel for adafruit-circuitpython-pm25 (setup.py) ... done
  Created wheel for adafruit-circuitpython-pm25: filename=adafruit_circuitpython_pm25-2.1.0-py3-none-any.whl size=8052 sha256=ce968dd0d9c014981f38934f6b2a9892e3935678fc5632730ac2eb86797172df
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/a3/e6/8a/57fd0507abba996196d5ccca4041383c779cfb148127c03d16
  Building wheel for Adafruit-Blinka (setup.py) ... done
  Created wheel for Adafruit-Blinka: filename=Adafruit_Blinka-5.10.0-py3-none-any.whl size=138835 sha256=1fa20cbb1abe454002f7333793a03ec2d24052936477e2860edd5d21bf0c0c74
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/62/b4/f9/7788d5c33d281ecdcfca4dfb420fd8be4b3330ad06c03e7468
  Building wheel for Adafruit-PlatformDetect (setup.py) ... done
  Created wheel for Adafruit-PlatformDetect: filename=Adafruit_PlatformDetect-2.25.0-py3-none-any.whl size=17243 sha256=3f241bed2232b18660f84e4255b641f128e56df51bb902a757a4d111fd091f85
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/ff/de/ee/2e893213c282533ede1fa64ad1b050df2759fb2c5738eca0d3
  Building wheel for Adafruit-PureIO (setup.py) ... done
  Created wheel for Adafruit-PureIO: filename=Adafruit_PureIO-1.1.8-py3-none-any.whl size=11566 sha256=10ea75a84102aa8e2c6f69fa87cd29de5824e42c88c75e53c10260ab81bb4d9b
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/da/53/11/945fe5fb948f05b5edd81ce5f0d0804decfa5cb58ecf2a25ba
  Building wheel for pyusb (PEP 517) ... done
  Created wheel for pyusb: filename=pyusb-1.1.0-py3-none-any.whl size=58025 sha256=a438cbeec406e5425c0b8cfd352cec7dd3c773bdc428ada3229210d2a321d20c
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/cc/7a/db/65c8ac2c10a8717256d548a1c7222b3028408f3f9be3224809
  Building wheel for adafruit-circuitpython-busdevice (setup.py) ... done
  Created wheel for adafruit-circuitpython-busdevice: filename=adafruit_circuitpython_busdevice-5.0.3-py3-none-any.whl size=7707 sha256=290dd3a30ad782d6f58b5aa66cfd5a851dcfc295bb1935b901f2c71713f6b705
  Stored in directory: /Users/askpatrickw/Library/Caches/pip/wheels/a9/c0/da/087d3e3021414f7a5ed3e639368614aff875a533dee617cc15
Successfully built adafruit-circuitpython-pm25 Adafruit-Blinka Adafruit-PlatformDetect Adafruit-PureIO pyusb adafruit-circuitpython-busdevice
Installing collected packages: pyusb, pyserial, pyftdi, Adafruit-PureIO, Adafruit-PlatformDetect, Adafruit-Blinka, adafruit-circuitpython-busdevice, adafruit-circuitpython-pm25
Successfully installed Adafruit-Blinka-5.10.0 Adafruit-PlatformDetect-2.25.0 Adafruit-PureIO-1.1.8 adafruit-circuitpython-busdevice-5.0.3 adafruit-circuitpython-pm25-2.1.0 pyftdi-0.52.9 pyserial-3.5 pyusb-1.1.0
WARNING: Target directory /Volumes/CIRCUITPY/lib/adafruit_bus_device already exists. Specify --upgrade to force replacement.

I should add... its not that the .PY versions of the packages won't work, its that they're not compressed which helps with storage space AND pip installs the CPython dependencies not needed on the CircuitPython Board.

You can also use --path on pip list and pip freeze. It does not understand the .mpy packages of course. There is also a --platform option which I don't understand yet.

Note: You can see in these --path examples the "extra" CPython packages which were installed on my FeatherS2.

pip list --path /Volumes/CIRCUITPY/lib
Package                          Version
-------------------------------- -------
Adafruit-Blinka                  5.10.0
adafruit-circuitpython-busdevice 5.0.3
adafruit-circuitpython-pm25      2.1.0
Adafruit-PlatformDetect          2.25.0
Adafruit-PureIO                  1.1.8
pyftdi                           0.52.9
pyserial                         3.5
pyusb                            1.1.0

pip freeze --path /Volumes/CIRCUITPY/lib
Adafruit-Blinka==5.10.0
adafruit-circuitpython-busdevice==5.0.3
adafruit-circuitpython-pm25==2.1.0
Adafruit-PlatformDetect==2.25.0
Adafruit-PureIO==1.1.8
pyftdi==0.52.9
pyserial==3.5
pyusb==1.1.0

Wait, you are using pip to install CP library on a CP board mounted on the Linux host?
That is ... impressive? Mind blowing?

For the *.py vs *.mpy then maybe it could be downloaded to a temporary location and then mpy-cross to have the mpy and then only copy to the device.

Not sure what kind of wrapper that would require, but it might be a good trick.

@dglaude, I'm not, but it looks like you could. ;-)

That's a great idea as an interim step.
I've not used MPY-Cross. When I look at it, it sounds like it's target OS specific. Am I misunderstanding that?

I notice this issue in CP Build Tools... maybe the mpy-cross referenced there could be used.
adafruit/circuitpython-build-tools#55

I think that while MPY-Cross exist on multiple platform, it always generate the same output MPY file for the same input PY file.
It is "just" cross compiling, or doing the parsing from Python file to file that can be executed in the CircuitPython virtual machine.

Yes the mpy-cross for Windows/Mac/??? are build and available from there, so there is no need to compile MPY-Cross yourself.

Could pip --implementation have mpy added?

--implementation
Only use wheels compatible with Python implementation , e.g. 'pp', 'jy', 'cp', or
'ip'. If not specified, then the current interpreter implementation is used. Use 'py' to force
implementation-agnostic wheels.

Or a CP PyPi...

Package Index Options:
-i, --index-url Base URL of the Python Package Index (default https://pypi.org/simple). This should point to a
repository compliant with PEP 503 (the simple repository API) or a local directory laid out in the
same format.
--extra-index-url Extra URLs of package indexes to use in addition to --index-url. Should follow the same rules as
--index-url.
--no-index Ignore package index (only looking at --find-links URLs instead).
-f, --find-links If a URL or path to an html file, then parse for links to archives such as sdist (.tar.gz) or wheel
(.whl) files. If a local path or file:// URL that's a directory, then look for archives in the
directory listing. Links to VCS project URLs are not supported.

@askpatrickw, Here is my investigation concerning --implementation: aivarannamaa/minipip#6

You may also find aivarannamaa/minipip#5 and https://github.com/aivarannamaa/packaging_experiment interesting

Related https://www.python.org/dev/peps/pep-0665/#non-goals

I haven't seen mention of it, so I'll offer that Mu Editor support for whatever the package management ends up looking like would be helpful.