Skip to content

Instantly share code, notes, and snippets.

@anjijava16
Last active December 2, 2021 03:56
Show Gist options
  • Save anjijava16/c41651e5efcdaa1a0a881a24908bc42d to your computer and use it in GitHub Desktop.
Save anjijava16/c41651e5efcdaa1a0a881a24908bc42d to your computer and use it in GitHub Desktop.
conda info
conda update -n base -c defaults conda
conda create --name data_ingestion python=3.6
(OR)
conda create --name data_ingestion
conda activate data_ingestion
conda list
conda info --envs
conda install teradatasql
conda install -c pending-opensource conda-pack
conda pack -n <conda_env_name> -o <conda_env_name>.tar.gz
#Create VirtualENV
virtualenv -p python3 data_gcp
anjaiahsprcloud2@cloudshell:~ (welcome-bigquery-load)$ virtualenv -p python3 data_gcp
created virtual environment CPython3.7.3.final.0-64 in 162ms
creator CPython3Posix(dest=/home/anjaiahsprcloud2/data_gcp, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/anjaiahsprcloud2/.local/share/virtualenv)
added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
virtualenv -p python3 data_gcp
#Activate
source data_gcp/bin/activate
# Install
pip install --upgrade google-cloud-pubsub
pip install --proxy http://username:[email protected]:8080 google-cloud-bigquery
Conda Pack
Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer.
To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI.
conda-forge:
<span>conda install </span><span>-</span><span>c conda</span><span>-</span><span>forge conda</span><span>-</span><span>pack</span>
PyPI:
<span>pip install conda</span><span>-</span><span>pack</span>
To package an environment:
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
To install the environment:
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run Python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack
https://www.alexvolok.com/2021/data-engineering-development-environment-using-wsl-and-miniconda/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment