Last active
December 2, 2021 03:56
-
-
Save anjijava16/c41651e5efcdaa1a0a881a24908bc42d to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
conda info | |
conda update -n base -c defaults conda | |
conda create --name data_ingestion python=3.6 | |
(OR) | |
conda create --name data_ingestion | |
conda activate data_ingestion | |
conda list | |
conda info --envs | |
conda install teradatasql | |
conda install -c pending-opensource conda-pack | |
conda pack -n <conda_env_name> -o <conda_env_name>.tar.gz | |
#Create VirtualENV | |
virtualenv -p python3 data_gcp | |
anjaiahsprcloud2@cloudshell:~ (welcome-bigquery-load)$ virtualenv -p python3 data_gcp | |
created virtual environment CPython3.7.3.final.0-64 in 162ms | |
creator CPython3Posix(dest=/home/anjaiahsprcloud2/data_gcp, clear=False, no_vcs_ignore=False, global=False) | |
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/anjaiahsprcloud2/.local/share/virtualenv) | |
added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2 | |
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator | |
virtualenv -p python3 data_gcp | |
#Activate | |
source data_gcp/bin/activate | |
# Install | |
pip install --upgrade google-cloud-pubsub | |
pip install --proxy http://username:[email protected]:8080 google-cloud-bigquery | |
Conda Pack | |
Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer. | |
To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI. | |
conda-forge: | |
<span>conda install </span><span>-</span><span>c conda</span><span>-</span><span>forge conda</span><span>-</span><span>pack</span> | |
PyPI: | |
<span>pip install conda</span><span>-</span><span>pack</span> | |
To package an environment: | |
# Pack environment my_env into my_env.tar.gz | |
$ conda pack -n my_env | |
# Pack environment my_env into out_name.tar.gz | |
$ conda pack -n my_env -o out_name.tar.gz | |
# Pack environment located at an explicit path into my_env.tar.gz | |
$ conda pack -p /explicit/path/to/my_env | |
To install the environment: | |
# Unpack environment into directory `my_env` | |
$ mkdir -p my_env | |
$ tar -xzf my_env.tar.gz -C my_env | |
# Use Python without activating or fixing the prefixes. Most Python | |
# libraries will work fine, but things that require prefix cleanups | |
# will fail. | |
$ ./my_env/bin/python | |
# Activate the environment. This adds `my_env/bin` to your path | |
$ source my_env/bin/activate | |
# Run Python from in the environment | |
(my_env) $ python | |
# Cleanup prefixes from in the active environment. | |
# Note that this command can also be run without activating the environment | |
# as long as some version of Python is already installed on the machine. | |
(my_env) $ conda-unpack | |
https://www.alexvolok.com/2021/data-engineering-development-environment-using-wsl-and-miniconda/ | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment