https://bit.ly/jupyter-ai-ollama
This project combines Jupyter AI with Ollama to create a local, containerized environment for AI-assisted data analysis and development. It uses Docker Compose to manage two services:
- An Ollama service for running large language models locally
- A Jupyter service with Jupyter AI integration
- Docker and Docker Compose
- Sufficient disk space for language models (varies by model)
- Optional: NVIDIA container toolkit (for GPU support)
- Clone this repository
- Configure your model in
.env
(default is qwen2.5-coder:1.5b) - Start the services:
docker-compose up -d
- Pull the language model manually (recommended):
docker-compose exec -it ollama bash
ollama pull $LANGUAGE_MODEL # as specified in .env
While the code includes automatic model pulling, manual pulling is recommended to monitor the download progress and handle any issues.
- Access Jupyter Lab at http://localhost:8888 (token: 123)
Edit .env
to configure:
LANGUAGE_MODEL
: Choose your Ollama modelOLLAMA_HOST
: Internal service URL (default: http://ollama:11434)
These are some recommended models to get started:
- qwen2.5-coder:1.5b (default)
- llama2
- codellama
- smollm2:135m
Browse the Ollama Model Library for many more available models. You can use any model from the library by updating the LANGUAGE_MODEL
in your .env
file.
.
├── .dockerignore # Excludes unnecessary files from builds
├── .env # Environment configuration
├── Dockerfile # Jupyter service configuration
├── Dockerfile.ollama # Ollama service configuration
├── config.json # Jupyter AI configuration
├── docker-compose.yml # Service orchestration
├── entrypoint.sh # Ollama initialization
└── jupyter-start.sh # Jupyter initialization
This project runs well on CPU-only systems. However, if you have a compatible NVIDIA GPU, you can enable GPU acceleration:
- Install the NVIDIA Container Toolkit:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
- Add GPU configuration to docker-compose.yml under the ollama service:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
Resource requirements vary by model:
- Small models (smollm2:135m): 2-4GB RAM
- Medium models (qwen2.5-coder:1.5b): 4-8GB RAM
- Large models (llama2): 8GB+ RAM
If the model fails to load:
- Stop the services:
docker-compose down
- Remove the
ollama_data
volume - Restart and manually pull the model as described above
For more issues, check container logs:
docker-compose logs -f jupyter
docker-compose logs -f ollama
-
Automated Model Download: The project suggests manually pulling the language model, though automatic model pulling is included in the code. Improving the reliability and transparency of the automatic pulling process would enhance the user experience. Consider adding progress bars or more detailed logging during the download process. Error handling could also be improved to automatically retry downloads or suggest alternative models if the specified model is unavailable.
-
Simplified GPU Configuration: While GPU support is mentioned, enabling it requires manual configuration of the
docker-compose.yml
file. Providing a simpler way to enable GPU support (e.g., through an environment variable or a separatedocker-compose.gpu.yml
file) would make it more accessible to users with NVIDIA GPUs. -
Health Checks: Implement Docker health checks for both the Ollama and Jupyter services. This would allow Docker Compose to automatically restart services if they become unhealthy, improving the overall robustness of the environment.
-
Dynamic Resource Allocation: Explore options for dynamic resource allocation based on the selected language model. This could involve adjusting the memory and CPU resources allocated to the Ollama service based on the model's requirements, optimizing resource utilization.
-
GUI for Model Selection: Consider adding a simple GUI or CLI tool to select and manage language models. This would make it easier for users to discover and switch between different models without manually editing the
.env
file. -
Security Considerations: While the project uses a default token for JupyterLab, it's essential to emphasize the importance of changing this token for security reasons. Consider adding a warning message or instructions on how to generate a secure token during the setup process.
-
Expand Documentation: Add more detailed documentation on how to use Jupyter AI with Ollama, including examples of common use cases and troubleshooting tips. This would help users get started quickly and effectively.
-
Pre-configured notebooks: Include sample Jupyter notebooks that demonstrate how to use the integrated environment for common tasks, like data analysis, code generation, and model exploration. This could serve as a quick start guide for new users.
-
Model management: Provide clear instructions on how to manage different language models, including how to remove models that are no longer needed and how to update models to the latest versions.
-
Multi-platform support: Expand the documentation to include specific instructions for setting up the environment on different operating systems (e.g., Windows, macOS, Linux).