hamelsmu

from metaflow import step, current, FlowSpec, Parameter, card
from mymodel import train_model

class NBFlow(FlowSpec):
    "A toy example of using the notebook card."

    @step
    def start(self):
        # Train a model, save the results in `model_results`
        self.model_results = train_model(...)

hamelsmu

Feedback on @pytorch_parallel

Related to [metaflow #907](Netflix/metaflow#907 and the related docs draft

1. Developer Ergonimics

To use the feature, the user must learn a brand new way of doing foreach. This adds a high degree of congntive load, as the user must remember that for this particular use case and this use case only, they need to use self.next(..., num_parallel=...).

The api makes it very unclear where the parallelization is happening. For example, pytorch_lightning.trainer has an argument gpus=-1, which means that it will use all available gpus. In this case, what does num_parallel add to this? The user has lots of cognitive overload to have to reason about where and what kind of parallelization is happening.

hamelsmu

import os
from azure.storage.blob import BlobServiceClient

blob_service_client = BlobServiceClient.from_connection_string(os.getenv('AZURE_STORAGE_CONNECTION_STRING'))

with open('gh_repo_topics.parquet', 'rb') as data:
    blob_client = container_client.upload_blob(name="sample_data/gh_repo_topics.parquet", data=data, overwrite=True)

####### Downloading files ##########
f = container_client.download_blob("sample_data/gh_repo_topics.parquet")

hamelsmu

hamelsmu

hamelsmu

export IMAGE_FAMILY="pytorch-latest-gpu"
export ZONE="us-west1-b"
export INSTANCE_NAME="dl-rig"

gcloud compute instances create $INSTANCE_NAME \
  --zone=$ZONE \
  --image-family=$IMAGE_FAMILY \
  --image-project=deeplearning-platform-release \
  --maintenance-policy=TERMINATE \
  --accelerator="type=nvidia-tesla-v100,count=2" \

hamelsmu

hamelsmu

hamelsmu

hamelsmu

#!/usr/bin/env python
# coding: utf-8

from fastcore.all import *
from shutil import rmtree
import yaml

rm=['_includes','_layouts','css','fonts','js','licenses','tooltips.json','Gemfile','Gemfile.lock']
raw='https://raw.githubusercontent.com/fastai/nbdev_template/master/docs/'