Skip to content

Instantly share code, notes, and snippets.

@williballenthin

williballenthin/setup-pachyderm.ipynb

Last active November 30, 2017 19:57
Show Gist options
  • Save williballenthin/8bbb7710ca8faa93d6eb57a44eda785b to your computer and use it in GitHub Desktop.
Save williballenthin/8bbb7710ca8faa93d6eb57a44eda785b to your computer and use it in GitHub Desktop.
Download ZIP
local deployment of pachyderm.io using minikube on a win10 system running hyper-v
Display the source blob
Display the rendered blob
Raw
setup-pachyderm.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# pachyderm.io\n",
"\n",
"our goal is to setup a local deployment of pachyderm.io using minikube on a win10 system running hyper-v.\n",
"\n",
"we're running win10 because that's what my host is. virtualization must be via hyper-v, since i also use docker for windows. probably can't run minikube in a linux hyper-v vm thats running kvm due to vm nesting (see: https://github.com/kubernetes/minikube/issues/1507). so, we'll figure out minikube + hyper-v.\n",
"\n",
"pachctl is only provided for linux/osx (windows support discussed here: https://github.com/pachyderm/pachyderm/pull/1636). so we'll use the windows subsystem for linux (WSL) to run that tool. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## install pachyderm.io"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we'll follow the instructions from here: http://pachyderm.readthedocs.io/en/latest/getting_started/local_installation.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### install minikube\n",
"\n",
"> Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.\n",
"\n",
"start with the guide here: https://kubernetes.io/docs/getting-started-guides/minikube/\n",
"however, note that its not very win10-specific.\n",
"\n",
"i found this tutorial more succint for win10+hyper-v: https://www.ibm.com/support/knowledgecenter/en/SS5PWC/minikube.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### fetch minikube\n",
"download https://github.com/kubernetes/minikube/releases/download/v0.24.0/minikube-windows-amd64.exe to `$PATH`"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" Volume in drive C has no label.\n",
" Volume Serial Number is 5C06-0277\n",
"\n",
" Directory of C:\\users\\user\\Documents\\path\n",
"\n",
"11/29/2017 05:59 PM 41,454,592 minikube.exe\n",
" 1 File(s) 41,454,592 bytes\n",
" 0 Dir(s) 107,529,314,304 bytes free\n"
]
}
],
"source": [
"!dir C:\\users\\user\\Documents\\path\\minikube.exe"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Minikube is a CLI tool that provisions and manages single-node Kubernetes clusters optimized for development workflows.\n",
"\n",
"Usage:\n",
" minikube [command]\n",
"\n",
"Available Commands:\n",
" addons Modify minikube's kubernetes addons\n",
" cache Add or delete an image from the local cache.\n",
" completion Outputs minikube shell completion for the given shell (bash or zsh)\n",
" config Modify minikube config\n",
" dashboard Opens/displays the kubernetes dashboard URL for your local cluster\n",
" delete Deletes a local kubernetes cluster\n",
" docker-env Sets up docker env variables; similar to '$(docker-machine env)'\n",
" get-k8s-versions Gets the list of Kubernetes versions available for minikube when using the localkube bootstrapper\n",
" ip Retrieves the IP address of the running cluster\n",
" logs Gets the logs of the running localkube instance, used for debugging minikube, not user code\n",
" mount Mounts the specified directory into minikube\n",
" profile Profile sets the current minikube profile\n",
" service Gets the kubernetes URL(s) for the specified service in your local cluster\n",
" ssh Log into or run a command on a machine with SSH; similar to 'docker-machine ssh'\n",
" ssh-key Retrieve the ssh identity key path of the specified cluster\n",
" start Starts a local kubernetes cluster\n",
" status Gets the status of a local kubernetes cluster\n",
" stop Stops a running local kubernetes cluster\n",
" update-check Print current and latest version number\n",
" update-context Verify the IP address of the running cluster in kubeconfig.\n",
" version Print the version of minikube\n",
"\n",
"Flags:\n",
" --alsologtostderr log to standard error as well as files\n",
" -b, --bootstrapper string The name of the cluster bootstrapper that will set up the kubernetes cluster. (default \"localkube\")\n",
" -h, --help help for minikube\n",
" --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)\n",
" --log_dir string If non-empty, write log files in this directory\n",
" --logtostderr log to standard error instead of files\n",
" -p, --profile string The name of the minikube VM being used. \n",
"\tThis can be modified to allow for multiple minikube instances to be run independently (default \"minikube\")\n",
" --stderrthreshold severity logs at or above this threshold go to stderr (default 2)\n",
" -v, --v Level log level for V logs\n",
" --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging\n",
"\n",
"Use \"minikube [command] --help\" for more information about a command.\n"
]
}
],
"source": [
"!minikube.exe"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The following Kubernetes versions are available when using the localkube bootstrapper: \n",
"\t- v1.8.0\n",
"\t- v1.7.5\n",
"\t- v1.7.4\n",
"\t- v1.7.3\n",
"\t- v1.7.2\n",
"\t- v1.7.0\n",
"\t- v1.7.0-rc.1\n",
"\t- v1.7.0-alpha.2\n",
"\t- v1.6.4\n",
"\t- v1.6.3\n",
"\t- v1.6.0\n",
"\t- v1.6.0-rc.1\n",
"\t- v1.6.0-beta.4\n",
"\t- v1.6.0-beta.3\n",
"\t- v1.6.0-beta.2\n",
"\t- v1.6.0-alpha.1\n",
"\t- v1.6.0-alpha.0\n",
"\t- v1.5.3\n",
"\t- v1.5.2\n",
"\t- v1.5.1\n",
"\t- v1.4.5\n",
"\t- v1.4.3\n",
"\t- v1.4.2\n",
"\t- v1.4.1\n",
"\t- v1.4.0\n",
"\t- v1.3.7\n",
"\t- v1.3.6\n",
"\t- v1.3.5\n",
"\t- v1.3.4\n",
"\t- v1.3.3\n",
"\t- v1.3.0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"========================================\n",
"kubectl could not be found on your path. kubectl is a requirement for using minikube\n",
"To install kubectl, please do the following:\n",
"\n",
"download kubectl from:\n",
"https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/windows/amd64/kubectl.exe\n",
"Add kubectl to your system PATH\n",
"\n",
"To disable this message, run the following:\n",
"\n",
"minikube config set WantKubectlDownloadMsg false\n",
"========================================\n"
]
}
],
"source": [
"!minikube.exe get-k8s-versions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### fetch kubectl\n",
"\n",
"follow the instructions in stderr above: download https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/windows/amd64/kubectl.exe to `$PATH`"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" Volume in drive C has no label.\n",
" Volume Serial Number is 5C06-0277\n",
"\n",
" Directory of C:\\users\\user\\Documents\\path\n",
"\n",
"09/28/2017 11:52 PM 52,721,152 kubectl.exe\n",
" 1 File(s) 52,721,152 bytes\n",
" 0 Dir(s) 107,565,211,648 bytes free\n"
]
}
],
"source": [
"!dir C:\\users\\user\\Documents\\path\\kubectl.exe"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"kubectl controls the Kubernetes cluster manager. \n",
"\n",
"Find more information at https://github.com/kubernetes/kubernetes.\n",
"\n",
"Basic Commands (Beginner):\n",
" create Create a resource from a file or from stdin.\n",
" expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service\n",
" run Run a particular image on the cluster\n",
" set Set specific features on objects\n",
" run-container Run a particular image on the cluster. This command is deprecated, use \"run\" instead\n",
"\n",
"Basic Commands (Intermediate):\n",
" get Display one or many resources\n",
" explain Documentation of resources\n",
" edit Edit a resource on the server\n",
" delete Delete resources by filenames, stdin, resources and names, or by resources and label selector\n",
"\n",
"Deploy Commands:\n",
" rollout Manage the rollout of a resource\n",
" rolling-update Perform a rolling update of the given ReplicationController\n",
" scale Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job\n",
" autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController\n",
"\n",
"Cluster Management Commands:\n",
" certificate Modify certificate resources.\n",
" cluster-info Display cluster info\n",
" top Display Resource (CPU/Memory/Storage) usage.\n",
" cordon Mark node as unschedulable\n",
" uncordon Mark node as schedulable\n",
" drain Drain node in preparation for maintenance\n",
" taint Update the taints on one or more nodes\n",
"\n",
"Troubleshooting and Debugging Commands:\n",
" describe Show details of a specific resource or group of resources\n",
" logs Print the logs for a container in a pod\n",
" attach Attach to a running container\n",
" exec Execute a command in a container\n",
" port-forward Forward one or more local ports to a pod\n",
" proxy Run a proxy to the Kubernetes API server\n",
" cp Copy files and directories to and from containers.\n",
" auth Inspect authorization\n",
"\n",
"Advanced Commands:\n",
" apply Apply a configuration to a resource by filename or stdin\n",
" patch Update field(s) of a resource using strategic merge patch\n",
" replace Replace a resource by filename or stdin\n",
" convert Convert config files between different API versions\n",
"\n",
"Settings Commands:\n",
" label Update the labels on a resource\n",
" annotate Update the annotations on a resource\n",
" completion Output shell completion code for the specified shell (bash or zsh)\n",
"\n",
"Other Commands:\n",
" api-versions Print the supported API versions on the server, in the form of \"group/version\"\n",
" config Modify kubeconfig files\n",
" help Help about any command\n",
" plugin Runs a command-line plugin\n",
" version Print the client and server version information\n",
"\n",
"Use \"kubectl <command> --help\" for more information about a given command.\n",
"Use \"kubectl options\" for a list of global command-line options (applies to all commands).\n"
]
}
],
"source": [
"!kubectl.exe"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### get kubectl working\n",
"\n",
"show hyper-v network switches:\n",
"\n",
"```\n",
"C:\\Users\\user\\Documents\\path\n",
"λ get-vmswitch\n",
"\n",
"Name SwitchType NetAdapterInterfaceDescription\n",
"---- ---------- ------------------------------\n",
"VirtualSwitch-Wifi External Intel(R) Dual Band Wireless-AC 7260\n",
"Default Switch Internal\n",
"DockerNAT Internal\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"lets try to run minikube (note: `--v=3` replaces `--show-libmachine-logs`, ref: https://github.com/kubernetes/minikube/pull/894):\n",
" \n",
"```\n",
"C:\\Users\\user\\Documents\\code\\pach\n",
"λ minikube start --vm-driver=hyperv --alsologtostderr --hyperv-virtual-switch=VirtualSwitch-Wifi --v=3\n",
"...\n",
"E1130 13:10:30.217828 8132 start.go:150] Error starting host: Error creating host: Error executing step: Running precreate checks.\n",
": Hyper-v commands have to be run as an Administrator.\n",
"\n",
" Retrying.\n",
"E1130 13:10:30.220289 8132 start.go:156] Error starting host: Error creating host: Error executing step: Running precreate checks.\n",
": Hyper-v commands have to be run as an Administrator\n",
"================================================================================\n",
"An error has occurred. Would you like to opt in to sending anonymized crash\n",
"information to minikube to help prevent future errors?\n",
"To opt out of these messages, run the command:\n",
" minikube config set WantReportErrorPrompt false\n",
"================================================================================\n",
"Please enter your response [Y/n]:\n",
"n\n",
"```\n",
"\n",
"...so we need to run as admin. lets try again.\n",
"\n",
"...after some trial and error, after ensuring any vpn is not active...\n",
"\n",
"```\n",
"C:\\Users\\user\\Documents\\path\n",
"λ minikube start --vm-driver=hyperv --alsologtostderr --hyperv-virtual-switch=VirtualSwitch-Wifi --v=3\n",
"...\n",
"Creating SSH key...\n",
"Creating VM...\n",
"Using switch \"VirtualSwitch-Wifi\"\n",
"Creating VHD\n",
"Starting VM...\n",
"Waiting for host to start...\n",
"...\n",
"Kubectl is now configured to use the cluster.\n",
"Loading cached images from config file.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"demonstrate minikube is working:\n",
" \n",
"```\n",
"λ kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080\n",
"deployment \"hello-minikube\" created\n",
"\n",
"λ kubectl expose deployment hello-minikube --type=NodePort\n",
"service \"hello-minikube\" exposed\n",
"\n",
"λ kubectl get pod\n",
"NAME READY STATUS RESTARTS AGE\n",
"hello-minikube-57889c865c-rrtw7 1/1 Running 0 50s\n",
"\n",
"λ minikube service hello-minikube --url\n",
"http://192.168.1.46:30226\n",
"```\n",
"\n",
"![in browser](hello-minikube.png)\n",
"\n",
"```\n",
"λ kubectl delete deployment hello-minikube\n",
"deployment \"hello-minikube\" deleted\n",
"\n",
"λ minikube stop\n",
"Stopping local Kubernetes cluster...\n",
"Machine stopped.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### install pachyderm.io\n",
"\n",
"#### install client\n",
"\n",
"first need to install pachctl. fetch it from the github releases page. i used: https://github.com/pachyderm/pachyderm/releases/download/v1.6.5/pachctl_1.6.5_amd64.deb\n",
"\n",
"pach only runs on linux/osx, so we'll have to us WSL to install it:\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:/mnt/c/Users/user/Documents/code/pach$ sudo dpkg -i pachctl_1.6.5_amd64.deb\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### install deployment\n",
"\n",
"can't use `pachctl deploy local` directly because there is no `pachctl` distribution for windows. so we'll use WSL, and import the deployment into kubectl manually.\n",
"\n",
"use `pachctl ... --dry-run` to emit a kubectl config. then use `kubectl create -f ...` to load that config. via: https://github.com/pachyderm/pachyderm/blob/395935ba9a5623cb408eef0c3493477898b40789/doc/deployment/on_premises.md\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:/mnt/c/Users/user/Documents/code/pach$ pachctl deploy local --dry-run\n",
"{\"kind\":\"ServiceAccount\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"pachyderm\",\"creationTimestamp\":null,\"labels\":{\"app\":\"\",\"suite\":\"pachyderm\"}}}\n",
"...\n",
"\n",
"user@DESKTOP-VBLTDLS:/mnt/c/Users/user/Documents/code/pach$ pachctl deploy local --dry-run > pachyderm.json\n",
"```\n",
"\n",
"then in powershell:\n",
"\n",
"```\n",
"λ minikube start --vm-driver=hyperv --alsologtostderr --hyperv-virtual-switch=VirtualSwitch-Wifi --v=3\n",
"...\n",
"Kubectl is now configured to use the cluster.\n",
"Loading cached images from config file.\n",
"\n",
"λ kubectl create -f .\\pachyderm.json\n",
"serviceaccount \"pachyderm\" created\n",
"deployment \"etcd\" created\n",
"service \"etcd\" created\n",
"service \"pachd\" created\n",
"deployment \"pachd\" created\n",
"secret \"pachyderm-storage-secret\" created\n",
"\n",
"λ kubectl get pod\n",
"NAME READY STATUS RESTARTS AGE\n",
"etcd-78f9d695c9-mbsr8 1/1 Running 0 2m\n",
"pachd-57fd698b9d-wx4ns 1/1 Running 0 2m\n",
"\n",
"λ kubectl get all\n",
"NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\n",
"deploy/etcd 1 1 1 1 2m\n",
"deploy/pachd 1 1 1 1 2m\n",
"\n",
"NAME DESIRED CURRENT READY AGE\n",
"rs/etcd-78f9d695c9 1 1 1 2m\n",
"rs/pachd-57fd698b9d 1 1 1 2m\n",
"\n",
"NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\n",
"deploy/etcd 1 1 1 1 2m\n",
"deploy/pachd 1 1 1 1 2m\n",
"\n",
"NAME DESIRED CURRENT READY AGE\n",
"rs/etcd-78f9d695c9 1 1 1 2m\n",
"rs/pachd-57fd698b9d 1 1 1 2m\n",
"\n",
"NAME READY STATUS RESTARTS AGE\n",
"po/etcd-78f9d695c9-mbsr8 1/1 Running 0 2m\n",
"po/pachd-57fd698b9d-wx4ns 1/1 Running 0 2m\n",
"\n",
"NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n",
"svc/etcd NodePort 10.102.228.219 <none> 2379:32379/TCP 2m\n",
"svc/hello-minikube NodePort 10.98.68.62 <none> 8080:30226/TCP 26m\n",
"svc/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28m\n",
"svc/pachd NodePort 10.98.106.198 <none> 650:30650/TCP,651:30651/TCP,652:30652/TCP,999:30999/TCP 2m\n",
"\n",
"λ minikube service list\n",
"|-------------|----------------------|--------------------------------|\n",
"| NAMESPACE | NAME | URL |\n",
"|-------------|----------------------|--------------------------------|\n",
"| default | etcd | http://192.168.1.47:32379 |\n",
"| default | hello-minikube | http://192.168.1.47:30226 |\n",
"| default | kubernetes | No node port |\n",
"| default | pachd | http://192.168.1.47:30650 |\n",
"| | | http://192.168.1.47:30651 |\n",
"| | | http://192.168.1.47:30652 |\n",
"| | | http://192.168.1.47:30999 |\n",
"| kube-system | kube-dns | No node port |\n",
"| kube-system | kubernetes-dashboard | http://192.168.1.47:30000 |\n",
"|-------------|----------------------|--------------------------------|\n",
"\n",
"λ minikube service pachd --url\n",
"http://192.168.1.47:30650\n",
"http://192.168.1.47:30651\n",
"http://192.168.1.47:30652\n",
"http://192.168.1.47:30999\n",
"```\n",
"\n",
"use this URL with `pachctl`:\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:/mnt/c/Users/user/Documents/code/pach$ ADDRESS=192.168.1.47:30650 pachctl version\n",
"COMPONENT VERSION\n",
"pachctl 1.6.5\n",
"pachd 1.6.5\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## work through pachyderm.io demo"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"user@DESKTOP-VBLTDLS:...$ ADDRESS=192.168.1.47:30650 pachctl create-repo images\n",
"\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ ADDRESS=192.168.1.47:30650 pachctl list-repo\n",
"NAME CREATED SIZE\n",
"images 15 seconds ago 0B\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ export ADDRESS=192.168.1.47:30650\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl put-file images master liberty.png -c -f http://imgur.com/46Q8nDz.png\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-repo\n",
"NAME CREATED SIZE\n",
"images 51 seconds ago 57.27KiB\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-commit images\n",
"REPO ID PARENT STARTED DURATION SIZE\n",
"images ff04623269934905b58046dd0500ac22 <none> 20 seconds ago Less than a second 57.27KiB\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-file images master\n",
"NAME TYPE SIZE\n",
"liberty.png file 57.27KiB\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl get-file images master liberty.png | file -\n",
"/dev/stdin: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 600x800, frames 3\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl create-pipeline -f https://raw.githubusercontent.com/pachyderm/pachyderm/master/doc/examples/opencv/edges.json\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-job\n",
"ID OUTPUT COMMIT STARTED DURATION RESTART PROGRESS DL UL STATE\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-repo\n",
"NAME CREATED SIZE\n",
"edges 18 seconds ago 0B\n",
"images 2 minutes ago 57.27KiB\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"i'm not quite sure how to know when the docker image is downloaded and the job is ready/run. trying to list the job a little later works:\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-job\n",
"ID OUTPUT COMMIT STARTED DURATION RESTART PROGRESS DL UL STATE\n",
"77c51975-d576-400f-9023-270a90eb14a4 edges/c9ddcfae54a04d6b9ba7c4cd6319fb55 2 minutes ago 1 second 0 1 + 0 / 1 57.27KiB 22.22KiB success\n",
"```\n",
"\n",
"#### aside: logging\n",
"\n",
"according to http://docs.pachyderm.io/en/v1.2.6/FAQ.html#how-does-pachyderm-handle-logging we can use `pachctl get-logs`, though this doesn't seem to work here:\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:...$ pachctl get-logs\n",
"no pods belonging to the rc \"pachd\" were found\n",
"\n",
"# verify pachd is still running\n",
"user@DESKTOP-VBLTDLS:...$ pachctl version\n",
"COMPONENT VERSION\n",
"pachctl 1.6.5\n",
"pachd 1.6.5\n",
"```\n",
"\n",
"asking kubectl does get us something:\n",
"\n",
"```\n",
"λ kubectl logs deploy/pachd\n",
"2017-11-30T19:32:23Z INFO pps.API.ListJobStream {\"duration\":0.0040309,\"request\":{},\"response\":\"stream containing 0 JobInfos\"}\n",
"time=\"2017-11-30T19:32:31Z\" level=info msg=\"objectCache stats: {Gets:3 CacheHits:1 PeerLoads:0 PeerErrors:0 Loads:2 LoadsDeduped:2 LocalLoads:2 LocalLoadErrs:0 ServerRequests:0}\"\n",
"...\n",
"2017-11-30T19:35:08Z INFO pps.API.ListJobStream {\"request\":{}}\n",
"2017-11-30T19:35:09Z INFO pps.API.ListJobStream {\"duration\":0.0995256,\"request\":{},\"response\":\"stream containing 1 JobInfos\"}\n",
"time=\"2017-11-30T19:35:31Z\" level=info msg=\"objectCache stats: {Gets:3 CacheHits:1 PeerLoads:0 PeerErrors:0 Loads:2 LoadsDeduped:2 LocalLoads:2 LocalLoadErrs:0 ServerRequests:0}\"\n",
"time=\"2017-11-30T19:35:31Z\" level=info msg=\"tagCache stats: {Gets:0 CacheHits:0 PeerLoads:0 PeerErrors:0 Loads:0 LoadsDeduped:0 LocalLoads:0 LocalLoadErrs:0 ServerRequests:0}\"\n",
"time=\"2017-11-30T19:35:31Z\" level=info msg=\"objectInfoCache stats: {Gets:5 CacheHits:3 PeerLoads:0 PeerErrors:0 Loads:2 LoadsDeduped:2 LocalLoads:2 LocalLoadErrs:0 ServerRequests:0}\"\n",
"2017-11-30T19:36:03Z INFO pps.API.GetLogs {\"request\":{}}\n",
"2017-11-30T19:36:03Z ERROR pps.API.GetLogs {\"duration\":0.0040348,\"error\":\"no pods belonging to the rc \\\"pachd\\\" were found\",\"request\":{}}\n",
"time=\"2017-11-30T19:36:31Z\" level=info msg=\"objectCache stats: {Gets:3 CacheHits:1 PeerLoads:0 PeerErrors:0 Loads:2 LoadsDeduped:2 LocalLoads:2 LocalLoadErrs:0 ServerRequests:0}\"\n",
"```\n",
"\n",
"but i don't know how to get the logs around job execution.\n",
"\n",
"#### back to the tutorial\n",
"\n",
"looks like the jobs are running no problem:\n",
"\n",
"```\n",
"user@DESKTOP-VBLTDLS:...$ pachctl get-file edges master liberty.png | file -\n",
"/dev/stdin: PNG image data, 600 x 800, 8-bit/color RGBA, non-interlaced\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl put-file images master AT-AT.png -c -f http://imgur.com/8MN9Kg0.png\n",
"user@DESKTOP-VBLTDLS:...$ pachctl put-file images master kitten.png -c -f http://imgur.com/g2QnNqa.png\n",
"\n",
"user@DESKTOP-VBLTDLS:...$ pachctl list-job\n",
"ID OUTPUT COMMIT STARTED DURATION RESTART PROGRESS DL UL STATE \n",
"8f147f53-5d1c-46d5-aa37-b8b3347e3ab2 edges/66a68b7aaa3848cb9602fb17c2518564 4 seconds ago Less than a second 0 1 + 2 / 3 102.4KiB 74.21KiB success\n",
"2cd242ef-fc43-4242-8395-3c596f9928c6 edges/623e6569c6364135a1abc44e8348c184 10 seconds ago Less than a second 0 1 + 1 / 2 78.7KiB 37.15KiB success\n",
"77c51975-d576-400f-9023-270a90eb14a4 edges/c9ddcfae54a04d6b9ba7c4cd6319fb55 15 minutes ago 1 second 0 1 + 0 / 1 57.27KiB 22.22KiB success\n",
"```\n",
"\n",
"success!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# conclusion\n",
"\n",
"its feasible to install a local instance of pachyderm.io using minikube on a win10 system running hyper-v. as my first experience with pachyerm.io, i completed the above in approximately 2.5h. as you can see from the logging aside, i'm not convinced everything works 100%, but its clear that the core job management processes are good to go."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment