- The latest documentation at https://dcos.io/docs/latest.
- Announcing Mesosphere DC/OS 1.9
- DC/OS 1.9.0 Release Candidate 2
- Bringing Production-Proven Data Services to DC/OS 1.9 with our Partners
- DC/OS provides one-click installation of data services such as databases, message queues, and analytics engines, on-par with cloud providers such as Amazon Web Services.
- DC/OS utilizes two-level scheduling with frameworks to implement application-aware lifecycle management
- Open source SDK provides a high-level interface for building new stateful services on DC/OS
- With the SDK, developers can write a stateful service complete with persistent volumes, fault tolerance, and configuration management in about 100 lines of code.
- DC/OS, an open source project to bring the power of Mesos to the broader community
- OCI and the Container Storage Interface (CSI)
- OCI, CSI, and CNI
- OCI v1.0: Bringing Containers Closer to Standardization
- Open Container Initiative (OCI) Releases v1.0 of Container Standards
NOTE: For now use the quite old Vagrant 1.8.4 with VirtualBox 5.0.26 due to bugs in the recent versions of Vagrant
Installation steps:
-
- Download DC/OS Installer
- Install DC/OS Vagrant
-
Local DC/OS cluster provisioning
export DCOS_CONFIG_PATH=etc/config-1.8.yaml- Deploy
vagrant up --provision m1 a1 p1 boot(see below)- Read PROVISIONING in the official documentation of Vagrant.
Using Minimal Cluster below (as the default small cluster might not be needed yet).
➜ dcos-vagrant git:(master) vagrant up m1 a1 boot
➜ dcos-vagrant git:(master) vagrant up m1 a1 boot
Bringing machine 'm1' up with 'virtualbox' provider...
Bringing machine 'a1' up with 'virtualbox' provider...
Bringing machine 'boot' up with 'virtualbox' provider...
==> m1: Box 'mesosphere/dcos-centos-virtualbox' could not be found. Attempting to find and install...
m1: Box Provider: virtualbox
m1: Box Version: ~> 0.8.0
==> m1: Loading metadata for box 'https://downloads.dcos.io/dcos-vagrant/metadata.json'
m1: URL: https://downloads.dcos.io/dcos-vagrant/metadata.json
==> m1: Adding box 'mesosphere/dcos-centos-virtualbox' (v0.8.0) for provider: virtualbox
m1: Downloading: https://downloads.dcos.io/dcos-vagrant/dcos-centos-virtualbox-0.8.0.box
m1: Calculating and comparing box checksum...
==> m1: Successfully added box 'mesosphere/dcos-centos-virtualbox' (v0.8.0) for 'virtualbox'!
...
==> boot: DC/OS Installation Complete
==> boot: Web Interface: http://m1.dcos/
Stopping a DC/OS cluster.
➜ dcos-vagrant git:(master) vagrant halt m1 a1 p1 boot
==> boot: Attempting graceful shutdown of VM...
==> p1: Attempting graceful shutdown of VM...
==> a1: Attempting graceful shutdown of VM...
==> a1: Forcing shutdown of VM...
==> m1: Attempting graceful shutdown of VM...
==> m1: Forcing shutdown of VM...
NOTE: Had to destroy the DC/OS cluster using vagrant destroy -f as I could not provision the machines.
NOTE: vagrant up m1 a1 p1 boot worked before quite fine, but I don't know if I need it.
$ curl -O https://downloads.dcos.io/binaries/cli/darwin/x86-64/dcos-1.8/dcos
$ chmod +x dcos
$ ./dcos help
Command line utility for the Mesosphere Datacenter Operating
System (DC/OS). The Mesosphere DC/OS is a distributed operating
system built around Apache Mesos. This utility provides tools
for easy management of a DC/OS installation.
Available DC/OS commands:
auth Authenticate to DC/OS cluster
config Manage the DC/OS configuration file
help Display help information about DC/OS
job Deploy and manage jobs in DC/OS
marathon Deploy and manage applications to DC/OS
node Administer and manage DC/OS cluster nodes
package Install and manage DC/OS software packages
service Manage DC/OS services
spark Run and manage Spark jobs
task Manage DC/OS tasks
Get detailed command description with 'dcos <command> --help'.
$ ./dcos --version
dcoscli.version=0.4.13
dcos.version=1.8.4
dcos.commit=e64024af95b62c632c90b9063ed06296fcf38ea5
dcos.bootstrap-id=5b4aa43610c57ee1d60b4aa0751a1fb75824c083
- Read Apache DC/OS Spark
- The latest version of Apache DC/OS Spark is Spark 1.1.1-2.2.0 (with Apache Spark 2.2.0 being the latest too)
- See https://github.com/mesosphere/dcos-spark/issues/25
➜ dcos-vagrant git:(master) pwd
/Users/jacek/dev/oss/dcos-vagrant
➜ dcos-vagrant git:(master) ./dcos config set core.dcos_url http://m1.dcos
[core.dcos_url]: set to 'http://m1.dcos'
➜ dcos-vagrant git:(master) ./dcos package install spark --cli
Installing CLI subcommand for package [spark] version [1.0.2-2.0.0]
Unable to install CLI subcommand. Missing required program 'virtualenv'.
Please see installation instructions: https://virtualenv.pypa.io/en/latest/installation.html
➜ dcos-vagrant git:(master) pip install virtualenv
Collecting virtualenv
Downloading virtualenv-15.0.3-py2.py3-none-any.whl (3.5MB)
100% |████████████████████████████████| 3.5MB 314kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-15.0.3
// you could use the following instead to debug the command
// ./dcos --log-level=debug package install spark
➜ dcos-vagrant git:(master) ./dcos package install spark --cli
Installing CLI subcommand for package [spark] version [1.0.2-2.0.0]
New command available: dcos spark
➜ dcos-vagrant git:(master) ls ~/.dcos/subcommands
spark
➜ dcos-vagrant git:(master) ./dcos --log-level=debug spark
Usage:
dcos spark --help
dcos spark --info
dcos spark --version
dcos spark --config-schema
dcos spark run --help
dcos spark run --submit-args=<spark-args>
[--docker-image=<docker-image>]
[--verbose]
dcos spark status <submissionId> [--verbose]
dcos spark log <submissionId>
[--follow]
[--lines_count=<lines_count>]
[--file=<file>]
dcos spark kill <submissionId> [--verbose]
dcos spark webui
➜ dcos-vagrant git:(master) ./dcos --log-level=INFO spark webui
http://m1.dcos/service/spark/ui
Quoting a discussion in DCOS slack channel:
A complete wipe of the spark pkg via
dcos package uninstall+ deleted spark related entries from the zookeeper via its exhibitor web ui. After a fresh re-install of Spark I was able to start spark drivers.
Quoting a discussion in DCOS slack channel:
{
"id": "spark-client",
"cmd": "export JAVA_HOME=$MESOS_SANDBOX/jre1.8.0_112 && $MESOS_SANDBOX/spark-2.1.1-bin-2.6/bin/spark-submit --master mesos://leader.mesos:5050 --deploy-mode client --conf spark.executor.memory=1g --conf spark.executor.cores=2 --conf spark.cores.max=8 --conf spark.mesos.executor.home=/opt/spark/dist --conf spark.mesos.executor.docker.image=mesosphere/spark:1.1.0-2.1.1-hadoop-2.6 --class org.apache.spark.examples.SparkPi $MESOS_SANDBOX/spark-2.1.1-bin-2.6/examples/jars/spark-examples_2.11-2.1.1.jar",
"cpus":1,
"mem":2048,
"fetch": [
{ "uri" : "https://downloads.mesosphere.com/spark/assets/spark-2.1.1-bin-2.6.tgz"},
{
"uri": "https://downloads.mesosphere.com/java/jre-8u112-linux-x64.tar.gz"
}
]
}
// similar to docker run
$ dcos marathon app add https://dcos.io/docs/latest/usage/nginx.json