> lsb_release -a | grep Description
Description: Ubuntu 12.04.4 LTS
> sudo lsof -i :2181 | grep '(LISTEN)'
java 29720 zookeeper 27u IPv4 236483534 0t0 TCP *:2181 (LISTEN)
> mesos-master --version
mesos 0.21.1
> ps aux | grep mesos-master
/usr/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 --log_dir=/var/log/mesos --cluster=pangramia --quorum=1 --work_dir=/var/lib/mesos
> ps aux | grep mesos-slave
/usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --log_dir=/var/log/mesos --containerizers=docker,mesos --executor_registration_timeout=5mins --attributes=containerizer_path:/usr/local/bin/deimos;isolation:external;rack:a
> curl -s http://localhost:5050/stats.json | python2.7 -mjson.tool | grep activated_slaves
"activated_slaves": 1,
"deactivated_slaves": 0,
> sudo /usr/share/zookeeper/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /mesos
[info_0000000000, log_replicas]
[zk: localhost:2181(CONNECTED) 3] ls /
[local, cassandraMesos, mesos, zookeeper]
[zk: localhost:2181(CONNECTED) 3] ls /local/service/mesos-native-log
[0000000010]
UPDATED: after deletion of zk_in_proc, aurora scheduler added aurora's path in zk
[zk: localhost:2181(CONNECTED) 1] ls /aurora/scheduler/
member_0000000011 singleton_candidate_0000000010
> env | grep AURORA
AURORA_HOME=/usr/local/aurora-scheduler
AURORA_DIST=/opt/apache-aurora-0.7.0-incubating
> cd $AURORA_DIST/third_party && ls -a
. mesos-0.20.1-py2.7-linux-x86_64.egg mesos.interface-0.20.1-py2.7.egg mesos.native-0.20.1-py2.7-linux-x86_64.egg
.. mesos-0.21.0-py2.7-linux-x86_64.egg mesos.interface-0.21.1-py2.7.egg mesos.native-0.21.1-py2.7-linux-x86_64.egg
> cd $AURORA_DIST/dist && ls -a
. .. aurora.pex gc_executor.pex thermos_executor.pex thermos_observer.pex thermos_runner.pex
> cat /etc/profile | grep AURORA | grep PATH
PATH=$PATH:$AURORA_HOME/bin:$AURORA_DIST/dist
vi $AURORA_HOME/bin/aurora-scheduler-devcluster
#!/bin/bash
AURORA_HOME=/usr/local/aurora-scheduler
AURORA_DIST=/opt/apache-aurora-0.7.0-incubating/dist
# Flags controlling the JVM.
JAVA_OPTS=(
-Xmx2g
-Xms2g
# GC tuning, etc.
-Djava.library.path=/usr/local/lib
-Dlog4j.configuration="file:///etc/zookeeper/conf/log4j.properties"
)
aurora_flags=(
-thermos_executor_path=$AURORA_DIST/thermos_executor.pex
-gc_executor_path=$AURORA_DIST/gc_executor.pex
-http_port=8091
## UPDATED: FIXED by comment out the following zk_in_proc line,
## since this forbid to use zk, and use in process (testing) zk instead
## -zk_in_proc=true
-zk_endpoints=localhost:2181
-zk_session_timeout=2secs
# -zk_digest_credentials=xxx:xxxx
-serverset_path=/aurora/scheduler
-mesos_master_address=zk://localhost:2181/mesos
# -mesos_ssl_keyfile=src/test/resources/com/twitter/aurora/scheduler/app/AuroraTestKeyStore
-cluster_name=devcluster
-native_log_quorum_size=1
-native_log_file_path=$AURORA_HOME/db
-native_log_zk_group_path=/local/service/mesos-native-log
-backup_dir=$AURORA_HOME/backups
-logtostderr
-vlog=INFO
)
export GLOG_v=0
export LIBPROCESS_PORT=8093
set -x
JAVA_OPTS="${JAVA_OPTS[*]}" exec "$AURORA_HOME/bin/aurora-scheduler" "${aurora_flags[@]}"
cd $AURORA_HOME && sudo ./aurora-scheduler-devcluster
Feb 17, 2015 8:37:04 PM com.twitter.common.args.ArgScanner process
INFO: hostname (org.apache.aurora.scheduler.app.SchedulerMain.hostname): null
Feb 17, 2015 8:37:04 PM com.twitter.common.args.ArgScanner process
INFO: http_port (org.apache.aurora.scheduler.http.JettyServerModule.http_port): 8091
Feb 17, 2015 8:37:04 PM com.twitter.common.args.ArgScanner process
INFO: serverset_path (org.apache.aurora.scheduler.app.SchedulerMain.serverset_path): /aurora/scheduler
Feb 17, 2015 8:37:04 PM com.twitter.common.args.ArgScanner process
I0217 20:37:08.957415 1559 group.cpp:385] Trying to create path '/local/service/mesos-native-log' in ZooKeeper
2015-02-17 20:37:08,957:1403(0x7fd7eb240700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:2181], sessionId=0x14b974b1b1d0018, negotiated timeout=4000
I0217 20:37:08.958426 1563 group.cpp:313] Group process (group(1)@10.149.254.37:8093) connected to ZooKeeper
I0217 20:37:08.958518 1563 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0217 20:37:08.958554 1563 group.cpp:385] Trying to create path '/local/service/mesos-native-log' in ZooKeeper
I0217 17:37:09.154 THREAD1 com.twitter.common.util.StateMachine$Builder$1.execute: storage state machine transition PREPARED -> READY
I0217 17:37:09.160 THREAD137 org.apache.aurora.scheduler.mesos.SchedulerDriverService.startUp: Connecting to mesos master: zk://localhost:2181/mesos
I0217 20:37:09.182327 1559 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0217 17:37:09.184 THREAD1 com.twitter.common.application.Lifecycle.awaitShutdown: Awaiting shutdown
I0217 20:37:09.184404 1559 detector.cpp:138] Detected a new leader: (id='0')
I0217 20:37:09.184581 1560 group.cpp:659] Trying to get '/mesos/info_0000000000' in ZooKeeper
I0217 20:37:09.185600 1562 detector.cpp:433] A new leading master (UPID=master@10.149.254.37:5050) is detected
I0217 20:37:09.185735 1562 sched.cpp:234] New master detected at master@10.149.254.37:5050
I0217 17:37:09.276 THREAD139 org.apache.aurora.scheduler.mesos.MesosSchedulerImpl.registered: Registered with ID value: "20150216-110011-637441290-5050-17760-0000"
, master: id: "20150217-142747-637441290-5050-29888"
ip: 637441290
port: 5050
pid: "master@10.149.254.37:5050"
hostname: "bymsq-bsu-hmetrics002"
W0217 17:37:09.362 THREAD140 com.twitter.common.zookeeper.ServerSetImpl.join: Joining a ServerSet without a shard ID is deprecated and will soon break.
I0217 17:37:09.398 THREAD140 com.twitter.common.zookeeper.Group$ActiveMembership.join: Set group member ID to member_0000000001
I0217 17:37:09.417 THREAD132 com.twitter.common.zookeeper.ServerSetImpl$ServerSetWatcher.logChange: server set /aurora/scheduler change: from 0 members to 1
joined:
ServiceInstance(serviceEndpoint:Endpoint(host:bymsq-bsu-hmetrics002, port:8091), additionalEndpoints:{http=Endpoint(host:bymsq-bsu-hmetrics002, port:8091)}, status:ALIVE)
I0217 17:37:09.417 THREAD132 org.apache.aurora.scheduler.http.LeaderRedirect$SchedulerMonitor.onChange: Found leader scheduler at [ServiceInstance(serviceEndpoint:Endpoint(host:bymsq-bsu-hmetrics002, port:8091), additionalEndpoints:{http=Endpoint(host:bymsq-bsu-hmetrics002, port:8091)}, status:ALIVE)]
curl -s http://localhost:8091/vars | grep framework_registered
framework_registered 1
[{
"name": "devcluster",
"zk": "localhost",
"zk_port":"2181",
"slave_root": "/var/lib/mesos",
"slaverundirectory": "latest",
"scheduler_zk_path": "/aurora/scheduler",
"auth_mechanism": "UNAUTHENTICATED"
}]
~
UPDATED: The following errors/warn disappeared after deletion of zk_in_proc line
aurora.pex job create devcluster/weblab/devel/hello_server hello_server.aurora
WARN] Could not connect to scheduler: No schedulers detected in devcluster!
WARN] Could not connect to scheduler: Failed to connect to Zookeeper within 10 seconds.
The log of aurora.pex client with verbose flag
UPDATED: Warning/errors disappeared, see the resolutions steps at the bottom of the comment
UPDATE: This was fixed by the following instructions
sudo /usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --ip=10.149.6.18 --hostname=10.149.6.18 --attributes="host:10.149.6.18;rack:a" --work_dir="/var/lib/mesos" --containerizers=mesoswith ip and rack attributes