BruceZu/gist:6054cc22ad8a298999013f3edee14330

BruceZu · 2017-05-25T00:54:52Z

$ cat docker-compose.yml
version: '3'

services:
mongo_arbiter:
container_name: arbiter
build:
context: .
dockerfile: nodes/mongo/Dockerfile

mongo_secondary_1:
    container_name: secondary_1
    build:
        context: .
        dockerfile: nodes/mongo/Dockerfile

mongo_secondary_2:
    container_name: secondary_2
    build:
        context: .
        dockerfile: nodes/mongo/Dockerfile

mongo:
    container_name: primary
    build:
        context: .
        dockerfile: nodes/mongo/Dockerfile
    links:
        - mongo_secondary_1
        - mongo_secondary_2
        - mongo_arbiter

BruceZu · 2017-05-25T22:26:46Z

bzu@bruce-laptop:~/tmp/test$ docker ps 
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7a69fffb8261        mongo:3.4.4         "docker-entrypoint..."   About an hour ago   Up 23 minutes       27017/tcp           tmp_mongo_arbiter_1
d1aad75d90fb        mongo:3.4.4         "docker-entrypoint..."   About an hour ago   Up 23 minutes       27017/tcp           tmp_mongo_secondary_2_1
5adc51f11a1d        mongo:3.4.4         "docker-entrypoint..."   About an hour ago   Up 23 minutes       27017/tcp           tmp_mongo_primary_1
0d77d10b6380        mongo:3.4.4         "docker-entrypoint..."   About an hour ago   Up 23 minutes       27017/tcp           tmp_mongo_secondary_1_1
bzu@bruce-laptop:~/tmp/test$ docker exec -it  tmp_mongo_primary_1 mongo 
MongoDB shell version v3.4.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.4
Server has startup warnings: 
2017-05-25T22:02:26.604+0000 I STORAGE  [initandlisten] 
2017-05-25T22:02:26.604+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-05-25T22:02:26.604+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] 
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] 
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] 
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-05-25T22:02:26.841+0000 I CONTROL  [initandlisten] 
ft-mongo-replica-set:PRIMARY> rs.status()
{
	"set" : "ft-mongo-replica-set",
	"date" : ISODate("2017-05-25T22:26:12.080Z"),
	"myState" : 1,
	"term" : NumberLong(10),
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1495751163, 1),
			"t" : NumberLong(10)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1495751163, 1),
			"t" : NumberLong(10)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1495751163, 1),
			"t" : NumberLong(10)
		}
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "mongo_primary:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 1426,
			"optime" : {
				"ts" : Timestamp(1495751163, 1),
				"t" : NumberLong(10)
			},
			"optimeDate" : ISODate("2017-05-25T22:26:03Z"),
			"electionTime" : Timestamp(1495749772, 1),
			"electionDate" : ISODate("2017-05-25T22:02:52Z"),
			"configVersion" : 1,
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "mongo_secondary_1:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1425,
			"optime" : {
				"ts" : Timestamp(1495751163, 1),
				"t" : NumberLong(10)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1495751163, 1),
				"t" : NumberLong(10)
			},
			"optimeDate" : ISODate("2017-05-25T22:26:03Z"),
			"optimeDurableDate" : ISODate("2017-05-25T22:26:03Z"),
			"lastHeartbeat" : ISODate("2017-05-25T22:26:11.588Z"),
			"lastHeartbeatRecv" : ISODate("2017-05-25T22:26:11.713Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "mongo_primary:27017",
			"configVersion" : 1
		},
		{
			"_id" : 2,
			"name" : "mongo_secondary_2:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1425,
			"optime" : {
				"ts" : Timestamp(1495751163, 1),
				"t" : NumberLong(10)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1495751163, 1),
				"t" : NumberLong(10)
			},
			"optimeDate" : ISODate("2017-05-25T22:26:03Z"),
			"optimeDurableDate" : ISODate("2017-05-25T22:26:03Z"),
			"lastHeartbeat" : ISODate("2017-05-25T22:26:11.588Z"),
			"lastHeartbeatRecv" : ISODate("2017-05-25T22:26:10.206Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "mongo_secondary_1:27017",
			"configVersion" : 1
		},
		{
			"_id" : 3,
			"name" : "mongo_arbiter:27017",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 1425,
			"lastHeartbeat" : ISODate("2017-05-25T22:26:11.589Z"),
			"lastHeartbeatRecv" : ISODate("2017-05-25T22:26:07.142Z"),
			"pingMs" : NumberLong(0),
			"configVersion" : 1
		}
	],
	"ok" : 1
}
ft-mongo-replica-set:PRIMARY>

BruceZu · 2017-05-26T00:35:16Z

Cases of accessing MongoDB replica set using PyMongo

As reference for upgrading existing MongoDB settings and application code to access MongoDB Replica set

Using PyMongo 3.4. It supports MongoDB 3.4

$ python  -c "import sys; print(sys.version)"
3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609]
$ python  -c "import pymongo; print(pymongo.version); print(pymongo.has_c())"
3.4.0
True

Reference case for update Settings and application code

Case1: Access AWS MongoDB replica set from EC2 'sshhop'

Access db 'foo'

(bzuvenv) ubuntu@ip-172-31-8-191:~/tmp$ python
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymongo
>>> primary_private_dns = 'ip-172-31-4-46.us-west-2.compute.internal'
>>> secondary_1_private_dns = 'ip-172-31-13-173.us-west-2.compute.internal'
>>> replica_set_name = 'ft-mongo-replica-set'
>>> uri = 'mongodb://{},{}/?replicaSet={}'.format(primary_private_dns, secondary_1_private_dns,replica_set_name)
>>> client = pymongo.MongoClient(uri)
>>> db = client.foo
>>> db.name
'foo'

Operation on db 'foo'

>>> db.my_collection.insert_one({"x":10}).inserted_id
ObjectId('5927b623ebe8ad792c83d5c3')
>>> db.my_collection.find_one()
{'x': 10, '_id': ObjectId('5927b623ebe8ad792c83d5c3')}

Check replica set nodes and the one the client is connecting

>>> print(client.nodes)
frozenset({('ip-172-31-13-173.us-west-2.compute.internal', 27017), ('ip-172-31-13-124.us-west-2.compute.internal', 27017), ('ip-172-31-4-46.us-west-2.compute.internal', 27017), ('ip-172-31-2-3.us-west-2.compute.internal', 27017)})
>>> db.client.address
('ip-172-31-4-46.us-west-2.compute.internal', 27017)

Check read preference and change the db 'foo' read preference to be ReadPreference.SECONDARY

>>> client.read_preference
Primary()
>>> from pymongo import ReadPreference
>>> db = client.get_database('foo', read_preference=ReadPreference.SECONDARY)
>>> db.read_preference
Secondary(tag_sets=None, max_staleness=-1)
>>>

Case2: Access local docker dev environment MongoDB replica set from local 'worker' Docker container

Check 'worker' Docker container Python and Pymongo version

$ docker exec -it $(docker ps | grep worker | cut -d" " -f 1) python
Python 3.5.3 (default, May 11 2017, 22:09:56) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymongo
>>> pymongo.version
'3.4.0'

Access Mongo replica set

>>> from pymongo  import MongoClient, ReadPreference
>>> uri = 'mongodb://{},{}/?replicaSet={}'.format('mongo_primary', 'mongo_secondary_1', 'ft-mongo-replica-set')
>>> client = MongoClient(uri, read_preference=ReadPreference.PRIMARY_PREFERRED)
>>> client.nodes
frozenset({('mongo_arbiter', 27017), ('mongo_secondary_2', 27017), ('mongo_primary', 27017), ('mongo_secondary_1', 27017)})
>>> db = client.foo
>>> db.name
'foo'
>>> db.client.address
('mongo_primary', 27017)

Common operations

>>> db.test_collection.insert_one({"bruce":1}).inserted_id
ObjectId('5928876a9559d4006ed890da')
>>> db.test_collection.find_one()
{'_id': ObjectId('592887609559d4006ed890d9'), 'bruce': 1}
>>>

Application code need care

1> ConnectionFailure

Check if the server is available like this:

from pymongo.errors import ConnectionFailure
client = MongoClient()
try:
    # The ismaster command is cheap and does not require auth.
    client.admin.command('ismaster')
except ConnectionFailure:
    print("Server not available")

2> AutoReconnect exception

"When this exception is raised our application code needs to decide whether to retry the operation or to simply continue, accepting the fact that the operation might have failed.

On subsequent attempts to run the query we might continue to see this exception. Eventually, however, the replica set will failover and elect a new primary (this should take no more than a couple of seconds in general). At that point the driver will connect to the new primary and the operation will succeed:"

3> Secondary Reads

options of MongoClient parameters: Read Preference:

primary,
primaryPreferred,
secondary,
secondaryPreferred,
nearest.

Defaults to primary. To use secondaries for queries we have to change the read preference:

(bzuvenv) ubuntu@ip-172-31-8-191:~/tmp$ python
Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import MongoClient, ReadPreference
>>> uri = 'mongodb://{},{}/?replicaSet={}'.format('ip-172-31-4-46.us-west-2.compute.internal', 'ip-172-31-13-173.us-west-2.compute.internal', 'ft-mongo-replica-set')
>>> client = MongoClient(uri, read_preference=ReadPreference.PRIMARY_PREFERRED)
>>> client.read_preference
PrimaryPreferred(tag_sets=None, max_staleness=-1)
>>>

Secondary Reads

Reference
http://api.mongodb.com/python/current/api/pymongo/mongo_client.html

BruceZu · 2017-05-26T16:26:55Z

=================
on sshhop prepare python environment same as the test environment : 3.5

$ sudo apt install virtualenv
$ pip install --upgrade virtualenv
$ pip install --upgrade pip
$ sudo apt install python3-venv

Reference
http://api.mongodb.com/python/current/examples/index.html
http://api.mongodb.com/python/current/api/pymongo/mongo_client.html
https://github.com/mongodb/mongo-python-driver?jmp=docs
https://stackoverflow.com/questions/22035257/django-non-rel-connecting-to-multiple-hosts-in-a-replica-set

BruceZu · 2017-05-27T00:10:08Z

connect mongo arbiter in docker

$ docker exec -it forticloudplatform_mongo_1    mongo mongodb://forticloudplatform_mongo_arbiter_1

run python from local 'worker' Docker container

$ docker exec -it $(docker ps | grep worker | cut -d" " -f 1) python

find out all place where the code need update: configuration and code

(.venv) bzu@bruce-laptop:~/project/FortiCloudPlatform$ grep -Rr "MONGODB" .  --exclude-dir=".venv" --exclude-dir=".idea" --exclude-dir=".git"

BruceZu/gist:6054cc22ad8a298999013f3edee14330

BruceZu commented May 25, 2017

BruceZu commented May 25, 2017 •

edited

Loading

BruceZu commented May 26, 2017 •

edited

Loading

BruceZu commented May 26, 2017

BruceZu commented May 27, 2017 •

edited

Loading

BruceZu/gist:6054cc22ad8a298999013f3edee14330

BruceZu commented May 25, 2017

BruceZu commented May 25, 2017 • edited Loading

BruceZu commented May 26, 2017 • edited Loading

Using PyMongo 3.4. It supports MongoDB 3.4

Reference case for update Settings and application code

Case1: Access AWS MongoDB replica set from EC2 'sshhop'

Case2: Access local docker dev environment MongoDB replica set from local 'worker' Docker container

Application code need care

1> ConnectionFailure

2> AutoReconnect exception

3> Secondary Reads

BruceZu commented May 26, 2017

BruceZu commented May 27, 2017 • edited Loading

BruceZu commented May 25, 2017 •

edited

Loading

BruceZu commented May 26, 2017 •

edited

Loading

BruceZu commented May 27, 2017 •

edited

Loading