Dr Nic Williams drnic

Orchestrating high-availability PostgreSQL for Cloud Foundry - Patroni, wal-e & Docker

PostgreSQL can be the only database you need - it offers SQL and NoSQL/JSON, it supports high-availability via synchronous or asynchronous replication to followers, and supports disaster recovery by storing backups and incremental write-ahead logs (WAL) offsite. But orchestrating all this is not easy; and can be a major engineering effort by DBAs or DevOps teams to build out and maintain. And that's just for one database. What if you want hundreds or thousands?

Fortunately in 2016 we have solutions. Patroni [1] is a major step forward in making it easy to setup a highly available PostgreSQL cluster. wal-e [2] provides a way to store base backups and progressive WAL updates offsite in AWS S3, Azure or similar. Docker [3] makes it easy to provision many isolated PostgreSQL services across a shared fleet of pre-provisioned machines.

Orchestrating high-availability PostgreSQL in 2016 - Patroni, wal-e & Docker

PostgreSQL can be the only database you need - it offers SQL and NoSQL/JSON, it supports high-availability via synchronous or asynchronous replication to followers, and supports disaster recovery by storing backups and incremental write-ahead logs (WAL) offsite. But orchestrating all this is not easy; and can be a major engineering effort by DBAs or DevOps teams to build out and maintain. And that's just for one database. What if you want hundreds or thousands?

Fortunately in 2016 we have solutions. Patroni [1] is a major step forward in making it easy to setup a highly available PostgreSQL cluster. wal-e [2] provides a way to store base backups and progressive WAL updates offsite in AWS S3, Azure or similar. Docker [3] makes it easy to provision many isolated PostgreSQL services across a shared fleet of pre-provisioned machines.

Currently a private broker can only be registered once in a single private space.

See https://blog.starkandwayne.com/2015/11/18/register-your-own-service-broker-with-any-cloud-foundry/

Proposal: allow arbitrary base path which will alter the service/plan UUIDs:

cf create-service-broker dingo-pg-production user pass https://dingo-pg.starkandwayne.com/space-scoped/production

Pivotal Partner Program Agreement

BY ELECTRONICALLY SIGNING THIS DOCUMENT, YOU ACCEPT AND AGREE TO BE BOUND BY THE TERMS AND CONDITIONS IN THIS AGREEMENT, THE PROGRAM RULES, ANY ADDENDA, AND PROGRAM SITE, ALL OF WHICH ARE INCORPORATED INTO AND FORM PART OF THIS AGREEMENT (COLLECTIVELY, THE "AGREEMENT"). THE AGREEMENT SHALL GOVERN YOUR ENROLLMENT AND PARTICIPATION IN THE PIVOTAL PARTNER PROGRAM). YOU REPRESENT THAT YOU HAVE READ AND UNDERSTAND ALL OF THE PROVISIONS OF THIS AGREEMENT. YOU MUST ACCEPT THIS AGREEMENT BEFORE YOU CAN PARTICIPATE IN THE PROGRAM. IF YOU ARE ENTERING INTO THIS AGREEMENT ON BEHALF OF A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT THAT YOU HAVE THE AUTHORITY TO BIND SUCH ENTITY TO THIS AGREEMENT, IN WHICH CASE THE TERMS "YOU," OR "YOUR," SHALL REFER TO SUCH ENTITY. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH THESE TERMS AND CONDITIONS, YOU MUST NOT SELECT SIGN AND MAY NOT ENROLL OR PARTICIPATE IN THE PROGRAM. THE "EFFECTIVE DATE" OF THIS AGREEMENT SHALL BE THE DATE O

Hieroglyph

"In Neoplatonism, especially during the Renaissance, a 'hieroglyph' was an artistic representation of an esoteric idea" - https://en.wikipedia.org/wiki/Hieroglyph

This Cloud Foundry CLI plugin, and Golang library, allows you to attach and retrieve data for a Cloud Foundry Service Instance, useful for non-core systems to attach data to a service instance (such as the URL for the latest backup of a postgresql or redis database).

As a CLI, first target the org/space:

	---
	name: bosh-lite
	director_uuid: f635b1ba-5000-48e1-92c8-19471d20e0e4

	releases:
	- name: bosh
	version: "256.2"
	url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=256.2
	sha1: ff2f4e16e02f66b31c595196052a809100cfd5a8
	- name: bosh-warden-cpi

	2016-04-15T11:06:31.39-0400 [APP/0] OUT org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
	2016-04-15T11:06:31.39-0400 [APP/0] OUT at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
	2016-04-15T11:06:31.39-0400 [APP/0] OUT at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	2016-04-15T11:06:31.39-0400 [APP/0] OUT at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
	2016-04-15T11:06:31.39-0400 [APP/0] OUT at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
	2016-04-15T11:06:31.39-0400 [APP/0] OUT at

	Copyright (c) 2016 GitHub Inc.

	Permission is hereby granted, free of charge, to any person obtaining
	a copy of this software and associated documentation files (the
	"Software"), to deal in the Software without restriction, including
	without limitation the rights to use, copy, modify, merge, publish,
	distribute, sublicense, and/or sell copies of the Software, and to
	permit persons to whom the Software is furnished to do so, subject to
	the following conditions:

	$ docker logs peter

	ttl: &ttl 30
	loop_wait: &loop_wait 10
	scope: &scope bI508YaO
	restapi:
	listen: 127.0.0.1:8008
	connect_address: 127.0.0.1:8008
	etcd:
	scope: *scope