Here are technical projects I contributed in both professional and personal settings. Also see resume.github.io. I would be delighted to connect with you on Twitter or LinkedIn.
Last updated on May 2023
As part of our push to the cloud, we need to automate data and software product delivery from the keyboard to production go-live, so that products are no longer manually packaged. I coordinate with the sales, client success, and QA teams, and with the CTO to deliver his vision on time. Design and kickstart implementation of this project from scratch. It is to be used internally and by financial institutions in 2023.
- High availability: minimum of three instances in an autoscaling group behind a load balancer.
- Data auditability: choose and introduce XTDB in the company. It is a bitemporal document store with Datalog as query language. Data are immutable, only changes are appended. This allows both backfilling (today I have a better knowledge of how the past should be) and time travel (what knowledge did we have of the world as of three months ago?).
- Data durability: use AWS Aurora with six instances in three availability zone.
- Multi-region deployment: the service is regional and can be deployed in each region independently of other regions. CICD jobs of different products are out of scope. Regional deployments may be for a single tenant (tier-one clients) or for multiple tenants.
- Disaster recovery: chose a pilot light plan where only data are replicated, and infrastructure can be automatically spawn in the DR region if need be. Aurora MySQL is used to provide managed cross-region replication.
- Tech stack: use Clojure / JVM / docker containers in EKS, and ClojureScript.
- API design: small REST API with reitit with ~20 routes.
- Frontend JS implementation using React, re-frame , reagent, figwheel.
- Observability: Datadog to receive logs and OpenTelemetry traces, derive metrics from logs, and real user monitoring, as well as for alerting. Locally exposed Prometheus, Grafana, Zipkin but didn't get them available in our infrastructure. Used flame graphs and VisualVM, circuit breaker.
- Tests: generative testing, headless unit tests in CI, and worked on back/front tests. Advocated for repeatable performance testing, fault tolerance (degraded > outage).
- Team player: helped other teams in their projects, maintained extensive written documentation, time-stamped architecture diagrams, pushed for logging, performance, and resilience best practices;
https://github.com/piotr-yuxuan/malli-cli
Out of the box command-line interface and configuration management from the comfort of a malli schema. Your configuration value is typed, validated and self-documented with a schema. The expected shape of your configuration being described as a malli schema so you can parse and decode strings as well as validating any constraints. It's quite powerful.
This library provides out of the box a configuration management system from different sources:
- Default configuration value;
- Environment variables when the program starts up (including nested data structures as strings);
- Value from some configuration other management system (including files);
- Command line arguments.
The return map can be used as a config fragment, or overrides, that you can later merge with the config value provided by any other system. As such it intends to play nicely with configuration tools, so the actual configuration value of your program is a map that is a graceful merge of several overlapping config fragment.
https://github.com/piotr-yuxuan/closeable-map
Application state management made simple: a Clojure map that implements java.io.Closeable. This small Clojure library
tries to get as close as possible to the bare essential complexity of state management in code. It defines a new type of
Clojure map that you may (.close m).
This library defines a new type of map that you may use like any other map. This map may contain stateful Java objects
like a server, a Kafka producer, a file output stream. When you want to clean your state, you just .close the map and
all nested stateful objects will be closed recursively. This library handle exceptions to make sure it doesn't stay in a
non-deterministic half-open, half-closed broken state.
Designed and implemented Audience API for audience data upload and fast queries, backed on DynamoDB, and now used in production for several markets: New Zealand, Australia, France, UK. Got thanked after I left the company as this API kept afloat under a load of 4k req/s – when nominal scenario was around 1k req/s.
- High availability: three instances behind a load balancer, deployed in Kubernetes.
- Data durability: used Optimus DB backed by AWS DynamoDB. Conceptually similar to LakeFS: you open a new version for a dataset, upload data. When you close it version becomes immutable. A dataset only have one published version at any given point in time.
- Tech stack: use Clojure / JVM / docker containers in EKS
- API design: small REST API with reitit with ~10 routes.
- Observability and testing: metrics (Prometheus), logs (Kibana), dashboards (Grafana), circuit breaker ( com.brunobonacci.safely), fault tolerance (Kafka retention period)
- Tests: performance/load testing with Gatling, code optimisation with criterium (conceptually similar to JMH
https://github.com/piotr-yuxuan/slava
https://github.com/piotr-yuxuan/walter-ci
https://github.com/piotr-yuxuan/ephemeral-server
https://github.com/piotr-yuxuan/algo-calisthenics
This is where I would put some of my training exercises when the problems looks peculiarly fun, and interesting to solve. Algorithmic questions asked in coding interviews will probably be similar to code here.
Nevertheless I shall add I'm always a bit disappointed by coding interviews where the job seeker is only given some riddles to solve quickly. It's a microscopic part of amy daily engineering work which ought to involve reasoning about the long-term technical vision to serve the business of the company, being cognizant about the platform observability, resilience, and robustness, and friendly rubbing shoulders with other team mates.
https://github.com/piotr-yuxuan/dove
Apache Avro is a way to serialize data in binary according to some schema which describes the data shapes from (nested) primitive types. I figured out it would be pretty useful to generate random sample of Avro messages corresponding to a schema. Using clojure.spec, test.check, and some Java interop, code turns out to be pretty concise.
This is an example of emerging design, exemplifying Eisenhower sentence: « plans are useless, but planning is indispensable ». I had a strong idea of what I need, I knew Clojure protocol would a suitable implementation choice, but I started from a white sheet and let the code click flow itself in the simplest way while I iteratively added new features and constraints.
Not the whole Apache Avro spec is implemented but I aim at making this side project useful for my main employer usecase.
Personal project: nabab, describe how data flow between core.async channels with a pinch of syntactic sugar (from Oct 2018)
https://github.com/piotr-yuxuan/nabab
- Early example of emerging design with a focus on simplicity and decomplection. I chose name Nabab in order to keep remembering me I don't want an abstraction above core.async, merely a side tooling.
- Example of litterate programming with Emacs.
- Inspiration from this project come from Prismatic plumbing graph, and timely dataflow.
Older source and website hosted on GitHub, more recent source and website hosted on GitLab.
- Strong accent on UX: explain interface behaviour in full, simple language so user can't be lost and understand as much as possible what he's doing.
- For the Greater Cause ;-) try to be useful for LineageOS, a floss Android
- Host, deploy from GitLab with GitLab CI
- Clojure, core.async, dataflow programming (something akin timely-dataflow in Rust), lein figwheel then shadow-cljs, React 16.1, Material UI
- Configure DNS, domain, deployment pipeline, and everything all from the ground up.
https://github.com/piotr-yuxuan/public-environment-configuration
Get a reproductible and declarative configuration of the programs I use for my daily life and document their use.
It's quite clear that any sharing of personal dot files contains a risk of leaking secrets or confidential data. To mitigate this risk I've copied a private repo, deleted its history, then committed a carefully current state. Obviouly it's pretty outdated now and I don't plan to update the public version any soon.
Hint: this repository makes heavy use of Emacs litterate programming capabilities (with org-mode and babel). Hence, please forgive broken links in the README. They would perfectly work when read in Emacs, but GitHub org-mode parser can't handle them properly.
https://piotr-yuxuan.github.io/1111101000-pieces/index.html
(The centipede was making a hundred-piece puzzle)
Let's be honnest and acknowledge it's not an very interesting project from a technical viewpoint. I worked with a friend of mine with is an artist. He was interested in studying the relation of human vision to disorder.
https://github.com/piotr-yuxuan/illegal-activity
Are you using the same laptop at home and at work? if you're company policies forbid usage of torrent software (even for legit, sharing reasons), how do you make sure you'll never be caught /by design/? At least you can prove you tried something to be compliant.
In this project I got a bit more acquainted to macOS architecture. Here I use a =plist= file to detect and react to some events.
https://github.com/piotr-yuxuan/polygam
Learning of logic programming with Clojure library core.logic. Given a tree of ancestors / descendants with some elicited vertices, what are all vertices that are elicited or descend from an elicited one?
Creation and integration of a hiking map layers and tracks for a 1500-km trail throughout France and Spain.
https://github.com/piotr-yuxuan/PoC
Learning of logic programming with Clojure library core.logic, deepening my understanding of this language. PGF notation file parsing and discovery available legal moves. However, nothing to be compared to Alpha Zero ;-) Here I do nothing against combinatorial explosion so you can't actually predict all possible three-move-far positions because it would boil the ocean.
https://github.com/piotr-yuxuan/chinese-huawen
- Functional programming with Clojure (JVM-based language)
- Formal grammars with instaparse.
This project is aimed at analysis of the sinogram set with graph and set theories. What’s the best way to learn and remember Chinese characters? Most of Chinese handbooks are built in a Western fashion: daily-life conversation samples bring new vocabulary and syntax examples. This fashion doesn’t take advantage of intrinsic Chinese language structure, which makes easier to discover 貴 when knowing yet中, 一 and 貝. 貴 stands for expensive or honourable while 貝 refers to shell or money. This semantic link lets it easy to remember both of them.
The first character 華 of the project name means sinity; 文 stands for language. The whole phrase 華文 means Chinese language and is readㄏㄨㄚˊ ㄨㄣˊ or huáwén.
This site has been in some extent redacted in French because part of this project is being developped whithin the frame of a school project.