Renkai

Motivation

Currently, if the data in pulsar was offloaded to the second storage layer, data can still exists in bookkeeper for a period of time, but the client will directly read data from the second layer.

This may lead to several problems:

• Read from second layer have different performance characteristics, which may lead wrong estimate from users if they didn't know which layer they are reading.
• The second layer may be managed by another team rather than Pulsar management team(for example, a independent HDFS management team), they may have independent quota or authority policy to users.
• The second layer storage can be infinite in theory, if user set cursor to an error time in accident, it will cause a lot of resource waste.

Renkai

find ~/.m2/repository/ -atime +1 -iname '*.pom' -print0 | while read -d '' -r pom; do echo rm -rf "$(dirname $pom)"; done

Renkai

object TimedFlow {
  def apply[In, Out](innerFlow: Flow[In, Out, NotUsed], func: (Long, Long) => Any): Flow[In, Out, NotUsed] = {
    val flowWithLong = Flow.fromGraph(GraphDSL.create() {
      implicit builder =>
        import akka.stream.scaladsl.GraphDSL.Implicits._
        val unzip = builder.add(Unzip[In, Long]())
        val zip = builder.add(Zip[Out, Long]())

        unzip.out0 ~> innerFlow ~> zip.in0
        unzip.out1 ~> zip.in1

Renkai

import base64
import json
import sys

demo_obj = {
    'module_name': "spock_spore_engine.tasks.demo_task",
    'class_name': "DemoTask",
    'class_args': [],
    'class_kwargs': {

Renkai

import subprocess
import shlex
import json
import sys
from pathlib import Path
from numpy import fromfile, uint8  # pip install numpy

ROOT = Path(sys.argv[0]).resolve().parent
for dir in ROOT.iterdir():
    if not dir.name.startswith("22"):