kinow

Apache Jena project is like a box full of interesting things—at least if you love programming. One of its many features, is stream processing.

It may contain very large datasets, with gigabytes of data about graphs. Some queries may be quite large, so sending the whole result would be simply impracticable.

Instead, the data will go through ARQ. ARQ is a query engine for Jena that supports Sparql. There is one piece of code there that I found interesting while reviewing a small pull request: org.apache.jena.riot.system.StreamRDFWriter.

It is responsible for writing graph data in a streaming fashion. (See stream processing for programming models and more.)

Stream factories

kinow

Running a simple cyclic workflow with Cylc Flow and with StackStorm st2

First we need a simple workflow.

[scheduling]
    initial cycle point = 20130808T00
    final cycle point = 20130809T00
    [[graph]]
 R1 = "prep => foo"

kinow

Avoiding sudo with JupyterHub

Goal: avoid running sudo to spawn Cylc UI Server, while still allowing the JupyterHub to run with one user, while the Cylc UI Server applications are spawned individually running with the user accounts.

The SystemdSpawner can spawn applications with the user account. But - besides requiring Systemd - the JupyterHub and systemd-run need to be both executed as root. If these constraints are OK, then this spawner is probably a solution.

kinow

Make Tree data flat again

See subscription-reply.json for an example that includes families, cycle points, tasks, jobs, etc.

See subscription-reply.json for an example of what the InfiniteTree flat tree looks like (i.e. GraphQL --> JS code to add hierarchy as in Cylc 7 --> flattree).

NOTE: open Vue Dev Tools, click on the Workflow component, search for the Tree

kinow

Document created for Cylc meetup in NIWA Wellington on February 2020.

• Docker
  • Cylc Flow
    • Running the container
    • Volumes
  • Cylc UI Server
    • Running Cylc UI Server in a Docker container
    • Running a workflow inside the Cylc UI Server container
• Security

kinow

Document created for Cylc meetup in NIWA Wellington on February 2020.

Cylc 8 is mainly Python for the backend, and JS for the frontend. I use PyCharm when working on the backend, and WebStorm when on the frontend.

Here's a useful page on debugging & profiling that mentions pdb too, a command line debugger.

And the following will describe how I debug Cylc with PyCharm and WebStorm, two graphical IDE's.

• Python

kinow

Preparing the environment

The branch used for this test was this one.

After checking it out, and installing everything in a virtual environment, you will need pip and Docker (or have a running Postgres server).

# install Postgres Python driver/module

kinow

const express = require('express')
const bodyParser = require('body-parser')
const { graphqlExpress, graphiqlExpress } = require('apollo-server-express')
const { makeExecutableSchema } = require('graphql-tools')
const { PubSub } = require('graphql-subscriptions')
const { createServer } = require('http')
const { SubscriptionServer } = require('subscriptions-transport-ws')
const { execute, subscribe } = require('graphql')

const statuses = [

kinow

final ConsoleHandler consoleHandler = new ConsoleHandler();
consoleHandler.setLevel(Level.FINEST);
consoleHandler.setFormatter(new SimpleFormatter());
Logger logger = Logger.getLogger("");
logger.setLevel(Level.ALL);
logger.addHandler(consoleHandler);

kinow

[...(document.getElementById("stories").children)]
  .filter(e => e.tagName.toLowerCase() === 'div')
  .filter((e) => {
    const images = [...(e.querySelectorAll('img'))];
    const audioIcon = images.filter((i) => i.title.toLowerCase().includes('audio'))
    return audioIcon.length === 0
  })
  .map((e) => e.remove())
;