xtrmstep

Custom Logs in GCP Logging

Logs can be sent to GCP logging very simple.

Basic Usage

You need to install the package google-cloud-logging.

Then the code will looks somehting like this:

xtrmstep

I accumulate here the list of articles which have interesting information about Apache Airflow. Subjects of the articles covers topics from installation and management to implementation of DAGs and best practices.

Best Practices

• Airflow: Lesser Known Tips, Tricks, and Best Practises

Comparison

• Airbnb’s Airflow Versus Spotify’s Luigi
• Airflow vs. Luigi vs. Argo vs. MLFlow vs. KubeFlow

xtrmstep

Here I gathered answers to small pitfalls which from time to time appears because you’re not doing some actions often. Such actions include using of .dockerignore, mounting files, SSL certificates for web servers in containers, etc.

Building of Docker images

1. The file want be applied if it’s not located in the same directory where you run docker build command.
2. Building of an image will fail if you use specific file names which you are ignoring. For example, if you write in Dockerfile COPY filetoignore.txt . and you are ignoring such file(s) in .dockerignore, you’ll get an error
3. Paths inside Dockerfile are relative to the folder where you’re running command docker build and not to the one where your Dockerfile is located.
4. Relative paths should use only current or child folders, since the parent one (or through it) is out of Docker context during building.

Docker-compose

xtrmstep

• Git: Ignore changes to already-tracked files
• How do I configure git to ignore some files locally?

xtrmstep

A glance under the hood to what happens in memory when you do operations upon Pandas DataFrame and its columns. In monitoring I’ll use the library memory_profiler and use it in Jupyter Notebook as described here.

Since the module is using Python scripts its usage a little bit wired. The general approach is to create a method which I want to profile and execute it by means of magic code %mprun.

xtrmstep

Sparse checkout

From scratch (enable)
1) $ git init
2) $ git remote add -f origin URL
3) $ git config core.sparseCheckout true
4) create a file with exclusion rules: /.git/info/sparse-checkout
5) $ git pull origin master

On already cloned folder

xtrmstep

import os
import sys

#  it's required to work in Airflow with packages
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.utils.trigger_rule import TriggerRule
from airflow.operators.dummy import DummyOperator

xtrmstep

public class BulletTrainRemoteConfiguration : IRemoteConfiguration
{
    private readonly string _userIdentity;
    private static readonly object LockObject = new object();        

    public BulletTrainRemoteConfiguration(RemoteConfigurationSettings settings)
    {
        if (BulletTrainClient.instance == null)
            lock (LockObject)
                if (BulletTrainClient.instance == null)

xtrmstep

// main interface
public interface IRemoteConfiguration
{
    Task<List<Feature>> GetFeatures();
    Task<List<Setting>> GetSettings(List<string> keys = null);
    Task<bool> IsFeatureEnabled(string key);
    Task<string> GetSettingValue(string key);
}
// other classes
public abstract class RemoteConfigEntry

xtrmstep

version: '2'
services:
  zookeeper1:
    restart: always
    image: wurstmeister/zookeeper:latest
    container_name: zookeeper1
    ports:
      - "2181:2181"
      
  kafka1: