mdfranz

Telegraf → Logfire Metrics: Final Design and Relay Investigation

Decision

Use Telegraf as the collector, transformer, and OTLP exporter. Send metrics directly to Logfire:

Telegraf inputs
 │

mdfranz

SELECT table,
    formatReadableSize(sum(bytes)) as size,
    min(min_date) as min_date,
    max(max_date) as max_date
    FROM system.parts
    WHERE active
GROUP BY table

mdfranz

base.yaml

---
- hosts: all 
  tasks:
    - name: Set authorized key taken from file
      authorized_key:
        user: "{{ansible_user_id}}"
        state: present
        key: "{{ lookup('file', '/home/mfranz/.ssh/id_rsa.pub') }}"

mdfranz

data_dir: "/var/lib/vector"
sources:
    journald:
        type: journald

transforms:
    drop_xino_events:
        type: remap
        inputs:
            - journald

mdfranz

Refs

• See https://clickhouse.com/docs/en/operations/utilities/clickhouse-local

Conn.log

mfranz@opti3070:~/zeek/2024-12-14$ clickhouse local -q "select count(*) as cnt, id.orig_h from 'conn.*.log' group by id.orig_h order by cnt desc limit 50"
111901  192.168.2.167
40425   192.168.3.109
35433   192.168.3.135

mdfranz

The Data

Records from https://ja4db.com/

D describe jadb;
┌────────────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│      column_name       │ column_type │  null   │   key   │ default │  extra  │
│        varchar         │   varchar   │ varchar │ varchar │ varchar │ varchar │
├────────────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤

mdfranz

#!/bin/bash

# Get Credentials from Resources
AWS_9431=$(curl -s -H "Authorization: Bearer $WM_TOKEN" \
  "$BASE_INTERNAL_URL/api/w/$WM_WORKSPACE/resources/get_value_interpolated/u/mdfranz/aws_9431" | jq)

export ACCESS_KEY=`echo $AWS_9431 | jq .awsAccessKeyId`
export SECRET_KEY=`echo $AWS_9431 | jq .awsSecretAccessKey`

# Download S3M

mdfranz

helm install falco falcosecurity/falco \
    --version 4.11.1 \
    --namespace falco \
    --create-namespace \
    --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true \
    --set collectors.kubernetes.enabled=true \
    --set tty=true \
 --set-json 'falco.append_output=[{"match": {"source": "syscall"},"extra_output": "pod_uid=%k8smeta.pod.uid, pod_name=%k8smeta.pod.name, namespace_name=%k8smeta.ns.name"}]'

mdfranz

#!/usr/bin/env python3

import sys, pyshark

cap = pyshark.FileCapture(sys.argv[1])

for e in cap:
    if 'IP' in e:
        if 'TCP' in e:
            print (e['eth'].src,e['ip'].src,e['eth'].dst,e['ip'].dst)

mdfranz

hub:
  revisionHistoryLimit:
  config:
    Authenticator:
      enable_auth_state: true
      allowed_users:
        - user1
        - user2
        - user3
        - notadmin