Script for infra-monitoring-demo for Tinybird

README.md

Real-Time Infrastructure Monitoring with Tinybird

This demo shows you how to build an infrastructure monitoring tool, similar to a mini-datadog, that keeps track of some stats from all of the computers running this included script. Using this data, we can plot time series and build something based on "real" data being captured in the moment.

The script

This script essentially is a Python script that gathers some information (what's your OS - e.g: Windows, macOS or Ubuntu), how many CPUs you have, available memory and available disk and sends an event every second with it.

This is how the messages look:

{
    'timestamp': '2023-02-21 03:11:31.709',
    'nickname': 'Sancha',
    'team': 'Tinybird',
    'text': 'Good luck!',
    'os': 'Darwin',
    'cpus': 8,
    'mem_size': 34359738368,
    'mem_free': 15164309504,
    'disk_size': 494384795648,
    'disk_free': 112696381440
}

I'll build a time series in Tinybird that plots free memory over time per OS and/or number of CPUS, and a couple of endpoints that expose the data.

1. Save in a local folder

2. Go that folder in the terminal and run the following:

python3 -m venv .venv
source .venv/bin/activate
pip install pytz kafka-python psutil confluent-kafka

4. Configure the script

Open the index.py and customize lines 22 and 24:

your_nickname = 'Joe' #enter a nickname
message_to_tinybird = 'Good luck!' #Or say something funny, but keep it tidy... 8)

5. Run the script

In a terminal, run the following:

python3 index.py

Note: If you are running this on macOS, be sure to run the following to prevent the computer from sleeping:

caffeinate -s python3 index.py

If you are in Ubuntu, you probably know better than me how to prevent the computer from sleeping. If you are in Windows, your guess is as good as mine.

The script will sleep until an hour before the start of the demo. Just leave it running. Once the time of the event has past, the script will stop automatically.

Authors

• Jorge Sancha
• Joe Karlsson

---

Need help?: Community Slack • Tinybird Docs • Email

index.py

import time
from datetime import datetime
import pytz
import psutil
import platform
import json
from confluent_kafka import Producer
from kafka import KafkaProducer

def send_data_to_kafka():
    producer = KafkaProducer(
        bootstrap_servers=['pkc-l6wr6.europe-west2.gcp.confluent.cloud:9092'],
        sasl_mechanism='PLAIN',
        security_protocol='SASL_SSL',
        sasl_plain_username='UQR4M6LACV5WWD7E',
        sasl_plain_password='9w/zYWlOzMJePfyNleAfAYjgJ19UgqWDZe51/I9zs1KKsUGQ0skk+Su+CwQuM1rT',
        value_serializer=lambda x: json.dumps(x).encode('utf-8')
    )

    print('Sending events...')

    your_nickname = 'Joe Karlsson'  # enter a nickname
    # Say something funny, but keep it clean...
    message_to_tinybird = 'I love Kafka and Tinybird'

    TOPIC = 'infra-monitoring-demo'

    # Convert the start and end times from local time to ET
    ny_tz = pytz.timezone('America/New_York')

    # Send a message every second
    while True:
        # Get the current time in ET
        timestamp = ny_tz.localize(datetime.now()).strftime(
            '%Y-%m-%d %H:%M:%S.%f')[:-3]
        message = {}
        message['timestamp'] = timestamp
        message['nickname'] = your_nickname
        message['team'] = 'Tinybird'
        message['text'] = message_to_tinybird
        message['os'] = platform.system()
        message['cpus'] = psutil.cpu_count()
        mem = psutil.virtual_memory()
        message['mem_size'] = mem.total
        message['mem_free'] = mem.available
        disk = psutil.disk_usage('/')
        message['disk_size'] = disk.total
        message['disk_free'] = disk.free

        # Send the message to Kafka
        producer.send(TOPIC, value=message)

        # Wait for a few seconds before sending the next message
        print(message)
        time.sleep(1)


send_data_to_kafka()