Skip to content

Instantly share code, notes, and snippets.

@ilaydadastan

ilaydadastan/gsoc24-jitsi-log-analyser.md

Last active October 25, 2024 08:49
Show Gist options
  • Save ilaydadastan/1ed05b8e39bcc5a80d33bf8e6ed90aed to your computer and use it in GitHub Desktop.
Save ilaydadastan/1ed05b8e39bcc5a80d33bf8e6ed90aed to your computer and use it in GitHub Desktop.
Download ZIP
This file is for the final submission of the Jitsi Meet Log Analyser project within the scope of Google Summer of Code 2024.
Raw
gsoc24-jitsi-log-analyser.md

Google Summer of Code 2024 Final Submission - Jitsi Meet Log Analyser

Organization: Jitsi

Project Name: Log Analyser for Jitsi Meet using Grafana, Loki, and OpenTelemetry

Mentors: Aaron van Meerten Boris Grozev Lucian-Paul Torje

Contributor: Ilayda Nur Dastan

Introduction

During the Google Summer of Code 2024 program, I worked with Jitsi to develop a log analysis solution for Jitsi Meet. This report details the achievements of the project, which aimed to integrate Grafana, Loki, and OpenTelemetry with Jitsi Meet’s Docker components to facilitate efficient log management and visualization.

Project Overview

The primary objective of this project was to enhance the observability and monitoring of Jitsi Meet components by developing a comprehensive log analysis system. This system uses Grafana, Loki, and OpenTelemetry to improve log management and provide actionable insights into system performance and health. Key goals included:

  • Automatic Collection: Aggregating logs from Docker components running Jitsi Meet to ensure comprehensive log data collection.
  • Visualization: Creating interactive Grafana dashboards to visualize and analyze logs, enabling efficient monitoring and troubleshooting.
  • Configuration: Implementing OpenTelemetry for standardized log collection and parsing, streamlining data flow into Loki.
  • Integration: Ensuring seamless operation between Grafana, Loki, and OpenTelemetry for real-time log querying and visualization.
  • Documentation: Providing clear setup instructions and usage guidelines to facilitate deployment and effective use of the log analysis system.

This project aimed to build a robust log analysis framework to simplify log management and improve monitoring capabilities for Jitsi Meet deployments.

Demo

To better understand how the log analysis and visualization work, please watch the demo video below:

jitsi-log-analyser.mov

Pull Requests and Contributions

During the project, I submitted several pull requests to the Jitsi Docker repository to enhance log analysis capabilities. Most of these PRs have been merged, while a few are expected to be merged soon:

Pull Request Description
PR #1844 Added Grafana, Loki, and OpenTelemetry integration for log analysis
PR #1855 Added Grafana dashboard provisioning
PR #1874 Added Jitsi network to Loki service
PR #1884 Transferred Docker logs from containers and updated README file
PR #1885 Updated Grafana dashboards
PR #518 Enhanced the Jitsi handbook with log analysis guidelines

These contributions were crucial in setting up the log analysis infrastructure and ensuring it met the project requirements.

Technical Architecture

Components

The log analysis solution for Jitsi Meet integrates key components to streamline log management. Docker Compose coordinates the deployment of Jitsi Meet with log management tools. Loki aggregates logs from Jitsi Meet services, while Grafana visualizes these logs through pre-configured dashboards. OpenTelemetry handles log collection and parsing, ensuring proper formatting and transmission. Together, these components work seamlessly to enhance log monitoring and analysis.

  1. Docker Compose:

    • docker-compose.yml: Configures the Jitsi Meet components.
    • log-analyser.yml: Sets up Loki and the OpenTelemetry Collector for log management.
    • grafana.yml: Deploys Grafana for visualizing log data.

  2. Grafana:

    • Used for visualizing logs collected by Loki.
    • Pre-configured dashboards are provided for ease of use.

  3. Loki:

    • A log aggregation system that collects logs from Jitsi Meet components.
    • Integrates with Grafana for querying and visualization.

  4. OpenTelemetry:

    • Facilitates the collection and parsing of logs from Jitsi Meet components.

Setup Instructions

  1. Clone the Repository:

    git clone https://github.com/jitsi/docker-jitsi-meet.git

  2. Start the Services:

    docker-compose -f docker-compose.yml -f log-analyser.yml -f grafana.yml up -d
    • This command starts Jitsi Meet components, Loki, OpenTelemetry, and Grafana. Logs from Jitsi Meet components are automatically forwarded to Grafana through Loki.

  3. Access Grafana:

    • Open your web browser and navigate to http://localhost:3000.

    • Log in with:

      • Username: admin
      • Password: admin

    • Explore the pre-configured dashboards to analyze logs.

Documentation

As part of the project, detailed documentation was created to ensure proper setup and usage of the log analysis system. This includes:

  • Detailed README for Log-Analyser: A comprehensive README file was added to the log-analyser directory. This document provides step-by-step instructions for setting up and configuring the log analysis components, including Docker Compose files and service configurations. It serves as a practical guide for users to deploy and manage the log analysis system effectively.

  • Jitsi Handbook: An extensive guide on using the log analysis system was also incorporated into the Jitsi Handbook. This document includes detailed explanations on configuring and utilizing the log analysis tools, ensuring that users can fully leverage the new capabilities provided by Grafana, Loki, and OpenTelemetry.

These resources provide essential information for both setting up the system and understanding its functionality, contributing to a smoother implementation and better user experience.

Usage

Once the setup is complete, you can fully leverage the log analysis system to gain valuable insights into the performance and behavior of Jitsi Meet components:

  • Visualize and Analyse Logs: Users can now easily analyze logs from various Jitsi Meet components, such as Jicofo, Jitsi Videobridge (JVB), Prosody, and the web frontend. The visual dashboards enable users to filter and explore logs, making it easier to troubleshoot issues, monitor performance, and ensure system health. With parsed log fields, users can quickly filter logs, simplifying the process of identifying and resolving problems.

  • Utilize Pre-configured Dashboards: To streamline the log analysis process, several pre-configured dashboards are available. These dashboards are specifically designed to highlight key metrics and logs from Jitsi components like Jicofo, JVB, Prosody, and the Web. With these dashboards, you can quickly access critical information, monitor system performance, and identify potential issues without needing to set up custom queries.

Pre-configured Dashboards

Grafana includes several pre-configured dashboards for monitoring Jitsi Meet components:

  • Jicofo Dashboard: Visualizes logs related to Jitsi Conference Focus (Jicofo).
  • JVB Dashboard: Focuses on Jitsi Videobridge (JVB) logs.
  • Prosody Dashboard: Monitors the Prosody XMPP server.
  • Web Dashboard: Displays logs related to the web frontend.
  • Jitsi All Dashboard: Aggregates logs from all Jitsi Meet components.
Screenshot 2024-08-21 at 23 39 57

Challenges Encountered During the Project

Throughout the project, I faced several challenges that required careful consideration and problem-solving:

  1. Setting Up Direct Log Collection from Docker Containers: One of the most significant challenges was configuring the system to directly collect logs from Docker containers. This involved ensuring that the logging drivers and configurations were correctly set up to forward logs efficiently without causing performance bottlenecks or data loss.

  2. Creating Effective Visualizations in Grafana Using LogQL: Developing meaningful visualizations in Grafana required a deep understanding of LogQL, the query language used by Loki. Crafting the right queries to extract and visualize relevant log data was challenging, as it demanded precise filtering, aggregation, and transformation of logs.

  3. Writing Accurate Regex Patterns for Parsing Logs: Parsing logs from different Jitsi components was another complex task. Each component had distinct log structures, requiring the creation of custom regular expressions (regex) for each. Ensuring that these regex patterns accurately captured the relevant log data without errors was a time-consuming and intricate process.

What I Learned from the Project

This project provided an opportunity to develop a broad range of skills and insights:

  1. Improved Docker Usage and Docker Logging Best Practices: Through this project, I have become more proficient in using Docker, particularly in the context of log management and containerized environments. I also gained extensive knowledge of Docker's logging capabilities, including the setup of logging drivers and the nuances of log forwarding from containers. This experience has equipped me with the skills to manage logs efficiently in containerized environments.

  2. Learning New Technologies: I was introduced to and gained practical experience with new technologies like OpenTelemetry and Grafana Loki, both of which are now valuable tools in my skill set for observability and log management.

  3. Learning LogQL for Log Analysis: Working with LogQL in Grafana was a key learning experience. I now have a strong understanding of how to construct complex queries to filter, aggregate, and visualize logs effectively, which will be invaluable for future log analysis tasks.

  4. Better Regular Expression (Regex) Writing: Developing regex patterns to parse logs from different components honed my ability to write and optimize regex. This skill is particularly useful for extracting specific data points from unstructured log files, a common task in log analysis.

  5. Business and Open Source Development Insights: I gained a deeper understanding of the value of effective log management for maintaining and optimizing large-scale applications. I also learned the importance of collaboration and community feedback in open source projects, enhancing both my technical skills and my ability to work in a team-oriented environment.

Future Work

While the project has achieved its primary objectives, future improvements include:

  • Enhance Dashboard Features: Add more visualizations based on feedback.
  • Expand Log Parsing: Parsing logs into more fields and improving regex patterns

Thank you for reviewing this report. Completing this project has been an incredibly rewarding experience. It not only allowed me to contribute to the Jitsi community but also provided invaluable learning opportunities and personal growth. I am grateful for the support and guidance from my mentors, as well as the entire Jitsi team. Their insights and encouragement were crucial to the success of this project. I also appreciate the Google Summer of Code program for providing this opportunity. This journey has significantly enhanced my skills and deepened my understanding of log analysis and visualization. Thank you all for your support and for being part of this amazing experience.

@shabbirahmad52
Copy link

shabbirahmad52 commented Oct 24, 2024 •
edited
Loading

Hello @ilaydadastan,
I have followed all this documentation and I am using jitsi meet in ocker and all containers are up and running but i can see no logs on grafana
When i do docker compose logs -f otel this is what i am seeing

2024-10-24T08:16:18.557Z    info    fileconsumer/file.go:265    Started watching file    {"kind": "receiver", "name": "filelog/jitsi-containers", "data_type": "logs", "component": "fileconsumer", "path": "/var/lib/docker/containers/a8c685131fae3fc61d66d14cf0c1720a788a2ad332b3818f01a3daf8ecdde8ac/a8c685131fae3fc61d66d14cf0c1720a788a2ad332b3818f01a3daf8ecdde8ac-json.log"}
2024-10-24T08:16:18.557Z    info    fileconsumer/file.go:265    Started watching file    {"kind": "receiver", "name": "filelog/jitsi-containers", "data_type": "logs", "component": "fileconsumer", "path": "/var/lib/docker/containers/b17c61871d7b173a5418ec2e4e6f6c373a11fae08c7a17a33e7964babd0b4250/b17c61871d7b173a5418ec2e4e6f6c373a11fae08c7a17a33e7964babd0b4250-json.log"}
2024-10-24T08:16:18.557Z    info    fileconsumer/file.go:265    Started watching file    {"kind": "receiver", "name": "filelog/jitsi-containers", "data_type": "logs", "component": "fileconsumer", "path": "/var/lib/docker/containers/cc7b29a4dda77206ba428cf4cbcb378ca5ee8d9b9421fee3d494a54430f4306b/cc7b29a4dda77206ba428cf4cbcb378ca5ee8d9b9421fee3d494a54430f4306b-json.log"}
2024-10-24T08:16:18.558Z    info    fileconsumer/file.go:265    Started watching file    {"kind": "receiver", "name": "filelog/jitsi-containers", "data_type": "logs", "component": "fileconsumer", "path": "/var/lib/docker/containers/cc86cc71e8187aed74fa300d12bd35c2029d43e8d18637b6341c79045540b9c3/cc86cc71e8187aed74fa300d12bd35c2029d43e8d18637b6341c79045540b9c3-json.log"}




2024-10-24T08:16:39.758Z    error    helper/transformer.go:100    Failed to process entry    {"kind": "receiver", "name": "filelog/jitsi-containers", "data_type": "logs", "operator_id": "jitsi_jvb_parser", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send"}
[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError](http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError)

Can you please help me fix this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment