Experiment having an LLM summarize Atlas logs

llm-log-analysis.py

import json
import pymongo
import pandas as pd
import params
import openai 

MONGODB_URI = params.MONGODB_URI
DATABASE_NAME = params.DATABASE_NAME
COLLECTION_NAME = params.COLLECTION_NAME
OPENAI_API_KEY = params.OPENAI_API_KEY
OPENAI_TYPE = params.OPENAI_TYPE
OPENAI_API_VERSION = params.OPENAI_API_VERSION
OPENAI_AZURE_ENDPOINT = params.OPENAI_AZURE_ENDPOINT
OPENAI_AZURE_DEPLOYMENT = params.OPENAI_AZURE_DEPLOYMENT

mongodb_client = pymongo.MongoClient(MONGODB_URI)  
azure_client = openai.AzureOpenAI(
    azure_endpoint=OPENAI_AZURE_ENDPOINT,
    api_key=OPENAI_API_KEY,
    api_version=OPENAI_API_VERSION
)


# Specify the log to get, for example, 'global'  
response = mongodb_client.admin.command('getLog', 'global')  
log_json = []
# Check for errors  
if response.get("ok") != 1.0:  
    print(f"Error: {response.get('errmsg')}")  
else:  
    # Access total lines written and log data  
    total_lines = response.get("totalLinesWritten")  
    log_data = response.get("log")
    # Print summary  
    print(f"Total lines: {total_lines}")  
    # Print log entries
    for i,entry in enumerate(log_data):
        json_entry = json.loads(entry)
        print(json_entry)
        log_json.append(json_entry)
logs_df = pd.DataFrame(log_json)
print(logs_df.to_csv())
# For all possible arguments see https://platform.openai.com/docs/api-reference/chat-completions/create
response = azure_client.chat.completions.create(
    model=OPENAI_AZURE_DEPLOYMENT,
    messages=[
        {"role": "system", "content": "You are a MongoDB expert with 10 years of experience."},
        {"role": "system", "content": "Your job is to help the user understand their MongoDB logs."},
        {"role": "user", "content": f"""
            Using the below context, answer the question further below:
            [csv context]
            {logs_df.to_csv()[:5000]}
            [end csv context]

            [question]
                What has happened based on the logs?
            [end question]

            [response outline]
                - Summary
                - Warnings
                - Errors
                - Observations
                - Suggested Action
            [end response outline]

            THINK CAREFULLY AND STEP BY STEP. 
            ONLY FORMULATE RESPONSES BASED ON THE CONTEXT PROVIDED. 
            DO NOT MAKE UP ANY PART OF THE ANSWER.
            
"""},
    ],
    temperature=0,
)

print(f"{response.choices[0].message.role}: {response.choices[0].message.content}")

"""
assistant: Based on the provided logs, here is the analysis:

Summary:
- Two connections were ended.
- One connection was accepted.
- An ingress TLS handshake was completed.
- Two slow queries were executed.
- An operation was interrupted due to client disconnection.

Warnings:
- None.

Errors:
- None.

Observations:
- The connections were made from remote IP addresses: '34.XXX.1X2.XXX:34290' and '1X2.XXX.24X.1X:38636'.
- The client metadata shows that the client is using the 'mongo-go-driver' version 'v1.9.0-cloud'.
- The slow queries were executed on the 'X.abc_demo' collection.
- The first slow query had a duration of 100 milliseconds and examined 12,863 documents.
- The second slow query had a duration of 121 milliseconds and examined 12,863 documents.

Suggested Action:
- Monitor the slow queries and investigate if any optimizations can be made to improve their performance.
- Check the network connectivity and stability for the connections that were ended and accepted.
- Investigate the reason for the interrupted operation and ensure that the client is not experiencing any connectivity issues.

Based on the logs, it appears that there were some slow queries executed on the 'X.abc_demo' collection. 
These queries took some time to execute and examined a significant number of documents. 
It is recommended to monitor these slow queries and optimize them if possible to improve performance. 
Additionally, the connections that were ended and accepted should be checked for any network connectivity issues. 
Finally, the interrupted operation due to client disconnection should be investigated to ensure a stable connection.
"""

exit()