Skip to content

Instantly share code, notes, and snippets.

@lavantien

lavantien/Onboarding to a Large Go Monorepo.md

Created February 5, 2025 17:02
Show Gist options
  • Save lavantien/bea0094e557e78bd37bb328ee2150593 to your computer and use it in GitHub Desktop.
Save lavantien/bea0094e557e78bd37bb328ee2150593 to your computer and use it in GitHub Desktop.
Download ZIP
Onboarding to a Large Go Monorepo: A LLM-Assisted Learning Plan
Raw
Onboarding to a Large Go Monorepo.md

Onboarding to a Large Go Monorepo: A LLM-Assisted Learning Plan

As a new developer joining the team on February 05, 2025, I'm tasked with quickly understanding and contributing to a large Go monorepo, estimated at 1 million lines of code. This presents a significant onboarding challenge. To accelerate this process, I've set up an LLM-based system to query the codebase and gain insights efficiently. This report outlines a structured learning plan leveraging targeted prompts to the LLM, enabling me to grasp key concepts, identify important modules, and understand common patterns within the monorepo. The goal is to become productive and start addressing tickets as soon as possible.

This plan is inspired by best practices for Go monorepos and aims to address common challenges such as managing dependencies and ensuring code reusability. The prompts are designed to extract information about the project structure, identify core packages, and understand the relationships between different modules. Furthermore, the plan incorporates techniques for prompt engineering to optimize the LLM's output for specific tasks like code generation, debugging, and test case generation. By systematically querying the codebase with these prompts, I aim to gain a comprehensive understanding of the monorepo and contribute effectively to the team's goals.

Table of Contents

  • Understanding the Monorepo Structure and Core Packages
      1. Identifying the High-Level Directory Structure
      1. Discovering Core Packages and Their Dependencies
      1. Analyzing Package Functionality and Key Data Structures
      1. Identifying Common Design Patterns and Idioms
      1. Understanding the Build and Deployment Process
  • Identifying Key Modules and Dependencies
    • Leveraging LLMs for Dependency Visualization
    • Identifying Cross-Cutting Concerns and Shared Libraries
    • Analyzing the Impact of Dependency Updates
    • Identifying God Classes and Anti-Patterns
    • Understanding Module Boundaries and Cohesion
  • Exploring Testing Strategies and Configuration Management
    • Understanding Testing Frameworks and Methodologies
    • Analyzing Test Coverage and Quality
    • Investigating CI/CD Pipeline and Testing Automation
    • Understanding Configuration Management Strategies
    • Identifying Common Testing Patterns and Anti-Patterns

Understanding the Monorepo Structure and Core Packages

Navigating a large Go monorepo with a million lines of code can be daunting for a new developer. Leveraging LLM-based queries can significantly accelerate the onboarding process. This report outlines a structured approach to understanding the monorepo's structure and identifying core packages using LLM prompts.

1. Identifying the High-Level Directory Structure

The first step is to understand the overall organization of the monorepo. This involves identifying top-level directories and their intended purpose.

LLM Prompt: "List the top-level directories in the monorepo and provide a brief description of the purpose of each directory. Focus on directories that appear to contain Go code."

This prompt aims to get a bird's-eye view of the codebase. Common directory structures in Go monorepos include:

  • cmd: Contains the main applications or services within the monorepo. Each subdirectory typically represents a separate executable.
  • internal: Holds internal packages that are not intended for external use. This enforces encapsulation and prevents accidental dependencies.
  • pkg: Contains reusable libraries or packages that can be used by other projects within or outside the monorepo.
  • services: Similar to cmd, but may contain more complex services with multiple components.
  • libs: Another common name for shared libraries.
  • api: Defines API contracts, often using Protocol Buffers or gRPC.
  • tools: Utility scripts and tools used for development, testing, or deployment.
  • docs: Documentation for the monorepo and its components.
  • examples: Example usage of libraries and services.

Understanding this high-level structure is crucial for navigating the codebase effectively. According to slaptijack.com, a well-structured monorepo should reflect the package-centric design of Go, making it intuitive and straightforward to navigate.

2. Discovering Core Packages and Their Dependencies

Identifying core packages and their dependencies is essential for understanding the key functionalities of the monorepo.

LLM Prompt: "Identify the most important or frequently used Go packages in the monorepo. For each package, list its direct dependencies (packages it imports) and any packages that depend on it (packages that import it). Rank the packages by the number of other packages that depend on them."

This prompt helps to identify the central components of the system. The LLM should be able to analyze import statements across the codebase to determine dependencies. The ranking by the number of dependents helps prioritize learning efforts.

For example, a core package might be responsible for:

  • Database interaction (e.g., internal/db)
  • Authentication and authorization (e.g., internal/auth)
  • Message queue integration (e.g., pkg/queue)
  • API handling (e.g., api/server)

Understanding the dependencies of these core packages reveals the relationships between different parts of the system. As mentioned in the context, tools like go-mono (GitHub - chrusty/go-mono) analyze dependencies to determine which services need to be rebuilt after a change.

3. Analyzing Package Functionality and Key Data Structures

Once core packages are identified, the next step is to understand their functionality and the key data structures they use.

LLM Prompt: "For the following Go package [package path], provide a summary of its main functionality. Identify the key data structures (structs, interfaces) defined in the package and explain their purpose. Provide examples of how these data structures are used within the package."

This prompt focuses on understanding the internal workings of a specific package. The LLM should be able to analyze the code and identify the key components.

For example, if the package is internal/auth, the LLM might identify the following:

  • Functionality: Provides authentication and authorization services for the monorepo.
  • Key Data Structures:
    • User: Represents a user account with attributes like ID, username, and password.
    • Session: Represents a user session with attributes like user ID, session token, and expiration time.
    • Authenticator: An interface that defines methods for authenticating users and managing sessions.
  • Usage Examples: The User struct is used to store user information in the database. The Session struct is used to track active user sessions. The Authenticator interface is implemented by different authentication providers (e.g., local authentication, OAuth).

By understanding the functionality and data structures of core packages, the new developer can gain a deeper understanding of the system's architecture.

4. Identifying Common Design Patterns and Idioms

Large codebases often employ common design patterns and idioms to improve maintainability and readability. Identifying these patterns can help the new developer understand the code more quickly.

LLM Prompt: "Identify any common design patterns or idioms used in the monorepo. Provide examples of where these patterns are used and explain their benefits in this context. Focus on patterns specific to Go, such as interfaces, error handling, and concurrency."

This prompt aims to uncover the underlying design principles of the codebase. Common Go patterns include:

  • Interfaces: Used extensively for abstraction and dependency injection.
  • Error Handling: Go's explicit error handling is often implemented consistently throughout the codebase.
  • Concurrency: Go's goroutines and channels are used for concurrent operations.
  • Functional Options: A pattern for configuring structs with optional parameters.
  • Context: Used for managing request-scoped data and cancellation signals.

For example, the LLM might identify that the monorepo uses the "functional options" pattern for configuring database connections. It would then provide an example of how this pattern is used in the internal/db package and explain the benefits of using this pattern (e.g., improved readability, flexibility).

5. Understanding the Build and Deployment Process

Understanding how the monorepo is built and deployed is crucial for making changes and contributing to the project.

LLM Prompt: "Describe the build and deployment process for the monorepo. Identify the tools and technologies used for building, testing, and deploying the code. Explain how changes are integrated and released. Focus on any monorepo-specific tooling or configurations."

This prompt aims to provide an overview of the CI/CD pipeline. The LLM should be able to analyze build scripts, configuration files, and deployment manifests to understand the process.

Key aspects of the build and deployment process include:

  • Build System: Tools like make, bazel, or go mod are used to build the code.
  • Testing Framework: Go's built-in testing framework or external libraries like testify are used for unit and integration tests.
  • CI/CD Pipeline: Tools like Jenkins, GitLab CI, or GitHub Actions are used to automate the build, test, and deployment process.
  • Deployment Environment: The code is deployed to environments like Kubernetes, AWS, or Google Cloud.

The LLM should also be able to identify any monorepo-specific tooling or configurations. For example, the monorepo might use a custom tool to manage dependencies or to build and deploy individual services. Tools like go-mono (GitHub - chrusty/go-mono) can be used to optimize the build process by only rebuilding services that have changed.

By understanding the build and deployment process, the new developer can contribute to the project more effectively and avoid introducing breaking changes. According to www.wisp.blog, efficient CI/CD integration is one of the best practices for monorepo management.

Identifying Key Modules and Dependencies

Leveraging LLMs for Dependency Visualization

Dependency visualization is crucial for understanding complex relationships within a large codebase. While tools like govulncheck are helpful, LLMs can enhance this process by providing contextual insights.

LLM Prompt: "Generate a dependency graph visualization of the monorepo, highlighting the critical paths and potential circular dependencies. Use a format that can be easily rendered by a graph visualization tool (e.g., DOT language). For each dependency, include a brief description of its purpose and potential impact on other modules."

This prompt extends beyond simple dependency listing by requesting a visual representation and contextual information. The LLM should analyze the codebase and generate a graph that illustrates the connections between modules. The inclusion of descriptions and impact assessments adds a layer of understanding that is not typically available in standard dependency analysis tools. This is different from the previous section, which focused on identifying core packages and their direct dependencies. This section focuses on creating a visual representation of the entire dependency structure.

For example, the LLM might identify a critical path involving the following modules:

  • api/gateway (Handles incoming requests and routes them to other services.)
  • internal/auth (Authenticates and authorizes users.)
  • internal/db (Provides database access.)
  • pkg/cache (Caches frequently accessed data.)

The visualization would show the dependencies between these modules, highlighting the fact that api/gateway depends on internal/auth, which depends on internal/db, and so on. It would also identify any circular dependencies, which can lead to build problems and runtime errors.

Identifying Cross-Cutting Concerns and Shared Libraries

Understanding cross-cutting concerns and shared libraries is crucial for avoiding code duplication and ensuring consistency across the monorepo.

LLM Prompt: "Identify the packages that implement cross-cutting concerns such as logging, metrics, tracing, and security. For each concern, list the packages that use it and describe how it is implemented. Also, identify shared libraries or utility packages that are used by multiple services."

This prompt helps to identify common functionalities that are used throughout the codebase. The LLM should be able to analyze the code and identify patterns that indicate the use of cross-cutting concerns and shared libraries. This differs from the previous sections by focusing on identifying common functionalities used across the codebase rather than just core packages or dependency graphs.

For example, the LLM might identify the following cross-cutting concerns:

  • Logging: Implemented using the go.uber.org/zap package and used by almost all services.
  • Metrics: Implemented using the github.com/prometheus/client_golang package and used by all services to expose performance metrics.
  • Tracing: Implemented using the go.opentelemetry.io/otel package and used by services to trace requests across multiple services.
  • Security: Implemented using custom packages in the internal/security directory and used by services that handle sensitive data.

The LLM might also identify the following shared libraries:

  • pkg/utils: Contains utility functions for string manipulation, data validation, and error handling.
  • pkg/config: Provides a mechanism for loading and managing configuration settings.
  • pkg/queue: Provides a common interface for interacting with message queues.

Analyzing the Impact of Dependency Updates

Before making changes to a dependency, it is important to understand the potential impact on the rest of the codebase.

LLM Prompt: "Given a specific Go package (e.g., pkg/queue), identify all packages that directly and indirectly depend on it. For each dependent package, assess the potential impact of updating pkg/queue to a newer version, considering potential breaking changes or compatibility issues. Provide a risk assessment (high, medium, low) for each dependent package."

This prompt helps to assess the risk associated with updating a dependency. The LLM should be able to analyze the codebase and identify all packages that depend on the specified package. It should also be able to analyze the release notes or code changes of the newer version to identify potential breaking changes. This goes beyond simply listing dependencies by adding a risk assessment component.

For example, if pkg/queue is updated to a newer version that introduces breaking changes, the LLM might identify the following potential impacts:

  • api/gateway: High risk, as it directly depends on pkg/queue and may need to be updated to be compatible with the new version.
  • internal/order: Medium risk, as it indirectly depends on pkg/queue through api/gateway and may be affected by changes in api/gateway.
  • internal/user: Low risk, as it does not depend on pkg/queue and is unlikely to be affected by the update.

Identifying God Classes and Anti-Patterns

Identifying "God Classes" (classes that do too much) and other anti-patterns can help to improve the overall design and maintainability of the codebase.

LLM Prompt: "Analyze the codebase for instances of 'God Classes' or other anti-patterns. Identify classes or packages that have a high degree of complexity, a large number of methods, or a high degree of coupling with other classes. Suggest potential refactoring strategies to address these issues."

This prompt helps to identify areas of the codebase that may be difficult to understand and maintain. The LLM should be able to analyze the code and identify patterns that indicate the presence of anti-patterns. This is a more advanced analysis than simply identifying dependencies or cross-cutting concerns.

For example, the LLM might identify a "God Class" in the internal/order package that is responsible for handling all aspects of order processing, including order creation, order validation, payment processing, and order fulfillment. The LLM might suggest refactoring this class into smaller, more focused classes, such as OrderCreator, OrderValidator, PaymentProcessor, and OrderFulfiller.

Understanding Module Boundaries and Cohesion

Analyzing module boundaries and cohesion helps ensure that modules are well-defined and focused on specific responsibilities.

LLM Prompt: "For a given module (e.g., internal/auth), analyze its internal structure and dependencies. Determine the degree of cohesion within the module (how well its components work together) and the strength of its boundaries (how well it is isolated from other modules). Identify any potential violations of modularity principles and suggest improvements."

This prompt focuses on the internal structure of modules and their relationships with other modules. The LLM should be able to analyze the code and identify potential violations of modularity principles, such as excessive coupling between modules or lack of cohesion within a module. This is a more in-depth analysis than simply identifying dependencies or cross-cutting concerns. It focuses on the design principles of modularity.

For example, the LLM might analyze the internal/auth module and determine that it has a high degree of cohesion, as all of its components are related to authentication and authorization. However, it might also determine that the module has weak boundaries, as it directly depends on several other modules, such as internal/db and pkg/cache. The LLM might suggest reducing the dependencies on other modules by introducing interfaces or using dependency injection.

These prompts provide a structured approach to exploring and understanding a large Go monorepo, leveraging the capabilities of LLMs to accelerate the learning process and facilitate effective contributions. By focusing on practical, actionable queries, new developers can quickly grasp key concepts, identify important modules, and understand common patterns within the codebase.

Exploring Testing Strategies and Configuration Management

Understanding Testing Frameworks and Methodologies

Testing is a critical aspect of software development, especially in large monorepos. Understanding the testing landscape within the Go monorepo is essential for ensuring code reliability and preventing regressions. This section outlines how to use LLM-based queries to identify the testing frameworks used, the types of tests implemented, and the testing methodologies followed.

LLM Prompts:

  1. "List all testing frameworks used in the monorepo. Include examples of their usage." This prompt helps identify which frameworks, such as the built-in testing package, testify (https://github.com/stretchr/testify), or Ginkgo (https://github.com/onsi/ginkgo), are prevalent. The response should provide code snippets demonstrating how these frameworks are used in practice.
  2. "Identify the different types of tests (unit, integration, end-to-end) present in the monorepo. Provide examples of each." This prompt aims to categorize the tests based on their scope and purpose. Unit tests focus on individual functions or methods, integration tests verify the interaction between different modules, and end-to-end tests simulate user interactions with the application. (https://reliasoftware.com/blog/golang-testing-framework)
  3. "What testing methodologies (e.g., TDD, BDD) are followed in the monorepo? Provide evidence from the codebase." This prompt explores the development practices adopted by the team. Test-Driven Development (TDD) involves writing tests before writing the actual code, while Behavior-Driven Development (BDD) focuses on defining the expected behavior of the application in a human-readable format.
  4. "Find examples of mock implementations used for testing external dependencies. Which mocking libraries are commonly used?" This prompt helps understand how external services (databases, APIs) are handled during testing. Common mocking libraries in Go include gomock (https://github.com/golang/mock) and testify/mock.
  5. "Show examples of test fixtures used in the monorepo. Where are these fixtures typically located?" Test fixtures are pre-prepared data or configurations used to ensure consistent and repeatable test results. They are often stored in a testdata directory. (https://betterstack.com/community/guides/testing/intemediate-go-testing/)

Expected Outcomes:

  • A list of testing frameworks used in the monorepo, along with code examples.
  • Categorization of tests based on their type (unit, integration, end-to-end).
  • Identification of testing methodologies followed (TDD, BDD).
  • Examples of mock implementations and mocking libraries used.
  • Location and examples of test fixtures.

Analyzing Test Coverage and Quality

Understanding the extent to which the codebase is covered by tests and the quality of those tests is crucial for maintaining a healthy monorepo. This section focuses on using LLM-based queries to assess test coverage, identify areas with low coverage, and evaluate the quality of existing tests.

LLM Prompts:

  1. "What tools are used to measure test coverage in the monorepo? Show examples of how coverage reports are generated." Go provides built-in support for test coverage analysis using the go test -cover command. This prompt aims to identify if this tool is used and how the coverage reports are generated and interpreted.
  2. "Identify packages or modules with low test coverage (below X%). Provide a list of files in those packages and their corresponding coverage percentages." This prompt helps pinpoint areas of the codebase that require more testing effort. Setting a threshold (e.g., X = 70%) allows for focusing on the most critical areas.
  3. "Find examples of flaky tests in the monorepo. How are these tests handled (e.g., retries, exclusion)?" Flaky tests are tests that sometimes pass and sometimes fail without any code changes. Identifying and addressing flaky tests is crucial for maintaining confidence in the test suite.
  4. "Show examples of parameterized tests used in the monorepo. What libraries are used for parameterization?" Parameterized tests allow running the same test logic with different input values, reducing code duplication and improving test coverage.
  5. "Analyze the test descriptions and names. Do they clearly describe the expected behavior? Provide examples of good and bad test descriptions." Clear and descriptive test names are essential for understanding the purpose of each test and for debugging failures.

Expected Outcomes:

  • Identification of tools used for test coverage analysis.
  • A list of packages with low test coverage.
  • Examples of flaky tests and how they are handled.
  • Examples of parameterized tests and libraries used.
  • Assessment of test description quality.

Investigating CI/CD Pipeline and Testing Automation

The CI/CD pipeline plays a vital role in automating the testing process and ensuring code quality. This section explores how to use LLM-based queries to understand the CI/CD setup, identify the testing stages, and analyze the integration of tests within the pipeline.

LLM Prompts:

  1. "What CI/CD system is used in the monorepo (e.g., Jenkins, GitLab CI, GitHub Actions)? Provide the configuration file." This prompt identifies the CI/CD platform used and provides access to its configuration file, which defines the pipeline stages and steps.
  2. "Describe the testing stages in the CI/CD pipeline. What types of tests are executed in each stage?" This prompt outlines the different phases of testing within the pipeline, such as unit tests, integration tests, and end-to-end tests.
  3. "How are test results reported and visualized in the CI/CD pipeline? Are there any dashboards or reporting tools used?" Understanding how test results are presented is crucial for monitoring code quality and identifying failures.
  4. "Find examples of automated code analysis tools integrated into the CI/CD pipeline (e.g., linters, static analyzers). What rules are enforced?" Automated code analysis tools help enforce coding standards and identify potential issues early in the development process.
  5. "How are dependencies managed and cached in the CI/CD pipeline to speed up builds?" Efficient dependency management is essential for reducing build times and improving the overall CI/CD performance.

Expected Outcomes:

  • Identification of the CI/CD system used and its configuration file.
  • Description of the testing stages in the CI/CD pipeline.
  • Information on how test results are reported and visualized.
  • Examples of automated code analysis tools integrated into the pipeline.
  • Details on dependency management and caching strategies.

Understanding Configuration Management Strategies

Configuration management is crucial for managing different environments and settings in a large monorepo. This section focuses on using LLM-based queries to understand how configuration is handled, identify configuration files, and analyze the use of environment variables.

LLM Prompts:

  1. "How is configuration managed in the monorepo? Are there any specific libraries or patterns used (e.g., Viper, Envconfig)?" This prompt identifies the configuration management approach adopted by the team. Libraries like Viper (https://github.com/spf13/viper) and Envconfig (https://github.com/kelseyhightower/envconfig) are commonly used for reading configuration from files and environment variables.
  2. "Locate the main configuration files in the monorepo. What format are they in (e.g., YAML, JSON, TOML)?" This prompt helps identify the location and format of the configuration files used by the application.
  3. "How are environment variables used to configure the application? Provide examples of environment variables and their usage." Environment variables are often used to configure applications in different environments (development, staging, production).
  4. "Is there a mechanism for managing different configuration profiles for different environments? How are these profiles selected?" This prompt explores how the application handles different configuration settings for different environments.
  5. "Find examples of how sensitive information (e.g., API keys, passwords) is handled in the configuration. Is there any use of secrets management tools (e.g., HashiCorp Vault)?" Securely managing sensitive information is crucial for protecting the application and its data.

Expected Outcomes:

  • Identification of configuration management libraries and patterns used.
  • Location and format of main configuration files.
  • Examples of environment variables and their usage.
  • Description of mechanisms for managing different configuration profiles.
  • Information on how sensitive information is handled.

Identifying Common Testing Patterns and Anti-Patterns

Recognizing common testing patterns and anti-patterns can help improve the quality and maintainability of the test suite. This section focuses on using LLM-based queries to identify these patterns and anti-patterns in the monorepo.

LLM Prompts:

  1. "Identify examples of the Arrange-Act-Assert (AAA) pattern in the test suite. How consistently is this pattern followed?" The AAA pattern is a common testing pattern that involves arranging the test data, acting on the code under test, and asserting the expected results. (https://www.softwaretestingstuff.com/golang-testing)
  2. "Find examples of table-driven tests in the monorepo. What are the benefits of using this pattern?" Table-driven tests allow running the same test logic with different input values, reducing code duplication and improving test coverage.
  3. "Identify any test smells or anti-patterns in the test suite (e.g., overly complex tests, slow tests, brittle tests). Provide examples." Test smells are indicators of potential problems in the test suite, such as tests that are difficult to understand, slow to run, or prone to failure.
  4. "Are there any helper functions or utilities used to simplify test setup and teardown? Provide examples." Helper functions can reduce code duplication and improve the readability of the test suite.
  5. "How are concurrency and parallelism handled in the tests? Are there any potential race conditions or deadlocks?" Testing concurrent code requires careful attention to avoid race conditions and deadlocks.

Expected Outcomes:

  • Examples of the Arrange-Act-Assert pattern.
  • Examples of table-driven tests.
  • Identification of test smells and anti-patterns.
  • Examples of helper functions used in the test suite.
  • Information on how concurrency and parallelism are handled in the tests.

Conclusion

This research outlines a structured approach for new developers to efficiently onboard to a large Go monorepo using LLM-based queries. The core strategy involves a phased exploration, starting with understanding the monorepo's high-level directory structure and identifying core packages and their dependencies. Subsequent steps focus on analyzing package functionality, key data structures, common design patterns, and the build/deployment process. Furthermore, the research emphasizes leveraging LLMs for dependency visualization, identifying cross-cutting concerns, assessing the impact of dependency updates, and recognizing potential anti-patterns like "God Classes" to improve code maintainability. Finally, it delves into understanding testing strategies, configuration management, and common testing patterns within the monorepo.

The most important findings highlight the power of LLMs in accelerating the learning process by providing contextual insights and actionable information. The ability to generate dependency graphs, identify cross-cutting concerns, and assess the risk of dependency updates significantly reduces the time required to understand the codebase. Moreover, the prompts designed to uncover testing methodologies, configuration management strategies, and common testing patterns equip new developers with the knowledge to contribute effectively and maintain code quality.

The implications of this research are significant for developer onboarding and knowledge sharing in large software projects. By adopting this structured approach and utilizing LLM-based queries, new developers can quickly grasp key concepts, identify important modules, and understand common patterns within the codebase. As a next step, it would be beneficial to evaluate the effectiveness of these prompts in a real-world scenario, measure the time saved during onboarding, and refine the prompts based on user feedback. Further research could also explore the use of LLMs to automatically generate documentation, identify potential security vulnerabilities, and suggest code improvements. Efficient CI/CD integration is also one of the best practices for monorepo management.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment