Skip to content

Instantly share code, notes, and snippets.

@jsquire
Last active March 28, 2023 19:52
Show Gist options
  • Save jsquire/cfff24f50da0d5906829c5b3de661a84 to your computer and use it in GitHub Desktop.
Save jsquire/cfff24f50da0d5906829c5b3de661a84 to your computer and use it in GitHub Desktop.
Azure SDK Repository Automation Rules

Azure SDK Repository Automation Rules

Goal

This document attempts to define a set of logical rules based on the existing Fabric Bot implementation. Rules are described based on their logical association; in some cases, these may need to be expanded into multiple physical rules due to how GitHub events and/or Actions work. Likewise, concepts and flow are intentionally kept abstract and do not intend to describe a specific implementation. An example of this is the different data integrations. Multiple data items may drive from a single source, but they constitute different logical concepts.

Nomenclature

  • Trigger: An event that occurs in GitHub to which automation should respond to.
  • Target: An item in GitHub that can trigger an event. Generally, this will be an issue or pull request.
  • Criteria: A set of evaluations performed for a trigger target that determines which actions, if any, result.
  • Actions: A set of "things that happen" in response to a trigger. For example, adding a label to an issue.

Conventions

  • Textual content for comments generated by rules generally makes reference to existing content in the .NET ruleset. For brevity, these read similar to Content from .NET rule #. These rule numbers can be viewed by loading the .NET repository into the Fabric Bot portal.

Data integrations

  • Service partners: Each partner team that owns packages in our repository will need to have an association tracking a service label to one or more GitHub handles. The contacts for service teams will follow the same pattern as the "Service Attention" process does today; it must be possible to assign a set of service owners that differs from the individuals assigned as pull request reviwers.

    Example:

    {
      "label" : "Storage", 
      "contacts" : ["@person1", "@person2"]
    }
  • Azure SDK team owners: Each package in our repository that is owned by our team will need to have an association tracking a service label to one or more GitHub handles. This is an unofficial mapping that does not necessarily correlate to authoritative "Azure SDK Team" membership, such as that represented by our security groups or the "azure-sdk-team" GitHub team. That said, using one of these groups to validate membership on top of the label association would be just fine.

    The set of owners responsible for triage for a given service may not be the same as those who should be tagged as pull request reviewers. It should not be assumed that triage owners can be correctly inferred from the existing path/label definitions in CODEOWNERS.

    Example:

    {
      "label" : "Event Hubs", 
      "contacts" : ["@jsquire"]
    }
  • File paths: A service label may have one or more file paths relative to the repository structure associated with it.

    Example:

    {
      "label" : "Tables", 
      "paths" : ["/sdk/tables/Azure.Data.Tables", "/sdk/samples/some-table-sample"]
    }

External integrations

AI label service

This is a REST API that evaluates the content of an issue and attempts to predict a set of labels that should be applied to it. If a prediction is made with a reasonable level of confidence, the service will return exactly one service label and one category label. If confidence thresholds were not met for both label types, no labels are returned.

Example payload:

{
  "issueNumber": 32227,
  "title": "Move Snippet Generator to .NET 6",
  "body": "The snippet generator tool currently targets netcoreapp3.1 which reaches end-of-life in December, 2022. The target framework should be updated to net6.0 and the associated Update-Snippet script should be updated to reference the new version.",
  "issueUserLogin": "jsquire",
  "repositoryName": "azure-sdk-for-net",
  "repositoryOwnerName": "Azure"
}

Example responses:

Predictions made

{
  "labels": [
    "Storage",
    "Client"
  ]
}

No predictions

{
  "labels": []
}

Important context

Today, this functionality exists as part of an Azure Function that hooks into our Event Hubs feed for GitHub events and operates independent of FabricBot and other automation. It will need to be revised to expose an API that can be called on demand from the actions-based rules, or we will need to determine a way to more directly integrate it into the new platform.

For the purposes of the rule definitions herein, it is assumed that this is a stand-alone service providing a REST API.

Issue Rules

Initial issue triage

Trigger

  • Issue created

Criteria

  • Issue has no labels
  • Issue has no assignee

Actions

  • Evaluate the user that created the issue:

    IF creator is NOT an Azure SDK team owner   
      AND is NOT a member of the Azure organization
      AND is does NOT have a collaborator association
      AND does NOT have write permission
      AND does NOT have admin permission:
      
          - Add "customer-reported" label
          - Add "question" label
    
  • Query AI label service for suggestions:

    IF labels were predicted:
        - Assign returned labels to the issue
        - Add "needs-team-attention" label to the issue
        
        IF service label is associated with an Azure SDK team member:
            IF a single team member:
                - Assign team member to the issue
            ELSE
                - Assign a random team member from the set to the issue
                - Add a comment mentioning the other team members from the set
            
            - Add comment indicating issue was routed for assistance  
                (text: "Thank you for your feedback.  Tagging and routing to the team member best able to assist.")
        ELSE
            - Add "CXP Attention" label to the issue
            - Create a comment mentioning (content from .NET rule #30)
    ELSE
        - Add "needs-triage" label to the issue
    

Manual issue triage

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open
  • Issue has "needs-triage" label
  • Label being added is NOT "needs-triage"

Actions

  • Remove "needs-triage" label

Service Attention

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open
  • Label added is "Service Attention"
  • One or more service team contacts is associated with the set of labels assigned to the issue

Actions

  • Create a comment mentioning the service team contacts (content from .NET rule #14)

CXP Attention

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open
  • Label added is "CXP Attention"
  • Issues does NOT have label "Service Attention"

Actions

  • Create a comment mentioning (content from .NET rule #30)

Manual triage after external assignment

Trigger

  • Issue modified for:
    • Label removed

Criteria

  • Issue is open
  • Issue has "customer-reported" label
  • Label removed is "Service Attention" OR "CXP Attention"

Actions

  • Add "needs-team-triage" label

Author feedback

Trigger

  • Issue comment created

Criteria

  • Issue is open
  • Issue has "needs-author-feedback" label
  • Commenter is the original issue author

Actions

  • Remove "needs-author-feedback" label
  • Add "needs-team-attention" label

Reset issue activity

Trigger

  • Issue modified for:
    • Reopen
    • Content edited

OR

  • Issue comment created

Criteria

  • Issue is open OR being reopened
  • Issue has "no-recent-activity" label
  • Account modifying the issue is NOT a known bot

Actions

  • Remove "no-recent-activity" label

Reopen issue

Trigger

  • Issue comment created

Criteria

  • Issue is closed
  • Issue has label "no-recent-activity"
  • Issue has label "needs-author-feedback"
  • Issue was closed for 7 days or less
  • Commenter is the original issue author
  • Action is not "comment and close"

Actions

  • Reopen the issue
  • Remove "no-recent-activity" label
  • Remove "needs-author-feedback" label
  • Add "needs-team-attention" label

Decline to reopen issue

Trigger

  • Issue comment created

Criteria

  • Issue is closed
  • Issue was closed for more than 7 days ago
  • Commenter does NOT have a collaborator association
  • Commenter does NOT have write permission
  • Commenter does NOT have admin permission
  • Action is not "comment and close"

Actions

  • Create a comment (content from .NET rule #10)

Require attention for non-milestone

Trigger

  • Issue modified for:
    • Label added
    • Label removed

Criteria

  • Issue is open
  • Issue has label "customer-reported"
  • Issue does NOT have label "needs-team-attention"
  • Issue does NOT have label "needs-triage"
  • Issue does NOT have label "needs-team-triage"
  • Issue does NOT have label "needs-author-feedback"
  • Issue does NOT have label "issue-addressed"
  • Issue is not in a milestone

Actions

  • Add "needs-team-attention" label

Author feedback needed

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open
  • Label added is "needs-author-feedback"

Actions

  • Remove "needs-triage" label
  • Remove "needs-team-triage" label
  • Remove "needs-team-attention" label
  • Create a comment
    (text: Hi @${issueAuthor}. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.)

Issue Addressed

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open
  • Label added is "issue-addressed"

Actions

  • Remove "needs-triage" label
  • Remove "needs-team-triage" label
  • Remove "needs-team-attention" label
  • Remove "needs-author-feedback" label
  • Remove "no-recent-activity" label
  • Create a comment mentioning (content from .NET rule #25)

Issue Addressed commands

Trigger

  • Issue comment created

Criteria

  • Issue has label "customer-reported"
  • Comment text contains the string "/unresolve"

Actions

  • Evaluate the permissions of the commenter:

    IF commenter is the issue author
      OR commenter has a collaborator association
      OR commenter has write permission
      OR commenter has admin permission:
      
          - Reopen the issue
          - Remove label "issue-addressed"
          - Add label "needs-team-attention"
    ELSE
        - Create a comment (content from .NET rule #28)
    

Issue Addressed reset

Trigger

  • Issue modified for:
    • Label added

Criteria

  • Issue is open

  • Issue has label "issue-addressed"

  • Label added is any one of:

    • "needs-team-attention"
    • "needs-author-feedback"
    • "Service Attention"
    • "CXP Attention"
    • "needs-triage"
    • "needs-team-triage"

Actions

  • Remove "issue-addressed" label

Pull Request Rules

Pull Request triage

Trigger

  • PR created

Criteria

  • Pull request has no labels

Actions

  • Evaluate the paths for each file in the PR:

    IF the path is associated with a label:
        - Assign the label
    
  • Determine if this is a community contribution:

    IF the PR author does not have write permission
      AND the PR author does not have write permission
      AND the PR author does not have a collaborator association:
        - Add "customer-reported" label
        - Add "Community Contribution" label
        - Create a comment (content from .NET rule #17)
    

Reset pull request activity

Trigger

  • PR modified for:
    • Reopen
    • Changes Pushed
    • Merged
    • Review Requested
    • Review Submitted

OR

  • Pull request comment created

Criteria

  • Pull request is open OR being reopened

  • Pull request has "no-recent-activity" label

  • User modifying the pull request is NOT a known bot

  • If triggered by comment creation, evaluate:

    • Commenter is the pull request author OR has write or admin permissions
    • Comment text does NOT contain the string "Check Enforcer"
    • Comment text does NOT contain the string "Since there hasn't been recent engagement, this is being closed out."

Actions

  • Remove "no-recent-activity" label

Reopen pull request

Trigger

  • Pull request comment created

Criteria

  • Pull request is closed
  • Pull request has "no-recent-activity" label
  • Commenter is the pull request author OR has write or admin permission
  • Comment text contains the string "/reopen"

Actions

IF commenter is the pull request author 
  OR the commenter has write permissions
  OR the commenter has admin permissions
  OR the commenter has a collaborator association:
    - Remove "no-recent-activity" label
    - Reopen pull request

ELSE
  - Create a comment (text: "Sorry, @commenter, only the original author can reopen this pull request.")

Reset auto-merge approvals on untrusted changes

Trigger

  • PR modified for:
    • Changes pushed

Criteria

  • Pull request is open
  • Pull request has "auto-merge" label
  • User who pushed the changes does NOT have a collaborator association
  • User who pushed changes does NOT have write permission
  • User who pushed changes does NOT have admin permission

Actions

  • Reset all approvals
  • Create a comment (content from .NET rule #24)

Scheduled Rules

Close stale issues

Trigger

  • CRON (daily at 1am)

Criteria

  • Issue is open
  • Issue has "needs-author-feedback" label
  • Issue has "no-recent-activity" label
  • Issue was last modified more than 14 days ago

Actions

  • Close the issue

Close stale pull requests

Trigger

  • CRON (every 6 hours)

Criteria

  • Pull request is open
  • Pull request has "no-recent-activity" label
  • Pull request was last modified more than 7 days ago

Actions

  • Add a comment (content from .NET rule #21)
  • Close the pull request

Identify stale issues

Trigger

  • CRON (every 6 hours)

Criteria

  • Issue is open
  • Issue has "needs-author-feedback" label
  • Issue does NOT have "no-recent-activity" label
  • Issue was last updated more than 7 days ago

Actions

  • Add "no-recent-activity" label
  • Create a comment (content from .NET rule #6)

Identify stale pull requests

Trigger

  • CRON (weekly, Friday at 5am)

Criteria

  • Pull request is open
  • Pull request does NOT have "no-recent-activity" label
  • Pull request was last updated more than 60 days ago

Actions

  • Add "no-recent-activity" label
  • Create a comment (content from .NET rule #23)

Close addressed issues

Trigger

  • CRON (every 6 hours)

Criteria

  • Issue is open
  • Issue has label "issue-addressed"
  • Issue was last updated more than 7 days ago

Actions

  • Create a comment (content from .NET rule #26)
  • Close the issue

Lock closed issues

Trigger

  • CRON (every 6 hours)

Criteria

  • Issue is closed
  • Issue was last updated more than 90 days ago

Actions

  • Lock issue conversations
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment