This report summarizes the results of an investigation into the behavior of tools provided by the Metals MCP server. The investigation was based on the information about MCP support found in the Metals release notes.
Test Environment:
- JVM: OpenJDK 24
- Metals Version: 1.5.3
- Scala Version: 3.7.0
- sbt Version: 1.10.11
- Project Configuration: A standard sbt project with
src/main/scalaandsrc/test/scala. MUnit was used as the testing framework. - AI Agent: Roo Code
The following table outlines the main MCP tools verified and a summary of their behavior.
| Tool Name | Stability/Accuracy | Notes |
|---|---|---|
compile-file |
Generally stable (no response, but no errors) | |
compile-full |
Generally stable (no response, but no errors) | |
test |
Unstable/Inconsistent | Behavior varies significantly with package structure. See details below. |
glob-search |
Stable | |
typed-glob-search |
Stable | |
inspect |
Discrepancy with documentation | Information returned for objects and methods is limited. See details. |
get-docs |
Stable | Correctly retrieves Scaladoc comments. |
get-usages |
Stable | Correctly detects symbol usages. |
import-build |
Generally stable (responds) | Did not affect the unstable behavior of the test tool. |
Premise (from release notes, etc.):
- The
testtool is intended to run Scala test suites. - It accepts
testClass(required),testFile(optional), andverbose(optional, default false) as arguments.
Expected Behavior:
- The specified test class should be executed consistently, regardless of its package structure, and return a result.
- The
verboseargument should be optional as per documentation.
Actual Behavior:
- Issue with
verboseargument:- On initial execution, an error occurred stating the
verboseargument was missing. It appears to be treated as a required argument. This can be circumvented by specifyingverbose: false.
- On initial execution, an error occurred stating the
- Inconsistent behavior based on package structure:
- Unstable behavior was observed across multiple test cases, where the
testtool would sometimes respond and sometimes not, depending on the test suite's package structure. - Observed Patterns:
- For one test suite, placing it in the same package hierarchy as the code under test resulted in no response, while placing it in a sub-package of the code under test resulted in normal operation.
- For another test suite, the opposite occurred: it operated normally when placed in the same package hierarchy as the code under test, but failed to respond when placed in a sub-package.
- A further minimal test suite failed to respond in either package structure.
- However, test suites located in the default package always operated normally.
- This inconsistent behavior persisted even after restarting the Metals server or executing
import-build.
- Unstable behavior was observed across multiple test cases, where the
Possible Causes:
- Potential issues in the Metals server's internal logic for test detection, classpath resolution, or test runner invocation, possibly influenced by factors beyond package hierarchy, such as specific test suite names, file structures, or processing order.
- Issues related to how files are handled when the same package name is used in
src/main/scalaandsrc/test/scala.
Premise (from release notes, etc.):
- The
inspecttool returns detailed information about Scala symbols.- For packages, objects, and traits: a list of members.
- For classes: a list of members and constructors.
- For methods: signatures of all overloaded methods.
Expected Behavior:
- Detailed information corresponding to the symbol type should be returned as described above.
Actual Behavior:
- For objects (e.g.,
dev.capslock.metalsmcpexercise.Minimal):- Expected: A list of members, including the
getOnemethod. - Actual: Only the symbol type and name were returned (e.g.,
object Minimal).
- Expected: A list of members, including the
- For methods (e.g.,
dev.capslock.metalsmcpexercise.Minimal.getOne):- Expected: The method signature.
- Actual: No response.
- This behavior was consistent for custom objects/methods within the project as well as for standard library classes (e.g.,
scala.collection.immutable.List).
Possible Causes:
- The current implementation of the
inspecttool may not yet provide the full range of information described in the documentation.
The Metals MCP server offers many useful information retrieval features and holds great potential for integration with AI agents.
Tools like get-docs, get-usages, glob-search, and typed-glob-search performed stably.
However, the unstable behavior of the test tool and the lack of detailed information from the inspect tool present challenges in terms of reliability and usability.
Addressing these points would significantly enhance the Metals MCP server.
Suggestions:
- Investigate and rectify the inconsistent behavior of the
testtool concerning package structures. - Enhance the
inspecttool to provide detailed information as per its documentation. - Update the documentation for the
testtool'sverboseargument to reflect its actual behavior (i.e., if it's mandatory).
We hope this report contributes to the quality improvement of Metals.