Python file showing issue with (spurious?) overlap in overloads

gistfile1.py

from typing import overload, Literal, List, Tuple, Any

class Example:
    pass

class Prediction:
    pass


class TestMypy:

    @overload
    def __call__(
            self,
            program,
            return_all_scores: Literal[True] = ...,
            return_outputs: Literal[True] = ...,
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]], List[float]]:
        ...

    @overload
    def __call__(
            self,
            program,
            return_all_scores: Literal[True] = ...,
            return_outputs: Literal[False] = ...,
    ) -> Tuple[float, List[float]]:
        ...

    @overload
    def __call__(
            self,
            program,
            return_all_scores: Literal[False] = ...,
            return_outputs: Literal[True] = ...,
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]]]:
        ...

    @overload
    def __call__(
            self,
            program,
            return_all_scores: Literal[False] = ...,
            return_outputs: Literal[False] = ...,
    ) -> float:
        ...

    def __call__(
            self,
            program,
            return_all_scores: bool = False,
            return_outputs: bool = False,
    ):
        if return_all_scores:
            if return_outputs:
                return 0.1, [(Example(), Prediction(), 0.1)], [0.1]
            else:
                return 0.1, [0.1]
        elif return_outputs:
            return 0.1, [(Example(), Prediction(), 0.1)]
        else:
            return 0.1

mypy-output.txt

$ poetry run mypy dspy/evaluate/mypy_issue.py
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 4 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:22: error: Overloaded function signatures 2 and 3 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:22: error: Overloaded function signatures 2 and 4 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:31: error: Overloaded function signatures 3 and 4 overlap with incompatible return types  [overload-overlap]
Found 6 errors in 1 file (checked 1 source file)

mypy-output2.txt

$ poetry run mypy dspy/evaluate/mypy_issue.py
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 2 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 3 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:13: error: Overloaded function signatures 1 and 4 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:23: error: Overloaded function signatures 2 and 3 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:23: error: Overloaded function signatures 2 and 4 overlap with incompatible return types  [overload-overlap]
dspy/evaluate/mypy_issue.py:33: error: Overloaded function signatures 3 and 4 overlap with incompatible return types  [overload-overlap]
Found 6 errors in 1 file (checked 1 source file)

mypy_issue2.py

from typing import overload, Literal, List, Tuple, Any

class Example:
    pass

class Prediction:
    pass


class TestMypy:

    @overload
    def __call__(
            self,
            program,
            foo: int = ...,
            return_all_scores: Literal[True] = ...,
            return_outputs: Literal[True] = ...,
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]], List[float]]:
        ...

    @overload
    def __call__(
            self,
            program,
            foo: int = ...,
            return_all_scores: Literal[True] = ...,
            return_outputs: Literal[False] = ...,
    ) -> Tuple[float, List[float]]:
        ...

    @overload
    def __call__(
            self,
            program,
            foo: int = ...,
            return_all_scores: Literal[False] = ...,
            return_outputs: Literal[True] = ...,
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]]]:
        ...

    @overload
    def __call__(
            self,
            program,
            foo: int = ...,
            return_all_scores: Literal[False] = ...,
            return_outputs: Literal[False] = ...,
    ) -> float:
        ...

    def __call__(
            self,
            program,
            foo: int = 5,
            return_all_scores: bool = False,
            return_outputs: bool = False,
    ):
        if return_all_scores:
            if return_outputs:
                return foo, [(Example(), Prediction(), 0.1)], [0.1]
            else:
                return foo, [0.1]
        elif return_outputs:
            return foo, [(Example(), Prediction(), 0.1)]
        else:
            return foo

If you get rid of all the = ... in the first 3 overloads in your example, mypy won't complain.

The reason you need to do that is that you have 4 possible combinations of the arguments return_all_scores and return_outputs. If you use the ..., then you are saying that the argument isn't required. But what changes the behavior in your case is when the arguments are explicit.

@Dr-Irv Sorry if I am being dense, but the actual definition of __call__ gives default values to those two parameters, so wouldn't it be wrong of me to omit the = ... in the overloads?

Also, in the real code those 2 arguments appear after other parameters that also have default values. So isn't it wrong to have parameters without defaults after parameters with defaults in the stubs? Or is that permissible in type stubs even though it's wrong in actual python code?

I will test this out, thanks.

@Dr-Irv Yes, I just checked, and when I drop the = ... in my full code, mypy errors out because there are non-defaulted parameters following defaulted parameters. So I can't omit that (unless there's something I am missing). I will make a new version of the gist

The additional two files show what goes wrong if I omit the = ....

Note that this argument is not required, but it is determined by the method form's default arguments. So this still looks like a bug to me.

These overloads worked:

    @overload
    def __call__(
        self,
        program,
        foo: float = ...,
        *,
        return_all_scores: Literal[True],
        return_outputs: Literal[True],
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]], List[float]]: ...

    @overload
    def __call__(
        self,
        program,
        foo: float = ...,
        *,
        return_all_scores: Literal[True],
        return_outputs: Literal[False],
    ) -> Tuple[float, List[float]]: ...

    @overload
    def __call__(
        self,
        program,
        foo: float = ...,
        *,
        return_all_scores: Literal[False],
        return_outputs: Literal[True],
    ) -> Tuple[float, List[Tuple[Example, Prediction, Any]]]: ...

    @overload
    def __call__(
        self,
        program,
        foo: float = ...,
        *,
        return_all_scores: Literal[False],
        return_outputs: Literal[False],
    ) -> float: ...

Thank you so much! I am very grateful. I made the corresponding changes to the real code from which this MWE was taken and all is well now.