marin-community/thalas#4 - 2-way sync via github-pr.py (ryan-williams/git-helpers)

thalas#4.md

marin-community/thalas#4 Factor executor code+history from marin-community/marin

Extend/Replace #3

• Include more paths in history replay (instead of adding them de novo on top of replayed history, as #3 does in https://github.com/marin-community/thalas/pull/3/commits/119ddf8c9e7ee7a3500454a647dfd9d98f28439f)
• Include "Original-Commit: <URL>" line at the end of each replayed commit's body (expand commit msgs below for examples)

Marin test: #1567

marin#1567 uses this branch, and has green CI.

Failing tests (worked around)

Previous writeup re: test failures; fixed / worked around by @dlwh in f3ffcee:

I'm not sure what's causing the test failures:

FAILED tests/test_executor.py::test_executor - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_force_run_failed - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_versioning - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_dedup_version - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_run_only_some_steps - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_parent_version_bubbles_into_skip_child - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_parent_doesnt_run_on_skip_parent - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_skippable_parent_will_run_if_asked - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.
FAILED tests/test_executor.py::test_parent_will_run_if_some_child_is_not_skippable - ray.util.state.exception.ServerUnavailable: Failed to make request to http://127.0.0.1:8265/api/v0/nodes. Failed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`.

• It may have to do with changes in marin#1546 (which landed after #3's history was created).
• We may also need some secrets from marin added to this repo.
• Here's an example of the same test_executor tests passing in Marin.

Scripts gist

This gist contains scripts used to:

• Generate filtered history
• Rebase on this repo's current initial empty commit
• Apply 2 patches on top:
  • Rename imports/files
  • Add additional "exports" to src/thalas/__init__.py

As long as we don't decide to change the replayed history, we can "merge" further upstream commits onto this branch (without force-pushing), while reviewing.