My two cents on debugging, if you have ssh access to some of your nodes, especially the one where an xp failed. There is also a possibility to remotely debug the code. This may seem a bit tricky however could help to check the pipeline. I tested it with VSCode.
- ssh to the machine and the environment (docker container etc)
- Install
debugpy
pip install debugpy
- Setup debug configuration on VSCode (https://code.visualstudio.com/docs/python/debugging#_remote-debugging)
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Remote Attach",
"type": "python",
"request": "attach",
"connect": {
"host": "localhost",
"port": 5678
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}",
"remoteRoot": "."
}
]
}
]
}
- Run
python -m debugpy --wait-for-client --listen 5678 script.py
- In VSCode hit F5
- Profit ๐
- Setup
pip install --upgrade ptvsd
on local and remote machines - Follow the guide on how to setup debugging interface from VSCode: https://code.visualstudio.com/docs/python/debugging#_remote-debugging
{
"name": "Python: Attach",
"type": "python",
"request": "attach",
"port": 1313,
"host": "localhost",
"pathMappings": [
{
"localRoot": "${workspaceFolder}", // Maps C:\Users\user1\project1
"remoteRoot": "." // To current working directory ~/project1
}
]
}
- On remote run:
python -m ptvsd --host 0.0.0.0 --port 1313 --wait $PWD/code_to_debug.py
- On local machine: Set some breakpoints, run debug configuration "Attach (Remote Debug)".
- Profit ๐
HTH