I've seen too many incompatibility issues with specific versions if Python - Spark - Arrow.
So, documenting which versions have worked for me in the past!
| Spark | Python | Arrow | Comment |
|---|---|---|---|
| 3.5.x | 3.8.x | 12.x | |
| 3.5.x | 3.9.x | 12.x | |
| 3.5.x | 3.10.x | 12.x | |
| 3.5.x | 3.11.x | 12.x | |
| 3.4.x | 3.9.x | 12.x | |
| 3.4.x | 3.10.x | 12.x | |
| 3.4.x | 3.11.x | 12.x | |
| 3.3.x | 3.7.x | 12.x | |
| 3.3.x | 3.8.x | 12.x | |
| 3.3.x | 3.9.x | 12.x | |
| 3.3.x | 3.10.x | 12.x | |
| 3.2.x | 3.6.x | 12.x | |
| 3.2.x | 3.7.x | 12.x | |
| 3.2.x | 3.8.x | 12.x | |
| 3.2.x | 3.9.x | 12.x | |
| 2.4.x | 3.7.x | 7.x | |
| 2.4.x | 3.6.x | 7.x | |
| 2.3.x | 3.6.x | 7.x | |
| 2.3.x | 3.7.x | 7.x |
References: