Flink's flink-filesystems modules use a MinIO Testcontainer
(MinioTestContainer) as the S3-compatible backend for integration tests of
flink-s3-fs-base, flink-s3-fs-hadoop, flink-s3-fs-presto, and
flink-s3-fs-native. This document evaluates whether SeaweedFS can replace
MinIO in that role.
The evaluation is a feature-by-feature comparison of every S3 API operation Flink actually invokes against the operations SeaweedFS implements. It is based on a complete read of the relevant source in both projects at the revisions listed below.
| Repository | URL | Commit |
|---|---|---|
| Apache Flink | https://github.com/apache/flink | 995609633b13ecd337f476f17f3eb9d50f832284 (2026-06-24) |
| SeaweedFS | https://github.com/seaweedfs/seaweedfs | fb168e2a361798ab355e056761372eb459e4c497 (2026-06-23) |
- Can MinIO be replaced with SeaweedFS? Yes from an S3-protocol perspective. Every S3 operation Flink calls against MinIO has a working SeaweedFS implementation with matching error codes, ETag formats, signing, and addressing modes.
- Are there features Flink uses that SeaweedFS does not support? No. Across all four S3 filesystem modules and their tests, no Flink code path invokes an operation that SeaweedFS lacks.
The required change is to the Testcontainer wrapper, not to Flink's S3 client code.
Operations actually invoked by Flink in production code paths:
| Operation | Flink call site |
|---|---|
GetObject (with Range: header) |
flink-s3-fs-native/.../NativeS3InputStream.java:178,186; NativeS3ObjectOperations.java:388 |
HeadObject |
NativeS3FileSystem.java:208; NativeS3ObjectOperations.java:399 |
ListObjectsV2 (delimiter, continuation token, pagination) |
NativeS3FileSystem.java:278,348 |
PutObject |
NativeS3OutputStream.java:208; NativeS3ObjectOperations.java:214,261 |
DeleteObject (with 404-as-success semantics) |
NativeS3FileSystem.java:388; NativeS3ObjectOperations.java:371 |
CopyObject (single-bucket rename) |
NativeS3FileSystem.java:486 |
CreateMultipartUpload (with SSE header propagation) |
NativeS3ObjectOperations.java:139 |
UploadPart |
NativeS3ObjectOperations.java:191 |
CompleteMultipartUpload (idempotent on NoSuchUpload) |
NativeS3ObjectOperations.java:311 |
AbortMultipartUpload |
NativeS3ObjectOperations.java:335 |
Bulk GetObject via S3TransferManager |
NativeS3BulkCopyHelper.java:214 |
STS AssumeRole, GetSessionToken (optional, for credentials) |
S3ClientProvider.java:517–522,711 |
Authentication and addressing:
- AWS Signature V4 (mandatory; SDK v2 default).
- Path-style addressing (used by
MinioTestContainer.java:100,118). - Virtual-host style addressing (default in production).
aws-chunkedstreaming SigV4 (AWS SDK v2 default forPutObject).
Encryption (configurable, off by default):
- SSE-S3, SSE-KMS with key-id and encryption context
(
S3EncryptionConfig.java).
Error codes Flink branches on:
NoSuchKey(404) – mapped toFileNotFoundException(NativeS3FileSystem.java:244–256).NoSuchUpload– treated as idempotent commit, retries by checking object existence (NativeS3ObjectOperations.java:314–320).BucketAlreadyOwnedByYou– tolerated when test setup creates an existing bucket.
Operations Flink does not use anywhere: tagging, ACL, versioning, lifecycle, replication, notification, inventory, analytics, S3 Select, presigned URLs, client-side encryption, restore-from-Glacier.
| Flink-used feature | SeaweedFS support | Evidence in seaweedfs/weed/ |
|---|---|---|
GetObject + Range: (incl. suffix ranges, 206 + Content-Range) |
Full | s3api/s3api_object_handlers.go:142–237 |
HeadObject |
Full | s3api/s3api_object_handlers.go:2119+ |
ListObjectsV2 (delimiter, continuation token) |
Full | s3api/s3api_server.go:923 |
PutObject (with checksums and metadata preservation) |
Full | s3api/s3api_object_handlers_put.go:393–700 |
DeleteObject |
Full | s3api/s3api_server.go:838 |
CopyObject (incl. all copy-source-if-* conditions) |
Full | s3api/s3api_object_handlers_copy.go:1444–1481 |
CreateMultipartUpload (SSE propagation to parts) |
Full | s3api/s3api_object_handlers_multipart.go:80–125 |
UploadPart |
Full | s3api/s3api_object_handlers_multipart.go:327–474 |
CompleteMultipartUpload (composite ETag <md5>-<n>) |
Full, AWS-spec format | s3api/filer_multipart.go:1224–1248 |
AbortMultipartUpload |
Full | s3api/s3api_object_handlers_multipart.go:196–234 |
NoSuchUpload error code (used for idempotent commit) |
Returned with matching code | s3api/s3err/s3api_errors.go:61,290 |
BucketAlreadyOwnedByYou |
Returned with matching code | s3api/s3err/s3api_errors.go:54,190 |
NoSuchKey 404 |
Returned with matching code | s3api/s3err/s3api_errors.go |
| AWS Signature V4 | Full | s3api/s3api_auth.go:22,92–96 |
| Path-style addressing | Full | s3api/s3api_server.go:735,753 |
| Virtual-host style addressing | Full | s3api/s3api_server.go:744–747 |
aws-chunked streaming SigV4 |
Full | s3api/chunked_reader_v4.go |
| SSE-S3, SSE-KMS, SSE-C | Full | s3api/s3_sse_s3.go, s3_sse_kms.go, s3_sse_c.go |
STS AssumeRole, GetSessionToken, AssumeRoleWithWebIdentity |
Full | s3api/s3api_server.go:991–1008 |
Bulk GetObject (parallel reads via S3TransferManager) |
Works (no special server-side feature needed) | n/a |
No gaps were found.
These are real differences between MinIO and SeaweedFS, but none of them break any Flink code path that exists today.
- Minimum part size. AWS S3 enforces a 5 MB minimum on non-final
multipart parts. MinIO enforces this. SeaweedFS does not. Flink already
enforces this client-side via
S3_MULTIPART_MIN_PART_SIZEinflink-s3-fs-base, so SeaweedFS being more permissive is harmless. - SSE-KMS key management. SeaweedFS handles SSE-KMS headers, key-id propagation, and encryption context. Without an external KMS configured, the keys live inside SeaweedFS. For tests this is equivalent to MinIO's behavior; for production parity an external KMS must be wired up separately.
- Read-after-write consistency. SeaweedFS routes writes and reads of the
same key to the same filer "ring owner"
(
s3api/s3api_handlers.go:35–42), giving strong consistency in single-region setups, which is what the recoverable writer relies on. Multi-filer deployments need attention to replication latency.
SeaweedFS is feature-complete for everything Flink does over S3. The switch is a Testcontainers refactor, not a filesystem-client refactor.