Skip to content

Instantly share code, notes, and snippets.

@robertmryan
Created October 9, 2024 15:06
Show Gist options
  • Save robertmryan/0cb300bdbcd560cfa999b817e8f71999 to your computer and use it in GitHub Desktop.
Save robertmryan/0cb300bdbcd560cfa999b817e8f71999 to your computer and use it in GitHub Desktop.
func upload(files: [URL]) async throws -> Bool {
try await withThrowingTaskGroup(of: Bool.self) { group in
let session = try SSH(host: "xxx", port:"xxx")
let sftp = try session.openSftp()
for file in files {
group.addTask {
try await self.upload(file: file, with: sftp, session: session)
}
}
var count = 0
for try await isSuccessful in group where isSuccessful {
count += 1
}
return count == files.count
}
}
@robertmryan
Copy link
Author

robertmryan commented Oct 10, 2024

@williamhqs

Regarding cancelation, consider:

actor Foo {
    func foo() async throws {
        try await withThrowingTaskGroup(of: Void.self) { group in
            for i in 0 ..< 20 {
                group.addTask {
                    try await self.bar(i)
                }
            }
            
            // if you use this pattern (or any pattern that results in `group.next()` to be called),
            // the whole group will be canceled as soon as any task fails.

            for try await _ in group { }
        }
    }
    
    func bar(_ value: Int) async throws {
        print("starting", #function, value)
        
        if value == 4 {
            print("failed", #function, value)
            throw FooError.foobar
        }
        
        try await Task.sleep(for: .seconds(1))
        
        print("successfully finished", #function, value)
    }
    
    enum FooError: Error {
        case foobar
    }
}

That will cancel the whole group as soon as one fails.

However, if you do something like the following, the group will not cancel all of its tasks (though the error will be thrown):

actor Foo {
    func foo() async throws {
        try await withThrowingTaskGroup(of: Void.self) { group in
            for i in 0 ..< 20 {
                group.addTask {
                    try await self.bar(i)
                }
            }
            
            // however if you use this pattern instead, the group is not. as the docs for `waitForAll` says:
            //
            // “If any of the tasks throw, the first error thrown is captured and re-thrown 
            // by this method although the task group is not cancelled when this happens.”

            try await group.waitForAll()
        }
    }
    
    func bar(_ value: Int) async throws {}
}

That is what I was trying to convey when I said:

But this begs the question: What do you want to happen if an upload fails? Continue trying the others or stop trying? And if one or more failed, do you want to know which ones failed?

I didn’t know what you wanted to happen if one failed. Hopefully this helps clarify a few of the options.


Note, this presumes that your upload method even supports cancellation. My trivial bar method does, but I can’t vouch for Shout.

@robertmryan
Copy link
Author

@williamhqs – Now, regarding your remaining error, that would appear to stem from Shout’s failure to define SSH or SFTP as Sendable. I can’t comment directly on that (as this is not a library that I’ve ever used), but given that it’s using a non-isolated class (rather than an isolated one or an actor), so that’s not promising. You might want to post an issue there. I confess that I don’t even know if it supports concurrent requests; glancing at the code, I don’t see any async functions, so that is not promising, either.

But if you post an issue there, the author might be able to comment further. But prepare yourself for the possibility that simply can’t do that. Or you might have to create a separate session for each upload. I can’t comment without doing a lot more investigation (which I am am afraid that I can’t tackle right now as I’m currently quite busy).

@williamhqs
Copy link

@robertmryan That's already a great help for me, thank you for you time, have a good day! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment