Skip to content

Instantly share code, notes, and snippets.

@jsam
Last active January 21, 2020 12:19
Show Gist options
  • Save jsam/9add673befd47487b01294a87f602e37 to your computer and use it in GitHub Desktop.
Save jsam/9add673befd47487b01294a87f602e37 to your computer and use it in GitHub Desktop.
The idea of what I'm trying to achieve (sync solution would be acceptable)
def upload(parts: Vector[Part[F]], blocker: Blocker): F[Response[F]] =
parts.find(isValidPart) match {
case Some(file) =>
val t0 = System.nanoTime()
val home = sys.env.getOrElse("DP_UPLOAD_DIR", "/tmp/data-parser/upload")
val filename = file.filename.getOrElse("filename")
val path = FSUtils.ensurePath(home, filename)
val writer = fs2.io.file.writeAll(path, blocker)
val response = file.body
.observe(writer)
.fold(0)(_ + _)
.map { size => // BUG: size produces wrong size?
// TODO: Execute sync task which process uploaded file and produces avro file (and deletes the uploaded file)
// For example (I have the following blocking effectful operation writen in Java):
// val data = Parser.fromString(path.toAbsolutePath.toString)
// data.persist()
// path.toFile.delete()
// TODO: Write to redis stats
redis.set(path.toAbsolutePath.toString, size.toString)
val parsedIn = (System.nanoTime() - t0) * 1e-9
ParsedResponse(
ReplayKey(path.toAbsolutePath.toString),
BytesProcessed(size),
ParsedIn(parsedIn.toFloat)
)
}
Ok(response)
case None => BadRequest()
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment