Skip to content

Instantly share code, notes, and snippets.

View dbousamra's full-sized avatar

Dominic Bou-Samra dbousamra

  • Piccolo Health
  • Australia
View GitHub Profile
package mq.ironmq
import java.util
import java.util.concurrent.{ExecutorService, Executors}
import com.typesafe.scalalogging.LazyLogging
import io.iron.ironmq.{Messages, Client}
import mq._
import scala.util.Try
import scalaz._
import Scalaz._
package mq.ironmq
import java.util
import java.util.concurrent.{ExecutorService, Executors}
import com.typesafe.scalalogging.LazyLogging
import io.iron.ironmq.{Client, Messages}
import mq._
import scalaz._
import Scalaz._
import scalaz.concurrent._
object Extras {
implicit class ConcurrentProcess[O](val process: Process[Task, O]) {
def concurrently[O2](concurrencyLevel: Int)(f: Channel[Task, O, O2]): Process[Task, O2] = {
val actions = process.zipWith(f)((data, f) => f(data))
val nestedActions = actions.map(Process.eval)
scalaz.stream.merge.mergeN(concurrencyLevel)(nestedActions)
}
// Library supports bulk receives of up to 100. I want to be able to pass 750 in, and issue 8 bulk queries.
// That idea of sort of doing it overcoming the limit of 100 is what I want to abstract.
// Every bulk operation supports 100, but I'd like to do more.
// Ideas? Some smaple code (I think receive might be incorrect - not working 100%, but yet to debug)
def consume[A](fetchCount: Int)(f: MessageContainer => AckResponse): Unit = {
val messages = receive(Nil, 1000, fetchCount)
}
package com.cammy.firehose.uploader
import java.io.File
import java.text.DecimalFormat
import com.cammy.firehose.config.Configuration
import com.cammy.firehose.ftp.user.impl.CameraDetails
import com.cammy.firehose.uploader.domain._
import com.cammy.firehose.uploader.imaging.Resizer
import com.cammy.firehose.uploader.storage.{StorageLocation, RemoteFilename, Container, Storage}
def getSnapshotForTable(key: SnapshotKey, tableName: TableName): Either[AmazonClientException, Option[Snapshot]] = ???
def getSnapshot(key: SnapshotKey): Either[AmazonClientException, Option[Snapshot]] = {
val fallbackTableName = tableNameFactory.tableConfig.fallbackTableName.map(PrefixOnlyTableName(_, ""))
val tableNames: List[TableName] = (List(Some(tableNameFactory.fromSnapshotKey(key))) :+ fallbackTableName).flatten
val futures = Future.sequence(tableNames.map(tableName => {
Future { getSnapshotForTable(key, tableName) }
}))
case class AzureStorage(storageAccount: CloudStorageAccount, blobClient: CloudBlobClient, implicit val ec: ExecutionContext) extends Storage {
def uploadFile(container: Container, data: Array[Byte], remoteFilename: RemoteFilename): AzureError \/ RemoteStorageUrl = {
for {
container <- getContainer(container)
uploadedBlob <- uploadBlob(container, data, remoteFilename)
url <- getURLFromBlob(uploadedBlob)
} yield RemoteStorageUrl(url)
}
private def getContainer(container: Container): Future[AzureError \/ CloudBlobContainer] = ???
package com.cammy.firehose.domain
import java.io.File
import java.nio.file.attribute.BasicFileAttributeView
import java.nio.file.{Paths, Files}
import org.joda.time.Instant
import scalaz._
import Scalaz._
sealed trait CameraImageTimestamp {
case class ImageParsingException(reason: String) extends Exception
sealed trait CameraImage {
def image: Image
}
case class FullRes(image: Image) extends CameraImage
case class StdRes(image: Image) extends CameraImage
case class Thumbnail(image: Image) extends CameraImage

Practical Property-Based Testing:

At first it can be tricky working out how exactly you test your code with properties. The good news is that there are some very obvious and useful patterns that can make writing generators and properties easy by mere-mortals. This talk aims to cover some of those patterns, and in doing so help both people working or struggling with property-based testing, and through that to also motivate the use of property-based testing for those still unfamiliar with the concept.

Pure Consensus in a World Full of Failure:

Distributed consensus protocols are notorious for doing people’s heads in. Machines fail and networks lag, which are unfortunate consequences of entropy. Without loss of generality we can assume that the machines despise us, our only hope is to enchant them with correctness proofs, and that we must somehow convince them that safety and liveness are our primary moral concerns.

This talk introduces a new Haskell implementation of Raft, deconstructed as a menag