Marko Milenković milenkovicm

Using DBRX with PySpark AI

This markdown shows a quick example of how to use Databricks DBRX to generate and run a transform query against a sammple dataset.

Requirements

Install the following

Configure and install databricks-cli
pip install langchain langchain-community mlflow setuptools

Applied Functional Programming with Scala - Notes

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x

References from my talk 'Without Resilience, Nothing Else Matters'

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

slow

This bash script offers quick shortcuts to simulate slower network connections. It is useful when you need to simulate a wireless network on a Linux network server, especially when you are using a virtual machine guest on your local machine or in the cloud.

slow 3G                   # Slow network on default eth0 down to 3G wireless speeds
slow reset                # Reset connection for default eth0 to normal
slow vsat --latency=500ms # Simulate satellite internet  with a high latency
slow dsl -b 1mbps         # Simulate DSL with a slower speed than the default

slow modem-56k -d eth0 # Simulate a 56k modem on the eth1 device. eth0 is unchanged.

	use std::{sync::Arc, time::Instant};

	use datafusion::{
	arrow::{array::RecordBatch, json::ArrayWriter},
	prelude::SessionContext,
	};
	use iceberg::{io::FileIO, table::StaticTable, TableIdent};
	use iceberg_datafusion::IcebergTableProvider;
	use lambda_runtime::{
	run, service_fn,

	// Please find the full, tested version in
	// https://github.com/influxdata/influxdb_iox/blob/fe155e15fb2ad166aee66b0458e63c24a8128dd4/query/src/exec/task.rs#L101-L118

	pub struct DedicatedExecutor {
	state: Arc<Mutex<State>>,
	}

	/// Runs futures (and any `tasks` that are `tokio::task::spawned` by
	/// them) on a separate Tokio Executor
	struct State {

	package flowcontrol

	import scala.concurrent.duration._

	import akka.actor.typed.ActorRef
	import akka.actor.typed.ActorSystem
	import akka.actor.typed.Behavior
	import akka.actor.typed.scaladsl.Behaviors

	/**

	// Just before switching jobs:
	// Add one of these.
	// Preferably into the same commit where you do a large merge.
	//
	// This started as a tweet with a joke of "C++ pro-tip: #define private public",
	// and then it quickly escalated into more and more evil suggestions.
	// I've tried to capture interesting suggestions here.
	//
	// Contributors: @r2d2rigo, @joeldevahl, @msinilo, @_Humus_,
	// @YuriyODonnell, @rygorous, @cmuratori, @mike_acton, @grumpygiant,

	ssize_t read(int fd, void *buf, size_t count);

	ssize_t write(int fd, const void *buf, size_t count);