HyShai

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

🏛 Lessons learned in 5 years in the Enterprise

I just hit my 5-year anniversary this week, and I've been reflecting on the past. Here's a collection of tip & tricks and lessons learned in the trenches creating Software for 5 years here at QL.
Disclaimer: I am not necessarily fully compliant with all these rules, I've just noticed them along the way.
Disclaimer 2: YMMV, these have worked well for me, but we're all different.
Disclaimer 3: We already have some really good ISM's and Tech Philosophies, this is just an addition on those.

🏆 Criticize in private, Praise in public

Although a Team Member (TM) may have screwed up, there's never a good reason to call out a specific TM for screwing up in a public setting. No one like being called out for mistakes. On the other hand, when a TM does something great, shout it from the rooftops.

Ever wanted to delete all your likes/favorites from Twitter but only found broken/expensive tools? You are in the right place.

Go to: https://twitter.com/{username}/likes
Open the console and run the following JavaScript code:

setInterval(() => {
  for (const d of document.querySelectorAll('div[data-testid="unlike"]')) {
    d.click()
 }

How to setup a practically free CDN using Backblaze B2 and Cloudflare

⚠️ Note 2023-01-21
Some things have changed since I originally wrote this in 2016. I have updated a few minor details, and the advice is still broadly the same, but there are some new Cloudflare features you can (and should) take advantage of. In particular, pay attention to Trevor Stevens' comment here from 22 January 2022, and Matt Stenson's useful caching advice. In addition, Backblaze, with whom Cloudflare are a Bandwidth Alliance partner, have published their own guide detailing how to use Cloudflare's Web Workers to cache content from B2 private buckets. That is worth reading,

So there were a few threads going around recently about a challenge to write the longest sequence of keywords in Javascript:

There are, however, a few problems:

These solutions use non-keyword tokens (null, true, false are actually Literals, not Keywords)
One of the solutions isn't quite valid (new super)

	# install packages
	# pip install faiss-cpu sentence-transformers

	import numpy as np
	import torch
	import faiss
	import time
	from sentence_transformers import SentenceTransformer

	# https://www.sbert.net/docs/pretrained_models.html#multi-qa-models

	// Click Rules > Customize Rules. Scroll to OnBoot() and inside the function add:

	static function OnBoot() {
	FiddlerApplication.oProxy.add_DetachedUnexpectedly(DoReattach);
	//...


	// Just before the OnBoot function, add the following new functions:
	static function DoReattach(o: Object, ea: EventArgs) {
	FiddlerObject.UI.sbpInfo.Text = "Scheduling automatic reattach at " + new Date();

	// Inspired by https://twitter.com/coderitual/status/1112297299307384833 and https://tapajyoti-bose.medium.com/7-killer-one-liners-in-javascript-33db6798f5bf

	// Remove any duplicates from an array of primitives.
	const unique = [...new Set(arr)]

	// Sleep in async functions. Use: await sleep(2000).
	const sleep = (ms) => (new Promise(resolve => setTimeout(resolve, ms)));
	// or
	const sleep = util.promisify(setTimeout);

	// First, clone the word list project here: https://github.com/dwyl/english-words.git
	// Then save this file in the same folder and run!

	open System.IO

	// Ok... Here's how it works...

	// First, we load all the words from the Moby text fil
	let allTheWords = File.ReadAllLines("words.txt")

	open FSharp.Data

	type Pizza = JsonProvider<"pizzas.json">

	type ToppingFrequencyPair = { Toppings: string[]; Frequency: int }

	let getToppingFrequencyPair grouping =
	{ Toppings = fst grouping; Frequency = snd grouping \|> Array.length }

	[<EntryPoint>]