ChrisLundquist

Hypercomplex Structure in Transformer Superposition

Experimental investigation of whether trained neural networks implicitly learn weight matrices that decompose into hypercomplex-like algebraic structures (quaternion, Clifford algebra, Kronecker-factored).

Key Findings

Experiment 3 (The Critical Test): Kronecker vs SVD Surgery

Baseline GPT-2 small perplexity: 29.88 (WikiText-2 test, standard strided evaluation)

ChrisLundquist

#!/usr/bin/env ruby
# The number, 197, is called a circular prime because all rotations of the digits: 197, 971, and 719, are themselves prime.
# There are thirteen such primes below 100: 2, 3, 5, 7, 11, 13, 17, 31, 37, 71, 73, 79, and 97.
#How many circular primes are there below one million?
#
require 'prime'

candidates = [2] + Prime.each(1_000_000).reject { |p| p.digits.any? { |d| d.even? } }

def siblings(int)

ChrisLundquist

//
//  fs_capabilities.m
//
//  Created by Charles Francoise on 13/05/14.
//  Copyright (c) 2014 Charles Francoise. All rights reserved.
//

#include <stdio.h>
#include <unistd.h>
#include <errno.h>

ChrisLundquist

//
//  fs_capabilities.m
//
//  Created by Charles Francoise on 13/05/14.
//  Copyright (c) 2014 Charles Francoise. All rights reserved.
//

#include <stdio.h>
#include <unistd.h>
#include <errno.h>

ChrisLundquist

#!/usr/bin/env ruby

def find_pair!(data, target)
  # We know we won't use a value greater than our target
  data.reject! { |i| i > target }
  
  # Sort it so we don't have to scan our list a bunch
  data.sort!
  
  # If our two biggest numbers are lower than the target, we're boned

ChrisLundquist

#!/usr/bin/env bash
wget http://www.chakoteya.net/NextGen/{101..277}.htm

ChrisLundquist

$ ruby -v
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]

Rehearsal --------------------------------------------------------------
map_compact                  0.020000   0.000000   0.020000 (  0.015315)
map_inplace_compact          0.010000   0.000000   0.010000 (  0.016069)
reject_map                   0.010000   0.000000   0.010000 (  0.014038)
reject_inplace_map           0.860000   0.010000   0.870000 (  0.860999)
reject_map_inplace           0.010000   0.000000   0.010000 (  0.013710)
reject_inplace_map_inplace   0.860000   0.000000   0.860000 (  0.866945)

ChrisLundquist

diff --git a/configure.ac b/configure.ac
index f713ce1..94b986c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2552,8 +2552,8 @@ AC_ARG_WITH(libmongoc, [AS_HELP_STRING([--with-libmongoc@<:@=PREFIX@:>@], [Path
         with_libmongoc="no"
  else
         with_libmongoc="yes"
-        LIBMONGOC_CPPFLAGS="$LIBMONGOC_CPPFLAGS -I$withval/include"
-        LIBMONGOC_LDFLAGS="$LIBMONGOC_LDFLAGS -L$withval/lib"

ChrisLundquist

Serves 4

• 1/4 cup white sugar
• 2 tablespoons fish sauce
• 2 tablespoons lime juice
• 2 tablespoon tamarind concentrate (mixed with 1/4 cup water)
• 1/2 (12 ounce) package dried rice noodles
• 1/4 cup vegetable oil
• 3/4 teaspoon minced garlic
• 2 eggs

ChrisLundquist

Do you speak English?

There I was in Seattle, just walking out from one of my favorite coffee shops. As soon as I walk out, a stranger makes eye contact and asks a very simple question. Do you speak English?

I was a little confused by the question. People were speaking English all around him. Why did he ask me? I answered, "Yes".

He then proceeded to tell me a story about how he needs bus fare to get to Portland.