scottjmaddox

import time
import os

# Base32 encoding alphabet without 'I', 'L', 'O', 'U'
CROCKFORD_BASE32_ALPHABET = '0123456789ABCDEFGHJKMNPQRSTVWXYZ'

def ulid():
    # Generate 48-bit timestamp (time in milliseconds since Unix epoch)
    timestamp = time.time_ns() // 1_000_000
    encoded_time = ''

scottjmaddox

# Pytorch implementation of the outlier suppression loss described in the paper
# "Improving generalization by loss modification" by Michael Tetelman 
# https://openreview.net/forum?id=vHOO1lxggJ

from torch import nn
import torch.functional as F

def outlier_supression_loss(input, target):
    return F.softplus(F.nll_loss(F.log_softmax(input, dim=-1), target, reduction='none')).mean()

scottjmaddox

"""
The JAX implementation of the Linear Recurrent Unit (LRU) layer, from the paper
"Resurrecting Recurrent Neural Networks for Long Sequences".
http://arxiv.org/abs/2303.06349
"""

import jax
import jax.numpy as jnp
import numpy as np
parallel_scan = jax.lax.associative_scan

scottjmaddox

Me:

How do encoder-only NLP models differ from encoder-decoder models?

ChatGPT:

Encoder-only models are a type of natural language processing (NLP) model that only have an encoder component, whereas encoder-decoder models have both an encoder and a decoder component. The encoder is responsible for converting the input text into a compact representation, or encoding, that captures the essential information in the input. The decoder then takes this encoding and generates the output text based on it.

Encoder-only models are typically used for tasks where the input and output are the same, such as language translation or summarization. In these tasks, the encoder takes the input text and creates an encoding that captures the meaning of the text, and the decoder then uses this encoding to generate the output text.

scottjmaddox

-- Copyright (c) 2020 Scott J Maddox
--
-- Licensed under either the Apache License, Version 2.0
-- (see https://www.apache.org/licenses/LICENSE-2.0),
-- or the ZLib license (see https://opensource.org/licenses/Zlib),
-- at your option.

module Data.BinOpTree
  ( BinOpTree (..),
    allFromList,