akhan619

Exploring Tokenizers from Hugging Face

Hugging Face (HF) has made NLP (Natural Language Processing) a breeze. In this post, we are going to take a look at tokenization using a hands on approach with the help of the Tokenizers library. We are going to load a real world dataset containing 10-K filings of public firms and see how to train a tokenizer from scratch based on the BERT tokenization scheme. In the process we will understand tokenization in detail and some gotchas to keep an eye out for.

Background on NLP (Optional)

If you already have an understanding of the NLP pipeline, you can safely skip this section.

For any NLP task, one of the first steps is pre-processing the data so that it can be fed into our NLP models. For those new to NLP, the general pipeline for any NLP task (text classification, question answering, etc.) is as follows:

manuelmazzuola

package main

import (
    "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/client-go/discovery"
    "k8s.io/client-go/discovery/cached/memory"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/restmapper"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/tools/clientcmd/api"

willprice

Install OpenCV 4.1.2 on Raspbian Buster

$ chmod +x *.sh
$ ./download-opencv.sh
$ ./install-deps.sh
$ ./build-opencv.sh
$ cd ~/opencv/opencv-4.1.2/build
$ sudo make install

mozillazg

from flask import Flask, render_template
from flask_paginate import Pagination, get_page_args


app = Flask(__name__)
app.template_folder = ''
users = list(range(100))


def get_users(offset=0, per_page=10):

hediet

type StringBool = "true"|"false";


interface AnyNumber { prev?: any, isZero: StringBool };
interface PositiveNumber { prev: any, isZero: "false" };

type IsZero<TNumber extends AnyNumber> = TNumber["isZero"];
type Next<TNumber extends AnyNumber> = { prev: TNumber, isZero: "false" };
type Prev<TNumber extends PositiveNumber> = TNumber["prev"];

simonw

How to recover lost Python source code if it's still resident in-memory

I screwed up using git ("git checkout --" on the wrong file) and managed to delete the code I had just written... but it was still running in a process in a docker container. Here's how I got it back, using https://pypi.python.org/pypi/pyrasite/ and https://pypi.python.org/pypi/uncompyle6

Attach a shell to the docker container

Install GDB (needed by pyrasite)

apt-get update && apt-get install gdb

joepie91

'use strict';

const parseExpression = require("./parse-expression");

function findAll(regex, target) {
	let results = [], match;
	
	while (match = regex.exec(target)) {
		results.push(match);
	}

markwallsgrove

package main

// https://godoc.org/github.com/inconshreveable/go-update

// This gist documents go-install using a SHA-256 checksum,
// elliptic curve (prime 256) encrypted signature & bsdiff
// formatted patch.

// These steps took me over a hour to figure. go-install currently
// doesn't include a tutorial or quick start guide, so I created this

shmup

Transmission Blocklist

The Transmission torrent client has an option to set a Blocklist, which helps protect you from getting caught and having the DMCA send a letter/email.

It's as simple as downloading and installing the latest client:

• https://transmissionbt.com/download/

abhishektomar

#!/bin/bash

# Checking whether user has enough permission to run this script
sudo -n true
if [ $? -ne 0 ]
    then
        echo "This script requires user to have passwordless sudo access"
        exit
fi