Steve Severance sseveran

How to calculate the alignment between BERT and spaCy tokens effectively and robustly

site: https://tamuhey.github.io/tokenizations/

Natural Language Processing (NLP) has made great progress in recent years because of neural networks, which allows us to solve various tasks with end-to-end architecture. However, many NLP systems still require language-specific pre- and post-processing, especially in tokenizations. In this article, I describe an algorithm that simplifies calculating correspondence between tokens (e.g. BERT vs. spaCy), one such process. And I introduce Python and Rust libraries that implement this algorithm. Here are the library and the demo site links:

repo: https://github.com/tamuhey/tokenizations

systemd for Administrators

Chronological list of the "systemd for Administrators" series published on 0pointer.net/blog:

	#! /usr/bin/env python
	# -- coding: utf-8 --
	# vim:fenc=utf-8
	#
	# Copyright © 2019 Pi-Yueh Chuang <[email protected]>
	#
	# Distributed under terms of the MIT license.

	"""An example of using tfp.optimizer.lbfgs_minimize to optimize a TensorFlow model.

	import com.github.tminglei.slickpg.ExPostgresProfile
	import slick.SlickException
	import slick.ast.ColumnOption.PrimaryKey
	import slick.ast.{ColumnOption, FieldSymbol, Insert, Node, Select}
	import slick.compiler.{InsertCompiler, Phase, QueryCompiler}
	import slick.dbio.{Effect, NoStream}
	import slick.jdbc.InsertBuilderResult
	import slick.lifted.Query

	// format: off

	var active = false;

	function changeRefer(details) {
	if (!active) return;

	for (var i = 0; i < details.requestHeaders.length; ++i) {
	if (details.requestHeaders[i].name === 'Referer') {
	details.requestHeaders[i].value = 'http://www.google.com/';
	break;
	}