-
-
Save andrei-m/982927 to your computer and use it in GitHub Desktop.
| /* | |
| Copyright (c) 2011 Andrei Mackenzie | |
| Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: | |
| The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. | |
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. | |
| */ | |
| // Compute the edit distance between the two given strings | |
| exports.getEditDistance = function(a, b){ | |
| if(a.length == 0) return b.length; | |
| if(b.length == 0) return a.length; | |
| var matrix = []; | |
| // increment along the first column of each row | |
| var i; | |
| for(i = 0; i <= b.length; i++){ | |
| matrix[i] = [i]; | |
| } | |
| // increment each column in the first row | |
| var j; | |
| for(j = 0; j <= a.length; j++){ | |
| matrix[0][j] = j; | |
| } | |
| // Fill in the rest of the matrix | |
| for(i = 1; i <= b.length; i++){ | |
| for(j = 1; j <= a.length; j++){ | |
| if(b.charAt(i-1) == a.charAt(j-1)){ | |
| matrix[i][j] = matrix[i-1][j-1]; | |
| } else { | |
| matrix[i][j] = Math.min(matrix[i-1][j-1] + 1, // substitution | |
| Math.min(matrix[i][j-1] + 1, // insertion | |
| matrix[i-1][j] + 1)); // deletion | |
| } | |
| } | |
| } | |
| return matrix[b.length][a.length]; | |
| }; |
Also mhosen's implementation is incorrect. Provides distance between "house" and "house" as 5, should be 0.
mhosen's implementation is actually computing hamming distance and NOT Levenshtein.
This is great. Thanks!
If I use this in my own MIT licensed github project... then what do I need to do to attribute the code?
the exact same algorithm, condensed for your courtesy:
String.prototype.levenstein = function(string) {
var a = this, b = string + "", m = [], i, j, min = Math.min;
if (!(a && b)) return (b || a).length;
for (i = 0; i <= b.length; m[i] = [i++]);
for (j = 0; j <= a.length; m[0][j] = j++);
for (i = 1; i <= b.length; i++) {
for (j = 1; j <= a.length; j++) {
m[i][j] = b.charAt(i - 1) == a.charAt(j - 1)
? m[i - 1][j - 1]
: m[i][j] = min(
m[i - 1][j - 1] + 1,
min(m[i][j - 1] + 1, m[i - 1 ][j]))
}
}
return m[b.length][a.length];
}
@bdelespierre - Thanks - that is a useful snippet, although there is a slight mistake when compared with the original -- missed the + 1 for the deletion case -- here is your snippet updated with that small fix
String.prototype.levenstein = function(string) {
var a = this, b = string + "", m = [], i, j, min = Math.min;
if (!(a && b)) return (b || a).length;
for (i = 0; i <= b.length; m[i] = [i++]);
for (j = 0; j <= a.length; m[0][j] = j++);
for (i = 1; i <= b.length; i++) {
for (j = 1; j <= a.length; j++) {
m[i][j] = b.charAt(i - 1) == a.charAt(j - 1)
? m[i - 1][j - 1]
: m[i][j] = min(
m[i - 1][j - 1] + 1,
min(m[i][j - 1] + 1, m[i - 1 ][j] + 1))
}
}
return m[b.length][a.length];
}
Cache matrix for even faster performance.
Thanks for sharing :)
Having four loops was really bugging me.
Also, Math.min can take more than two numbers
var levenshtein = function(a, b){
if(!a || !b) return (a || b).length;
var m = [];
for(var i = 0; i <= b.length; i++){
m[i] = [i];
if(i === 0) continue;
for(var j = 0; j <= a.length; j++){
m[0][j] = j;
if(j === 0) continue;
m[i][j] = b.charAt(i - 1) == a.charAt(j - 1) ? m[i - 1][j - 1] : Math.min(
m[i-1][j-1] + 1,
m[i][j-1] + 1,
m[i-1][j] + 1
);
}
}
return m[b.length][a.length];
};
Here a version that only needs O(min(m,n)) memory, instead of the original's O(m*n):
var levenshtein = function(a, b) {
if(a.length == 0) return b.length;
if(b.length == 0) return a.length;
// swap to save some memory O(min(a,b)) instead of O(a)
if(a.length > b.length) {
var tmp = a;
a = b;
b = tmp;
}
var row = [];
// init the row
for(var i = 0; i <= a.length; i++){
row[i] = i;
}
// fill in the rest
for(var i = 1; i <= b.length; i++){
var prev = i;
for(var j = 1; j <= a.length; j++){
var val;
if(b.charAt(i-1) == a.charAt(j-1)){
val = row[j-1]; // match
} else {
val = Math.min(row[j-1] + 1, // substitution
prev + 1, // insertion
row[j] + 1); // deletion
}
row[j - 1] = prev;
prev = val;
}
row[a.length] = prev;
}
return row[a.length];
}Runtime should be the same. Code is also licensed as MIT, same as OP.
I did some test and it turns out caching Math.min or passing it more than 2 arguments was a huge preformance loss (60% slower on V8)
My guess is that v8 has some highly optimised Math.min that takes only 2 args and he isn't detecting it if we don't call min from Math.
Caching the .length of the strings was actualy slower, == vs === made no significative differences.
so after some iterations this was the fastests i was able to get :
const levenshtein = (a, b) => {
if (a.length === 0) return b.length
if (b.length === 0) return a.length
let tmp, i, j, prev, val, row
// swap to save some memory O(min(a,b)) instead of O(a)
if (a.length > b.length) {
tmp = a
a = b
b = tmp
}
row = Array(a.length + 1)
// init the row
for (i = 0; i <= a.length; i++) {
row[i] = i
}
// fill in the rest
for (i = 1; i <= b.length; i++) {
prev = i
for (j = 1; j <= a.length; j++) {
if (b[i-1] === a[j-1]) {
val = row[j-1] // match
} else {
val = Math.min(row[j-1] + 1, // substitution
Math.min(prev + 1, // insertion
row[j] + 1)) // deletion
}
row[j - 1] = prev
prev = val
}
row[a.length] = prev
}
return row[a.length]
}Which is mainly just milto-mirdita version with the 2 Math.min, hoisting the variables, and fixed length array Array(a.length + 1)
wierdly enough it does a significative difference.
still MIT
Here is the jsperf test of @kigiri's 50% faster solution with @rksm's row variable declaration.
Here's a JSBench of almost every implementation on this post. I managed to cut another 5% off kigiri's fastest in everything except firefox.
https://jsperf.com/levenshtein-distance-bench/1
Below is the fastest. It's based off of kirgiri's fastest but improves it by caching array lengths and reducing and reusing variables where possible.
edit: updated with fix from @pukahontas
function dziemba_levenshtein(a, b) {
var tmp;
if (a.length === 0) { return b.length; }
if (b.length === 0) { return a.length; }
if (a.length > b.length) { tmp = a; a = b; b = tmp; }
var i, j, res, alen = a.length, blen = b.length, row = Array(alen);
for (i = 0; i <= alen; i++) { row[i] = i; }
for (i = 1; i <= blen; i++) {
res = i;
for (j = 1; j <= alen; j++) {
tmp = row[j - 1];
row[j - 1] = res;
res = b[i - 1] === a[j - 1] ? tmp : Math.min(tmp + 1, Math.min(res + 1, row[j] + 1));
}
row[j - 1] = res;
}
return res;
}
Thanks for all the hard work guys, very helpful!
I'm afraid there is a bug in your code. For my example ("badbadnotgood", "s") it returned 2 instead of 13.
Have you seen this git?
JS-Levenshtein
$ npm run bench
655,992 op/s » js-levenshtein
542,796 op/s » leven
497,966 op/s » talisman
386,839 op/s » levenshtein-edit-distance
254,941 op/s » fast-levenshtein
69,857 op/s » levenshtein-component
21,688 op/s » levdist
24,631 op/s » ld
21,834 op/s » natural
13,984 op/s » levenshtein
Anyone know how to implement this in Javascript?
String Matching and Clustering
I'm thinking of the clustering part. So calculate the score I'm looking at this package. a javascript port of the popular Python package fuzzywuzzy
function fuzzyMatched (comparer, comparitor, matchCount) {
var isMatched = false;
a = comparer.trim().toLowerCase();
b = comparitor.trim().toLowerCase();
if(a.length == 0) return b.length;
if(b.length == 0) return a.length;
var matrix = [];
// increment along the first column of each row
var i;
for(i = 0; i <= b.length; i++){
matrix[i] = [i];
}
// increment each column in the first row
var j;
for(j = 0; j <= a.length; j++){
matrix[0][j] = j;
}
// Fill in the rest of the matrix
for(i = 1; i <= b.length; i++){
for(j = 1; j <= a.length; j++){
if(b.charAt(i-1) == a.charAt(j-1)){
matrix[i][j] = matrix[i-1][j-1];
} else {
matrix[i][j] = Math.min(matrix[i-1][j-1] + 1, // substitution
Math.min(matrix[i][j-1] + 1, // insertion
matrix[i-1][j] + 1)); // deletion
}
}
}
var fuzzyDistance = matrix[b.length][a.length];
var cLength = Math.max(a.length, b.length);
var score = 1.0 - (fuzzyDistance / cLength);
if (score > matchCount)
isMatched = true;
return isMatched;
}
In the above method youll have to pass two parameters and the maximum fuzzy match count, for example if you are passing 0.9 then the it will check match count 90% and gives you the result true or false.
fuzzyMatched("abc","abc1",0.7)
true
fuzzyMatched("abc","abc1",0.8)
false
String.prototype.levenstein = function(string) {
var a = this, b = string + "", m = [], i, j, min = Math.min;
if (!(a && b)) return (b || a).length;
for (i = 0; i <= b.length; m[i] = [i++]);
for (j = 0; j <= a.length; m[0][j] = j++);
for (i = 1; i <= b.length; i++) {
for (j = 1; j <= a.length; j++) {
m[i][j] = b.charAt(i - 1) == a.charAt(j - 1)
? m[i - 1][j - 1]
: m[i][j] = min(
m[i - 1][j - 1] + 1,
min(m[i][j - 1] + 1, m[i - 1 ][j]))
}
}
return m[b.length][a.length];
}
function findNearestMatchingSymbol(symbol, coins){
let combinations = []
for(let i = 0; i < symbol.length; i++){
for(let j = i + 1; j <= symbol.length; j++){
combinations.push(symbol.substring(i,j))
}
}
let scores = []
coins.forEach(coin=>{
for(let current of combinations){
if(coin.includes(current)){
scores.push({
symbol: current,
coin: coin,
score: current.levenstein(coin)
})
}
}
})
scores.sort((a,b)=>a.score - b.score)
console.log(scores)
}
findNearestMatchingSymbol('BTR', coins.map(i=>i.symbol))
This gets all the close symbols but takes too many iterations
Providing that string is shorter than 128 chars, this version based on kigiri's is ~25% faster (chrome 63):
const levenshtein = (a,b)=>{
let alen = a.length;
let blen = b.length;
if (alen === 0) return blen;
if (blen=== 0) return alen;
let tmp, i, j, prev, val, row, ma, mb, mc, md, bprev;
if (alen> blen) {
tmp = a;
a = b;
b = tmp;
}
row = new Int8Array(alen+1);
// init the row
for (i = 0; i <= alen; i++) {
row[i] = i;
}
// fill in the rest
for (i = 1; i <= blen; i++) {
prev = i;
bprev = b[i - 1]
for (j = 1; j <= alen; j++) {
if (bprev === a[j - 1]) {
val = row[j-1];
} else {
ma = prev+1;
mb = row[j]+1;
mc = ma - ((ma - mb) & ((mb - ma) >> 7));
md = row[j-1]+1;
val = mc - ((mc - md) & ((md - mc) >> 7));
}
row[j - 1] = prev;
prev = val;
}
row[alen] = prev;
}
return row[alen];
}
@teleranek, your solution is broken as well.
levenshtein('helli', 'elli') returns 3, while levenshtein('elli', 'helli') returns 1
👏 Awesome!🤟
@DerekZiemba
Found the error in your algorithm as pointed out by @jonha892
It was missing the last write of the result to row so that value wasn't available the next iteration.
function dziemba_levenshtein(a, b){
var tmp;
if (a.length === 0) { return b.length; }
if (b.length === 0) { return a.length; }
if (a.length > b.length) { tmp = a; a = b; b = tmp; }
var i, j, res, alen = a.length, blen = b.length, row = Array(alen);
for (i = 0; i <= alen; i++) { row[i] = i; }
for (i = 1; i <= blen; i++) {
res = i;
for (j = 1; j <= alen; j++) {
tmp = row[j - 1];
row[j - 1] = res;
res = Math.min(tmp + (b[i - 1] !== a[j - 1]), res + 1, row[j] + 1);
}
row[j - 1] = res; // This was the missing line
}
return res;
}
Hey i wanna ask, if it is okay for you if I use your code in an own npm package, where i calculate the character-error-rate for ASR.
I can also give you credits on the code
@henning410 Sure! its MIT licensed, you don't have to ask for permission as long as you stick to the license terms.
boa noite consigo rodar bot mais nao consigo fazer nenhuma da funçoes rodar
Here's a fresh TypeScript version of it: https://gist.github.com/felixeichler/53c48ec5c398e6a01d675c42d430e7c6
@dotnetCarpenter I've used this as part of a simple suggestion system. It works by calculating edit distances between some input string and all known acceptable inputs. If the edit distance of any is within an acceptable range, the system can offer a suggestion - e.g. input 'JvaaScript' - "Did you mean 'JavaScript'?"