Skip to content

Instantly share code, notes, and snippets.

@jiqiujia
jiqiujia / lu_pivot.py
Last active November 17, 2017 08:06
numerical linear algebra
#from https://rosettacode.org/wiki/LU_decomposition#Python
from pprint import pprint
def matrixMul(A, B):
TB = zip(*B)
return [[sum(ea*eb for ea,eb in zip(a,b)) for b in TB] for a in A]
def pivotize(m):
"""Creates the pivoting matrix for m."""
n = len(m)
@jiqiujia
jiqiujia / data_loader.py
Created January 21, 2018 06:07 — forked from kevinzakka/data_loader.py
Train, Validation and Test Split for torchvision Datasets
# This is an example for the CIFAR-10 dataset.
# There's a function for creating a train and validation iterator.
# There's also a function for creating a test iterator.
# Inspired by https://discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/4
from utils import plot_images
def get_train_valid_loader(data_dir,
batch_size,
augment,
@jiqiujia
jiqiujia / finetune.py
Created January 21, 2018 06:07 — forked from panovr/finetune.py
Fine-tuning pre-trained models with PyTorch
import argparse
import os
import shutil
import time
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim
@jiqiujia
jiqiujia / bpe.py
Last active October 13, 2018 03:02
nlp
###byte pair encoding
###Neural Machine Translation of Rare Words with Subword Units
###from https://plmsmile.github.io/2017/10/19/subword-units/
import re
def process_raw_words(words, endtag='-'):
'''把单词分割成最小的符号,并且加上结尾符号'''
vocabs = {}
for word, count in words.items():
# 加上空格
word = re.sub(r'([a-zA-Z])', r' \1', word)
@jiqiujia
jiqiujia / mapping.json
Last active September 12, 2018 03:50
elasticsearch demo
{
"mappings": {
"docs": {
"dynamic": true,
"properties": {
"time": {
"type": "date",
"format": "yyyyMMdd",
"store": "true"
},
@jiqiujia
jiqiujia / gist:4fe14917fbfb5740667de08901b90e6d
Created November 1, 2018 02:33 — forked from CristinaSolana/gist:1885435
Keeping a fork up to date

1. Clone your fork:

git clone [email protected]:YOUR-USERNAME/YOUR-FORKED-REPO.git

2. Add remote from original repository in your forked repository:

cd into/cloned/fork-repo
git remote add upstream git://github.com/ORIGINAL-DEV-USERNAME/REPO-YOU-FORKED-FROM.git
git fetch upstream
@jiqiujia
jiqiujia / Running.txt
Created November 20, 2018 06:08 — forked from caiorss/Running.txt
C++ Scala JNI - Java Native Interface - Example - Calling C++ from Scala / Java
Compile the C++ code creating a shared library (or shared object in UNIX)
$ clang++ TestJNI.cpp -o libTestJNI.so -fPIC -shared -std=c++11 -I$HOME/opt/java/include -I$HOME/opt/java/include/linux
Run the application
$ scala -save load.scala
dir = /home/archbox/opengl/jni/libTestJNI.so
Hello world java
i = 0
@jiqiujia
jiqiujia / parallel.py
Created November 23, 2018 08:48 — forked from thomwolf/parallel.py
Data Parallelism in PyTorch for modules and losses
##+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
## Created by: Hang Zhang, Rutgers University, Email: [email protected]
## Modified by Thomas Wolf, HuggingFace Inc., Email: [email protected]
## Copyright (c) 2017-2018
##
## This source code is licensed under the MIT-style license found in the
## LICENSE file in the root directory of this source tree
##+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"""Encoding Data Parallel"""
@jiqiujia
jiqiujia / scel2txt.py
Created December 4, 2018 11:52
搜狐词库scel格式转txt
# -*- coding: utf-8 -*-
import struct
import os
# 由于原代码不适用python3且有大量bug
# 以及有函数没有必要使用且一些代码书写不太规范或冗余
# 所以本人在原有的大框架基本不动的情况下作了大量的细节更改。
# 使得没有乱码出现,文件夹导入更方便等等。
# Author:Ling Yue, Taiyuan U of Tech
@jiqiujia
jiqiujia / lsh.py
Created March 12, 2019 03:30 — forked from alexklibisz/lsh.py
ElastiK-Nearest-Neighbors LSH Example
import numpy as np
def make_lsh_model(nb_tables, nb_bits, nb_dimensions, vector_sample):
# vector_sample: np arr w/ shape (2 * nb_tables * nb_tables, nb_dimensions).
# normals, midpoints: np arrs w/ shape (nb_bits, nb_dimensions)
# thresholds: np arrs w/ shape (nb_bits)
# all_normals, all_thresholds: lists w/ one normal, one threshold per table.
all_normals, all_thresholds = [], []
for i in range(0, len(vector_sample), 2 * nb_bits):
vector_sample_a = vector_sample[i:i + nb_bits]