This is a Docker compose definition of a Hadoop + Spark + Livy cluster with one node only. You can use it for development or testing purposes.
Example usage:
docker compose build
docker compose up -d
docker exec -it /bin/bash
Jonny Li jonnyli1125
jonnyli1125
/ tictactoe.py
Last active
November 30, 2024 06:31
Self-play Reinforcement Learning (Q-Learning) for Tic Tac Toe in 100 lines of code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import argparse | |
| import random | |
| from collections import defaultdict | |
| import numpy as np | |
| from tqdm import tqdm | |
| BOARD_SIZE = 3 | |
jonnyli1125
/ neural_network_from_scratch.ipynb
Last active
July 22, 2024 00:39
neural_network_from_scratch.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
jonnyli1125
/ rnn_transducer.py
Last active
October 9, 2023 17:56
RNN Transducer in ~100 lines of NumPy code. Paper: https://arxiv.org/abs/1211.3711
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from dataclasses import dataclass | |
| import numpy as np | |
| vocab = [None, 'a', 'b', 'c'] | |
| null_idx = 0 | |
| V = len(vocab) | |
| @dataclass | |
| class LSTMWeights: | |
| xi: list[list[float]] # (V-1, V) |
jonnyli1125
/ .spark-cluster.md
Last active
September 16, 2023 20:28
One node Hadoop, Spark, Livy cluster setup with Docker compose that uses the official images for apache/hadoop and apache/spark.
jonnyli1125
/ preprocess_lang8.py
Last active
October 20, 2021 02:04
Preprocessing script for Japanese entries in NAIST Lang8 Corpus
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import argparse | |
| import json | |
| import re | |
| import unicodedata | |
| invalid_bytes_re = re.compile(r'[\x00-\x1F]+') | |
| sline_re = re.compile(r'\[sline\].*?\[/sline\]') | |
| color_tags = ['[f-blue]','[/f-blue]', |
jonnyli1125
/ stockfish_selfplay.py
Created
June 21, 2021 04:23
Stockfish self-play game generation script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import argparse | |
| import chess | |
| import chess.engine | |
| import chess.pgn | |
| def main(stockfish_path, output_path): | |
| engine = chess.engine.SimpleEngine.popen_uci(stockfish_path) |
jonnyli1125
/ weighted_sparse_categorical_crossentropy.py
Last active
November 4, 2025 14:40
SparseCategoricalCrossentropy with class weights for Keras/Tensorflow 2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Since Model.fit doesn't support class_weight when using multiple outputs, | |
| this custom loss subclass may be useful. | |
| Relevant issues: | |
| https://github.com/keras-team/keras/issues/11735 | |
| https://github.com/tensorflow/tensorflow/issues/40457 | |
| https://github.com/tensorflow/tensorflow/issues/41448 | |
| """ |
jonnyli1125
/ custom.css
Last active
June 19, 2020 04:50
japanese font solution for discord with betterdiscord custom css
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| div>*:not(code) { | |
| /* using the default discord theme's fonts */ | |
| font-family: | |
| Whitney, | |
| 'Helvetica Neue', | |
| Helvetica, | |
| Arial, | |
| 'Meiryo', /* you can change this to whatever japanese font you prefer */ | |
| sans-serif !important; | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| file_export_version=3.0 | |
| /instance/org.codehaus.groovy.eclipse.ui/groovy.editor.groovyDoc.keyword.enabled=true | |
| /instance/org.codehaus.groovy.eclipse.ui/groovy.editor.groovyDoc.link.enabled=true | |
| /instance/org.codehaus.groovy.eclipse.ui/groovy.editor.groovyDoc.tag.enabled=true | |
| /instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionForeground.SystemDefault=false | |
| /instance/org.eclipse.ui.editors/AbstractTextEditor.Color.SelectionBackground.SystemDefault=false | |
| /instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Background.SystemDefault=false | |
| /instance/org.eclipse.ui.editors/AbstractTextEditor.Color.Foreground.SystemDefault=false | |
| /instance/org.epic.perleditor/AbstractTextEditor.Color.Background.SystemDefault=false | |
| /instance/org.epic.perleditor/AbstractTextEditor.Color.Foreground.SystemDefault=false |
jonnyli1125
/ BinarySearch.cs
Last active
December 12, 2015 01:18
A little test of using a binary search algorithm to find indexes of values in a sorted list. Also just a little comparison between binary and linear/sequential searching.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System; | |
| using System.Collections.Generic; | |
| namespace BinarySearch | |
| { | |
| class Program | |
| { | |
| static void Main(string[] args) | |
| { | |
| List<int> list = new List<int>() { 5, 8, 900, 1, 22, 52, 567, 666, 123, 256, 765, 0, 500 }; // example list |
NewerOlder