-
Kafka Binary files : http://kafka.apache.org/downloads.html
-
Atleast 2 AWS machines : AWS EMR or EC2 will be preferable
-
A Kafka Manager Utility to watch up the Cluster : https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /usr/bin/env python | |
# | |
# mysql2cassandra.py | |
# Dump a MySQL result set to file and then import into a Cassandra column family | |
# | |
# Configuration | |
# mysql_params [host, port, user, password, db] MySQL conenction parameters | |
# mysql_columns [colname, colname2, ...] Columns for building MySQL query | |
# The column that will hold values of the row key in the Cassandra column family must be first |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: | |
$ sudo apt-get update | |
.......... | |
.......... | |
Err:13 http://extras.ubuntu.com/ubuntu xenial/main Sources | |
404 Not Found [IP: 91.189.92.152 80] | |
Ign:17 http://extras.ubuntu.com/ubuntu xenial/main amd64 Packages | |
Ign:20 http://extras.ubuntu.com/ubuntu xenial/main i386 Packages | |
Ign:22 http://extras.ubuntu.com/ubuntu xenial/main all Packages | |
Ign:23 http://extras.ubuntu.com/ubuntu xenial/main Translation-en_US |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ | |
{ | |
"time": "03:53:25 AM", | |
"milliseconds_since_epoch": 1362196405309, | |
"date": "03-02-2013", | |
"name": "alphabet" | |
}, | |
{ | |
"time": "03:53:26 AM", | |
"milliseconds_since_epoch": 1362196405300, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"""A setuptools based setup module. | |
See: | |
https://packaging.python.org/en/latest/distributing.html | |
https://github.com/pypa/sampleproject | |
""" | |
# Always prefer setuptools over distutils | |
from setuptools import setup, find_packages | |
# To use a consistent encoding | |
from codecs import open |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Install dependencies before running : https://dash.plot.ly/installation | |
""" | |
import dash | |
from dash.dependencies import Input, Output | |
import dash_core_components as dcc | |
import dash_html_components as html | |
import pandas as pd | |
pd.core.common.is_list_like = pd.api.types.is_list_like |
- Export out all your S3 data
- Source: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
# sync from bucket to local
aws s3 sync <bucket> <target_folder> <options>
aws s3 sync s3://mybucket . --acl public-read
aws s3 sync --region ap-northeast-1 s3://[移動元バケット名] s3://[移動先バケット名]
- データ加工を無料で学べる「データサイエンス100本ノック」 データサイエンティスト協会がGitHubに公開:データサイエンスの「基礎的なデータ処理の修行場」として - @IT
- 「機械学習の最先端」を効率的に情報収集! おすすめのメルマガ3選:AI・機械学習の独学リソース - @IT
- データサイエンス初学者向け学習環境「データサイエンス100本ノック(構造化データ加工編)」が公開:CodeZine(コードジン)
- 機械学習で使われる評価関数まとめ - Qiita
- ML/DL Engineering Made Easy with PyTorch’s Ecosystem Tools | by PyTorch | PyTorch | Medium
- ライブラリーを使わずにPythonでニューラルネットワークを構築してみる - Qiita
- 深層強化学習と汎用化が重要、ロボットへの応用が面白い Googleブレイン Shane Gu氏の基調講演「Deep Learning Digital Conference」 | ロボスタ
- [新たな活性化関数「FReLU」誕生&解説! - Qiita](https://qiita.com/omiita/items/bfbba77559762405698