Avinash avinash-mishra

S3 Syncing

Export out all your S3 data
Source: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

# sync from bucket to local
aws s3 sync <bucket> <target_folder> <options>
aws s3 sync s3://mybucket . --acl public-read

aws s3 sync --region ap-northeast-1 s3://[移動元バケット名] s3://[移動先バケット名]

GENERATORS:

Text Content Generator - http://www.lipsum.com
Favicon Generator - http://tools.dynamicdrive.com/favicon
Data Generator - https://mockaroo.com/
Mobile Mockup Generator - https://mockuphone.com
Logo Generator - https://www.logaster.com
UUID Generator - https://www.uuidgenerator.net/
Hash Generator - https://passwordsgenerator.net/sha256-hash-generator/
Ultimate Code Generator - https://webcode.tools/

Metadata

#Selecting a database	
USE database;	USE database;

#Listing databases	
SHOW DATABASES;	SHOW DATABASES;

	##################################################################
	## Native hdfs access (only on the cluster)
	# conda install -c conda-forge libhdfs3=2.3.0=1 hdfs3 --yes
	import hdfs3
	import pandas as pd

	nameNodeHost = 'hadoopnn1.localdomain'
	nameNodeIPCPort = 8020
	hdfs = hdfs3.HDFileSystem(nameNodeHost, port=nameNodeIPCPort)

	The Zen of Python, by Tim Peters

	Beautiful is better than ugly.
	Explicit is better than implicit.
	Simple is better than complex.
	Complex is better than complicated.
	Flat is better than nested.
	Sparse is better than dense.
	Readability counts.
	Special cases aren't special enough to break the rules.

	"""Diagnose script for checking OS/hardware/python/pip/mxnet/network.
	The output of this script can be a very good hint to issue/problem.
	"""
	import platform, subprocess, sys, os
	import socket, time
	try:
	from urllib.request import urlopen
	from urllib.parse import urlparse
	except ImportError:
	from urlparse import urlparse

	# See official docs at https://dash.plotly.com
	# pip install dash pandas

	from dash import Dash, dcc, html, Input, Output
	import plotly.express as px

	import pandas as pd

	df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/gapminderDataFiveYear.csv')

	import re
	import nltk

	sent_detector = nltk.RegexpTokenizer(u'[^　！？。]*[！？。.\n]')

	sents = sent_detector.tokenize(u"　原子番号９２のウランより重い元素は全て人工的に合成され、１１８番まで発見の報告がある。\
	１１３番については、理研と米露の共同チームがそれぞれ「発見した」と報告し、国際純正・応用化学連合と国際純粋・応用物理学連合の合同作業部会が審査していた。両学会は「データの確実性が高い」ことを理由に、理研の発見を認定し、３１日に森田さんに通知した。未確定だった１１５番と１１７番、１１８番の新元素は米露チームの発見を認めた。森田さんは「周期表に名前が残ることは感慨深い。大勢の共同研究者にまずは感謝したい」と述べた。 \n")

	for s in sents:
	print s, len(s)

	import time
	from concurrent.futures import ThreadPoolExecutor
	from logging import StreamHandler, Formatter, INFO, getLogger


	def init_logger():
	handler = StreamHandler()
	handler.setLevel(INFO)
	handler.setFormatter(Formatter("[%(asctime)s] [%(threadName)s] %(message)s"))
	logger = getLogger()

	import io
	import logging

	logger = logging.getLogger("your_logger")
	logger.setLevel(logging.INFO)

	logger_output = io.StringIO()
	logger.addHandler(logging.StreamHandler(logger_output))

	# do stuff that triggers some logs