Nguyễn Đăng Minh Lực lucndm

Overview

The steps below bootstrap an instance of airflow, configured to use the kubernetes airflow executor, working within a minikube cluster.

This guide works with the airflow 1.10 release, however will likely break or have unnecessary extra steps in future releases (based on recent changes to the k8s related files in the airflow source).

Prerequisites

Docker installed
Minikube installed and started

Personal Deep Learning Computer

	# -- coding: utf-8 --
	import scrapy
	from scrapy.contrib.linkextractors import LinkExtractor
	from scrapy.contrib.spiders import CrawlSpider, Rule
	from manga_scrapy.items import VideoItem

	#2sao.vn
	class SaoVnSpider(CrawlSpider):
	name = '2saovn'
	allowed_domains = ['2sao.vn']

	Tạo Http nhanh để download file
	python -m SimpleHTTPServer

	Tắt firewall
	iptables -F

	Mở lại firewall
	iptables-restore < /etc/sysconfig/iptables

	# https://stackoverflow.com/questions/3463930/how-to-round-the-minute-of-a-datetime-object-python/10854034#10854034

	def roundTime(dt=None, roundTo=60):
	"""Round a datetime object to any time laps in seconds
	dt : datetime.datetime object, default now.
	roundTo : Closest number of seconds to round to, default 1 minute.
	Author: Thierry Husson 2012 - Use it as you want but don't blame me.
	"""
	if dt == None : dt = datetime.datetime.now()
	seconds = (dt - dt.min).seconds

	using System;
	using System.Collections.Generic;
	using System.IO;
	using System.Linq;
	using System.Reflection;
	using log4net;

	namespace LichPhatSong.Scraper.Helpers
	{
	public static class ReflectionHelper

	os.environ["SPARK_HOME"] = "/opt/spark-2.0.0-bin-hadoop2.7/"
	os.environ["PYSPARK_PYTHON"] = "/home/lucndm/envs/spark/bin/python"

	http://stackoverflow.com/questions/34685905/how-to-link-pycharm-with-pyspark
	http://i.stack.imgur.com/LwAJ6.png