This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| license: gpl-3.0 | |
| height: 2910 | |
| border: no |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # https://www.r-graph-gallery.com/283-the-hourly-heatmap.html | |
| library(ggplot2) | |
| library(dplyr) # easier data wrangling | |
| library(viridis) # colour blind friendly palette, works in B&W also | |
| library(Interpol.T) # will generate a large dataset on initial load | |
| library(lubridate) # for easy date manipulation | |
| library(ggExtra) # because remembering ggplot theme options is beyond me | |
| library(tidyr) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import hmac | |
| from hashlib import sha1 | |
| from django.conf import settings | |
| from django.http import HttpResponse, HttpResponseForbidden, HttpResponseServerError | |
| from django.views.decorators.csrf import csrf_exempt | |
| from django.views.decorators.http import require_POST | |
| from django.utils.encoding import force_bytes | |
| import requests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import requests | |
| import base64 | |
| from tqdm import tqdm | |
| master_json_url = 'https://178skyfiregce-a.akamaihd.net/exp=1474107106~acl=%2F142089577%2F%2A~hmac=0d9becc441fc5385462d53bf59cf019c0184690862f49b414e9a2f1c5bafbe0d/142089577/video/426274424,426274425,426274423,426274422/master.json?base64_init=1' | |
| base_url = master_json_url[:master_json_url.rfind('/', 0, -26) + 1] | |
| resp = requests.get(master_json_url) | |
| content = resp.json() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ''' | |
| Spider for IMDb | |
| - Retrieve most popular movies & TV series with rating of 8.0 and above that have at least 5 award nominations | |
| - Crawl next pages recursively | |
| - Follow the details pages of scraped films to retrieve more information of each film | |
| ''' | |
| from scrapy.contrib.spiders import CrawlSpider, Rule | |
| from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor | |
| from scrapy.selector import Selector |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| DROP TABLE if exists d_date; | |
| CREATE TABLE d_date | |
| ( | |
| date_dim_id INT NOT NULL, | |
| date_actual DATE NOT NULL, | |
| epoch BIGINT NOT NULL, | |
| day_suffix VARCHAR(4) NOT NULL, | |
| day_name VARCHAR(9) NOT NULL, | |
| day_of_week INT NOT NULL, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # If bucket is private | |
| AWS_ACCESS_KEY = 'your access key' | |
| AWS_SECRET_KEY = 'your secret key' | |
| AWS_REGION = 'your region' | |
| session = boto3.session.Session( | |
| aws_access_key_id=AWS_ACCESS_KEY, | |
| aws_secret_access_key=AWS_SECRET_KEY, | |
| region_name=AWS_REGION | |
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| use Mojolicious::Lite; | |
| use DBM::Deep; | |
| use Mojo::JWT; | |
| plugin 'Bcrypt'; | |
| plugin 'Minion' => {SQLite => 'minion.db'}; | |
| helper users => sub { state $db = DBM::Deep->new('users.db') }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # by rubo77: https://gist.github.com/rubo77/6f3abb42685237dfca16 | |
| # choose 5MB, 10MB, 20MB, 50MB, 100MB, 200MB, 512MB or 1GB | |
| BIG=5MB | |
| # choose the maximum download size for one recursive crawl, for example 100k or 2m | |
| CRAWL_MAX=100k | |
| IN="http://download.thinkbroadband.com/$BIG.zip http://google.de/?q=test http://www.bbc.co.uk http://www.theregister.co.uk http://en.wikipedia.org/wiki/cloud_computing" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ############################################################################## | |
| # Calendar Heatmap # | |
| # by # | |
| # Paul Bleicher # | |
| # an R version of a graphic from: # | |
| # http://stat-computing.org/dataexpo/2009/posters/wicklin-allison.pdf # | |
| # requires lattice, chron, grid packages # | |
| ############################################################################## | |
| ## calendarHeat: An R function to display time-series data as a calendar heatmap |