DanNi0130
DanNi0130
/ reddit.js v3 output
Last active
August 1, 2018 17:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Russian Pipeline. Upvote so that this is the first image people see when they Google “Russian Pipeline” | |
| John F. Kennedy Jr. Sitting in the pilot seat of the Marine One circa 1963 | |
| I didn't take it as a compliment. | |
| How beautiful is this | |
| Hustle like Faye | |
| The power of a salt water crocodile's tail. | |
| I'm 36, and will be dead inside of a year. | |
| F***ing genius. | |
| TIL Anthony Daniels, who endured years of discomfort in the C-3PO costume, was so annoyed by Alan Tudyk (Rogue One) playing K-2SO in the comfort of a motion-capture suit that he cursed at Tudyk. Tudyk later joked that a "fuck you" from Daniels was among the highest compliments he had ever received. | |
| Reminder about the fact UC Davis paid over $100k to remove this photo from the internet. |
DanNi0130
/ browserless
Last active
August 2, 2018 23:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| docker pull browserless/chrome | |
| docker run -p 3000:3000 -e "MAX_CONCURRENT_SESSIONS=5" | |
| -e "MAX_QUEUE_LENGTH=0" | |
| -e "PREBOOT_CHROME=true" | |
| -e "TOKEN=YOURTOKEN" | |
| -e "ENABLE_DEBUGGER=false" | |
| -e "CONNECTION_TIMEOUT=300000" --restart always browserless/chrome |
DanNi0130
/ puppeteer_settings
Created
August 2, 2018 22:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // with browserless | |
| browser = await puppeteer.connect({ | |
| browserWSEndpoint: | |
| 'ws://' + | |
| browserless.ip + | |
| ':' + | |
| browserless.port + | |
| '?TOKEN=' + | |
| browserless.token + | |
| '&--proxy-server=' + proxy + |
DanNi0130
/ request_interception
Created
August 2, 2018 23:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const blockedResourceTypes = [ | |
| 'image', | |
| 'media', | |
| 'font', | |
| 'texttrack', | |
| 'object', | |
| 'beacon', | |
| 'csp_report', | |
| 'imageset', |
DanNi0130
/ page_settings
Created
August 2, 2018 23:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const blockedResourceTypes = [ | |
| 'image', | |
| 'media', | |
| 'font', | |
| 'texttrack', | |
| 'object', | |
| 'beacon', | |
| 'csp_report', | |
| 'imageset', | |
| ]; |
DanNi0130
/ proxy_server
Created
August 3, 2018 00:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const proxies = { | |
| 'useragent1': 'http://proxyusername1:proxypassword1@proxyhost1:proxyport1', | |
| 'useragent2': 'http://proxyusername2:proxypassword2@proxyhost2:proxyport2', | |
| 'useragent3': 'http://proxyusername3:proxypassword3@proxyhost3:proxyport3', | |
| }; | |
| const server = new ProxyChain.Server({ | |
| port: 8000, | |
| prepareRequestFunction: ({request}) => { | |
| const userAgent = request.headers['user-agent']; |
DanNi0130
/ requestsscrapeimport.py
Created
February 24, 2019 06:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import requests, bs4, re, time |
DanNi0130
/ requests output raw
Created
February 24, 2019 06:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| b'<html op="news"><head><meta name="referrer" content="origin"><meta name="viewport" content="width=device-width, initial-scale=1.0"><link rel="stylesheet" type="text/css" href="news.css?Swdnfjd2lvQXPAqH2Hs6">\n <link rel="shortcut icon" href="favicon.ico">\n <link rel="alternate" type="application/rss+xml" title="RSS" href="rss">\n <title>Hacker News</title></head><body><center><table id="hnmain" border="0" cellpadding="0" cellspacing="0" width="85%" bgcolor="#f6f6ef">\n <tr><td bgcolor="#ff6600"><table border="0" cellpadding="0" cellspacing="0" width="100%" style="padding:2px"><tr><td style="width:18px;padding-right:4px"><a href="https://news.ycombinator.com"><img src="y18.gif" width="18" height="18" style="border:1px white solid;"></a></td>\n <td style="line-height:12pt; height:10px;"><span class="pagetop"><b class="hnname"><a href="news">Hacker News</a></b>\n <a href="newest">new</a> | <a href="front">past</a> | <a href="newcomments">comments |
DanNi0130
/ requests output beautifulsoup
Created
February 24, 2019 06:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <html op="news"><head><meta content="origin" name="referrer"/><meta content="width=device-width, initial-scale=1.0" name="viewport"/><link href="news.css?Swdnfjd2lvQXPAqH2Hs6" rel="stylesheet" type="text/css"/> | |
| <link href="favicon.ico" rel="shortcut icon"/> | |
| <link href="rss" rel="alternate" title="RSS" type="application/rss+xml"/> | |
| <title>Hacker News</title></head><body><center><table bgcolor="#f6f6ef" border="0" cellpadding="0" cellspacing="0" id="hnmain" width="85%"> | |
| <tr><td bgcolor="#ff6600"><table border="0" cellpadding="0" cellspacing="0" style="padding:2px" width="100%"><tr><td style="width:18px;padding-right:4px"><a href="https://news.ycombinator.com"><img height="18" src="y18.gif" style="border:1px white solid;" width="18"/></a></td> | |
| <td style="line-height:12pt; height:10px;"><span class="pagetop"><b class="hnname"><a href="news">Hacker News</a></b> | |
| <a href="newest">new</a> | <a href="front">past</a> | <a href="newcomments">comments</a> | <a href="ask">ask</a> | <a href="show">show</a> | <a href="jobs"> |
DanNi0130
/ requestsscrapebs
Created
February 24, 2019 06:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cleanpagedata = bs4.BeautifulSoup(pagedata.text, 'html.parser') |