bradmontgomery

Web Scraping Workshop

Using Requests and Beautiful Soup, with the most recent Beautiful Soup 4 docs.

Getting Started

Install our tools (preferably in a new virtualenv):

pip install beautifulsoup4

johntyree

#!/usr/bin/env sh

# Download lists, unpack and filter, write to stdout
curl -s https://www.iblocklist.com/lists.php \
  | sed -n "s/.*value='\(http:.*=bt_.*\)'.*/\1/p" \
  | xargs wget -O - \
  | gunzip \
  | egrep -v '^#'

elssar

#!/usr/bin/env python

"""
Download all the pdfs linked on a given webpage

Usage -

    python grab_pdfs.py url <path/to/directory>
        url is required
        path is optional. Path needs to be absolute

willurd

Each of these commands will run an ad hoc http static server in your current (or specified) directory, available at http://localhost:8000. Use this power wisely.

Discussion on reddit.

Python 2.x

$ python -m SimpleHTTPServer 8000

dypsilon

Backend Development

• node.js
  • Installation paths: use one of these techniques to install node and npm without having to sudo.
  • Node.js HOWTO: Install Node+NPM as user (not root) under Unix OSes
  • Felix's Node.js Guide
  • Creating a REST API using Node.js, Express, and MongoDB
  • Node Cellar Sample Application with Backbone.js, Twitter Bootstrap, Node.js, Express, and MongoDB
  • JavaScript Event Loop
• Node.js for PHP programmers

jed

How to set up stress-free SSL on an OS X development machine

One of the best ways to reduce complexity (read: stress) in web development is to minimize the differences between your development and production environments. After being frustrated by attempts to unify the approach to SSL on my local machine and in production, I searched for a workflow that would make the protocol invisible to me between all environments.

Most workflows make the following compromises:

• Use HTTPS in production but HTTP locally. This is annoying because it makes the environments inconsistent, and the protocol choices leak up into the stack. For example, your web application needs to understand the underlying protocol when using the secure flag for cookies. If you don't get this right, your HTTP development server won't be able to read the cookies it writes, or worse, your HTTPS production server could pass sensitive cookies over an insecure connection.

• Use production SSL certificates locally. This is annoying

hgfischer

Benchmarking Nginx with Go

There are a lot of ways to serve a Go HTTP application. The best choices depend on each use case. Currently nginx looks to be the standard web server for every new project even though there are other great web servers as well. However, how much is the overhead of serving a Go application behind an nginx server? Do we need some nginx features (vhosts, load balancing, cache, etc) or can you serve directly from Go? If you need nginx, what is the fastest connection mechanism? This are the kind of questions I'm intended to answer here. The purpose of this benchmark is not to tell that Go is faster or slower than nginx. That would be stupid.

So, these are the different settings we are going to compare:

• Go HTTP standalone (as the control group)
• Nginx proxy to Go HTTP
• Nginx fastcgi to Go TCP FastCGI
• Nginx fastcgi to Go Unix Socket FastCGI

rbranson

Rust is the first language that has emerged in the past few years that solves enough of my problems that it would be worth not only learning & teaching an entirely new language, but also sacrificing the maturity of the language ecosystems I’ve become accustomed to.

I highly suggest you read the "Guide" provided by the language developers or this won't make much sense. These are just some of my thoughts and are intended to highlight particular things that stand out to me. I am just a practitioner and not an expert in any of these languages, so I have probably made some incorrect assumptions and out-of-date assertions. Bare with me.

Rust feels like the first time momentum has gained behind a true systems programming language that uses modern PL design techniques to prevent common errors when dealing with memory. It seems like others have previously either been too anemic to be worth adopting or too abstract to provide proper control. The type system and assignment semantics are designed specifically to preven

ericelliott

A Guide to Functional Programming Lingo for JavaScripters

Functional programming gets a bad wrap about being too hard for mere mortals to comprehend. This is nonsense. The concepts are actually quite simple to grasp.

The jargon is the hardest part. A lot of that vocabulary comes from a specialized field of mathematical study called category theory (with a liberal sprinkling of type theory and abstract algebra). This sounds a lot scarier than it is. You can do this!

All examples using ES6 syntax. wrap (foo) => bar means:

function wrap (foo) {

drkarl

Linux Backup Solutions

I've been looking for the best Linux backup system, and also reading lots of HN comments.

Instead of putting pros and cons of every backup system I'll just list some deal-breakers which would disqualify them.

Also I would like that you, the HN community, would add more deal breakers for these or other backup systems if you know some more and at the same time, if you have data to disprove some of the deal-breakers listed here (benchmarks, info about something being true for older releases but is fixed on newer releases), please share it so that I can edit this list accordingly.

Amanda (comments by sammcj)

• It has a lot of management overhead and that's a problem if you don't have time for a full time backup administrator.