This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GET _search | |
{ | |
"query": { | |
"match_all": {} | |
} | |
} | |
DELETE website | |
PUT website |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- name: Install lxml requirements | |
apt: "pkg={{ item }} state=present update_cache=yes" | |
sudo: yes | |
with_items: | |
- python-dev | |
- libxml2 | |
- libxml2-dev | |
- libxslt1-dev | |
- lib32z1-dev | |
- libssl-dev |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### Keybase proof | |
I hereby claim: | |
* I am fabianvf on github. | |
* I am fabianvf (https://keybase.io/fabianvf) on keybase. | |
* I have a public key whose fingerprint is 0D2E 3DAC 3091 715B 7687 2B45 A87E 7040 BF11 F6F0 | |
To claim this, I am signing this object: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Install elasticsearch | |
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb | |
sudo dpkg -i elasticsearch-1.4.2.deb | |
# Install the wikipedia river | |
/usr/share/elasticsearch/bin/plugin -i elasticsearch/elasticsearch-river-wikipedia/2.4.1 | |
# Start elasticsearch | |
sudo service elasticsearch start |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import requests | |
response = requests.get('https://osf.io/api/v1/search/', params={ | |
'q': 'test AND category:project' # Full lucene syntax accepted: http://lucene.apache.org/core/2_9_4/queryparsersyntax.html | |
}) | |
results = response.json() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# coding: utf-8 | |
import json | |
import requests | |
HEADERS = { | |
'Content-type': 'application/json' | |
} | |
URL = 'https://osf.io/api/v1/search/' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"collectionTime": "2014-03-14T00:00:00Z", | |
"creationDate": "2014-09-12", | |
"creator": [ | |
{ | |
"name": "Roger Danger Ebert", | |
"sameAs": [ | |
"/wrgr2/", | |
"DAE" | |
], |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import json | |
from scrapi.processing.base import BaseProcessor | |
class StorageProcessor(BaseProcessor): | |
NAME = 'storage' | |
def process_raw(self, raw): |
OlderNewer