composable_threads.md

First Peer

Role: Harvesting GDELT and publishing a master data feed over a Textile Thread

Create Master Thread

Use Textile Schema to match GDELT source: https://github.com/pinata-textile-collab-space/ETHDenver2019/blob/master/gdelt-schema.json

Master Thread ID: 12D3KooWEWCJWvByS69g6ihozWxXdc6e5egyAaBCauRttQJ6kkzC

Harvest GDELT

Python script

Each record is added as a new IPFS pin via the Textile Thread

Second Remote Peer

Role: Worker to transform data & publish a new Thread for consumption by other workers

Description: Subscribes to master Thread and performs a data transformation to prepare for writing to Eth. Publishes results to a second Thread

Subscribe to the master Thread

textile invites accept INVITE_ID -k SHARED_KEY

Create second Thread

textile threads add gdelt_eth -k eth-scan-reformat -t public -s invite_only --schema=Desktop/ethschema.json

Subscribe to shared thread | write to new thread

textile sub --type FILES --thread 12D3KooWJo51aEpftjXaNWnbGU8phUrM2888NgeXkmLEGHqLo1Wk |\
jq --unbuffered  -r -c \
'{"DataLink":.block.target, "EventId":".info.date.seconds", "EventType":"GDELTV2", "DataLinkType": "IPFS"}' |\
while read -r LINE;\
  do echo $LINE | textile files add -t 12D3KooWSDwKv71XnqqEcu6g6z1foXT1PexZXm5QQs8mLNoh9asy;  \
done`

Third Remote Peer

Role: Worker to transform data

Description: Subscribe to Second thread with prepared data, write that data to Ethereum

textile sub --type FILES --thread 12D3KooWSDwKv71XnqqEcu6g6z1foXT1PexZXm5QQs8mLNoh9asy |\
jq --unbuffered -r '.info.files[0].file.hash' | \
while read -r LINE;     \
    do  curl -H "Content-Type: application/json" -s -d $(textile cat $LINE)  http://localhost/postNewHashToETH;\
done

Ethereum

Timestamped GDELT data on Ethereum https://kovan.etherscan.io/address/0x074d32dccad3e3dd9c383900dfdfbba366861f6f

IPFS pins organized the Threads