adulau

Compression algorithms

• https://bellard.org/nncp/ - NNCP: Lossless Data Compression with Neural Networks
• https://www.byronknoll.com/cmix.html / https://github.com/byronknoll/cmix - CMIX cmix is a lossless data compression program aimed at optimizing compression ratio at the cost of high CPU/memory usage.
• https://tukaani.org/xz/ - XZ (LZMA)
• https://github.com/facebook/zstd - ZSTD

References

• http://www.mattmahoney.net/dc/text.html

adulau

Bloom filter papers

Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970, that is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not – in other words, a query returns either "possibly in set" or "definitely not in set".

Counting bloom filter

A counting Bloom filter (CBF) generalizes a Bloom filter data structure so as to allow membership queries on a set that can bechanging dynamically via insertions and deletions. As with a Bloom filter,a CBF obtains space savings by allowing false positives. We provide asimple hashing-based alternative based ond-left hashing called ad-leftCBF (dlCBF). The dlCBF offers the same functionality as a CBF, butuses less space, generally saving a factor of two or more. We describethe construction of dlCBFs, provide an analysis, and demonstrate theireffectiveness experimentally

• https://people.eecs.berkeley.

adulau

CPE mapping with the product or software name

Problem

IdentifyingNumber : {D307B5CF-D1F0-48A4-8DA3-54765F535208}
Name              : SQL Server 2012 SQL Data Quality Common
Vendor            : Microsoft Corporation
Version           : 11.2.5058.0
Caption : SQL Server 2012 SQL Data Quality Common

adulau

hashlookup.circl.lu

CIRCL hash lookup is a public API to lookup hash values against known database of files. NSRL RDS database is included. More database will be included in the future. The API is accessible via HTTP ReST API and the API is also described as an OpenAPI.

Get information about the hash lookup database (via ReST)

curl -X 'GET' \
  'https://hashlookup.circl.lu/info' \
 -H 'accept: application/json'

adulau

Warning: Analysis is based on the data leaked and subject to interpretation

Format

The original leak contains a zip with various files Zip per "country" with typographic errors and geographic errors. Some files are rar and 7z too.

CSV headers

There are multiple inconsistencies of position and size in the various contry files (merged from different sources?).

adulau

Tools

capa (from fireeye)

capa detects capabilities in executable files. You run it against a PE file or shellcode and it tells you what it thinks the program can do. For example, it might suggest that the file is a backdoor, is capable of installing services, or relies on HTTP to communicate.

• https://github.com/fireeye/capa

Papers

adulau

Official references

• Matrix synapse admin ReST API
• How to use the client-server API - examples with curl
• nio is a multilayered Matrix client library -

MISP and Matrix.org

• Matrix bot to interact with MISP - Documentation

adulau

Install pipe module for spacy.io

export CPPFLAGS="-std=c++98"; pip3 install textpipe

adulau

Generate PDFs from a set of CVE found by CPE

curl "http://127.0.0.1:5000/api/search/smiths-medical" | jq -r ".[] | .[].id" | parallel --gnu "python3 cve_doc.py -a http://127.0.0.1:5000/ -c {} >/tmp/{}.asc; cd /tmp; asciidoctor-pdf {}.asc"

adulau

Extract all names and synonyms

cat threat-actor.json | jq --raw-output ".values | [.[].value, .[].meta.synonyms] | flatten | .[] " | grep -v "null"