| title | author | institute | theme | colortheme | fonttheme | mainfont | fontsize | urlcolor | linkstyle | aspectratio | date | lang | section-titles |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Hashlookup service - how to improve your digital forensic investigations |
Alexandre Dulaunoy |
CIRCL |
Frankfurt |
beaver |
professionalfonts |
Hack Nerd Font |
10pt |
red |
bold |
169 |
30/09/2021 |
en-US |
false |
- State of current NIST NSRL databases and other known file filters (KFF)
- too few Operating Systems / Software available (e.g. OSX?, Linux distributions),
- nsrllookup.com / nsrlsrv use their own protocol, next version will be gRPC,
- nsrlsrv only support MD5,
- many sources are difficult to use (e.g. NSRL ISOs), ill-maintained or outdated.
- The need of a public and easy to use API for all sources (NSRL is not alone)
- A global public instance of all known sources,
- a common ReST API normalizes the access to several datasources,
- available for MD5, and SHA1 (and more),
- that includes fuzzy hashes,
- additional datapoints available through the intersection of datasources,
- https://hashlookup.circl.lu/ - OpenAPI Swagger
- Current databases:
- NIST NSRL - RDS hash sets including current, modern, android, iOS and legacy,
- Ubuntu packages distribution,
- CentOS core OS distribution,
- Fedora project EPEL repository,
- Kali linux packages distribution and OpenSUSE distribution.
Where is this errno.h (found in a specific package) include available?
https://hashlookup.circl.lu/lookup/sha1/40f68888766b102b94f4c414eb0574ac6467be93
{
"CRC32": "D6614BD0",
"FileName": "./usr/s390x-linux-gnu/include/errno.h",
"FileSize": "1832",
"MD5": "5CD0FF97AAC5391C34757DCA05E2B6BA",
"OpSystemCode": {
"MfgCode": "1006",
"OpSystemCode": "362",
"OpSystemName": "TBD",
"OpSystemVersion": "none"
},
"ProductCode": {
"ApplicationType": "Operating System",
"Language": "Multilanguage",
"MfgCode": "2529",
"OpSystemCode": "910",
"ProductCode": "217853",
"ProductName": "Linux Mint - Tricia",
"ProductVersion": "19.3"
},
"SHA-1": "40F68888766B102B94F4C414EB0574AC6467BE93",
"SHA-256": "B6DEBCA969A2C6AA756232E2DD69EBFC5EE8FEDBDC0229B869917518D15689A5",
"SSDEEP": "48:826ROuNkUynHJO+S8RGWv8J1AkJ8IZ9th0MGJ0:3uNA4+Ynj05J0",
"SpecialCode": "",
"TLSH": "T1D431FE192A15437395D20763A9CF20DAC41EAAEB32A6A4D0064D965E8C4BA7B13F36D8",
"db": "nsrl_modern_rds",
"insert-timestamp": "1630959315.4398303",
"source": "NSRL",
"parents": [
{
"FileSize": "2080372",
"MD5": "059D45A0ABF3D857B267C17260BDE9FD",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "Ubuntu Developers <[email protected]>",
"PackageName": "libc6-dev-mipsel-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3ubuntu1cross1",
"SHA-1": "03F71DB19EA877E81894B4B6CF0C05C92F82E0A8",
"SHA-256": "A6B5664F40EFC472FF167052F9B82F5268C1797CBB8EF433D3130D09E2FE6FAC"
},
{
"FileSize": "2548052",
"MD5": "E3E77A9F2DD94EF20D4BA4ED2F32BD3A",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "Ubuntu Developers <[email protected]>",
"PackageName": "libc6-dev-ppc64el-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3ubuntu1cross1.1",
"SHA-1": "05B73A3E3CCE0A194218FE86EA769B402E0FB8F8",
"SHA-256": "ED647442B2C6F8C5169C130CADDA3D0B664E8DF2CDADF7B3EE6D2C66D4AB98A3"
},
{
"FileSize": "2469956",
"MD5": "0ECD877AF006D8D6F37BAC9619F9D1FB",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-mips64el-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3cross1",
"SHA-1": "06B1989344676FFF1CD2D554E8EB7A864940B5E1",
"SHA-256": "3547033EADABF0C1F72E88C21B00CE68250861B8D3A38F6BD274B0307AA30F85"
},
{
"FileSize": "2347880",
"MD5": "09A161D0E52F411E3EF53A0F1C3522A3",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-hppa-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3cross3",
"SHA-1": "082386F6B58058E1149842666EB370F03FC4E0C4",
"SHA-256": "112381BACAA49D2F533E78144964E72DB70743332E9DED993EB439DB0568ED62"
},
{
"FileSize": "2280492",
"MD5": "6CC56E1C26C33DC88A1F9E777B4DD376",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-mipsel-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-6cross1",
"SHA-1": "099DCFA33665BABFE039CCDFA7BBB5B40FCA7842",
"SHA-256": "5E5A4CF231AFE98B8ABB4E48A082D1A19BBD32D013A9D235632C0CD8D143E595"
},
{
"FileSize": "2424220",
"MD5": "C134893A8E7F8889128DB8C4470B0C85",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "Ubuntu Developers <[email protected]>",
"PackageName": "libc6-dev-x32-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3ubuntu1cross1",
"SHA-1": "0AB737A2240CDF265CF2F2CD4DD20194DD3B951F",
"SHA-256": "C532514305E00BB0F3017689C60EDF3ECAB025FB49C50DE53AC529486D218BB9"
},
{
"FileSize": "2282248",
"MD5": "8E1B993D406D3A63236FDA01319B9415",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "Ubuntu Developers <[email protected]>",
"PackageName": "libc6-dev-mipsr6el-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3ubuntu1cross1.2",
"SHA-1": "0B2610137605678ABD4FB95B1691133D87534144",
"SHA-256": "0B8100CCAA5C723A40EB7661D326885AAC865A70FA6798BBAD1B1F9C5D9F5C25"
},
{
"FileSize": "2219952",
"MD5": "6CAAC685999DD5C1484C16AECB670757",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-sh4-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-2cross1",
"SHA-1": "0CBFF2F15C06E5772900E3DC00923BC66148237F",
"SHA-256": "91607A291D6A2DF448C382A00A333E41723B0A92D056F332084C4C3C5C38ACBC"
},
{
"FileSize": "2450276",
"MD5": "CDD89E81A7D366E6C4D60A0A8045389C",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-i386-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3cross1",
"SHA-1": "0D0FB569EE34282584DD4B2E62A9B52DDF34EAD3",
"SHA-256": "7A906141BC778069510FF11B6157EF25D3A79D1C03947C36432C5EE5A5883EC0"
},
{
"FileSize": "2480072",
"MD5": "46D2AA0A88BC8C0DF2152CD98B2A4083",
"PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
"PackageMaintainer": "GNU Libc Maintainers <[email protected]>",
"PackageName": "libc6-dev-mipsr6el-cross",
"PackageSection": "libdevel",
"PackageVersion": "2.27-3cross3",
"SHA-1": "0D8A3CB782F20B1203688AFEF84D3BAC807CA3D8",
"SHA-256": "8C42CE2D680AA1C03158907F54908CA7FFDB240785DC5682435C4CDE768369EA"
}
]
}A tar file found on a Ubuntu distribution
https://hashlookup.circl.lu/lookup/sha1/b0b105d1c3cb3da684a1d487b46156238579ffb4
{
"FileName": "./bin/tar",
"FileSize": "423312",
"MD5": "749AD0837C043E1AE63198678E0F5161",
"SHA-1": "B0B105D1C3CB3DA684A1D487B46156238579FFB4",
"SHA-256": "D45FE4E12DBE475EF2C283FA4A5A1658749F3FBD8EB8056EAE3C1EF9228876D0",
"SSDEEP": "6144:686SGkTRyemT4nKgLDlSYMKHa8lwlT2lRZtFaNsGohPMQ/bEFPoPw8wTBu:6864TRtm83QJawloBMNsRPMfN",
"TLSH": "T1E5943A16FDA214BCC1A6C9B0467B9376797274D843222A7F359CD5303E42F642F1EBA2",
"parents": [
{
"FileSize": "233760",
"MD5": "E3F8D82F755B246685409B087D312037",
"PackageDescription": "GNU version of the tar archiving utility\n Tar is a program for packaging a set of files as a single archive in tar\n format. The function it performs is conceptually similar to cpio, and to\n things like PKZIP in the DOS world. It is heavily used by the Debian package\n management system, and is useful for performing system backups and exchanging\n sets of files with others.",
"PackageMaintainer": "Ubuntu Developers <[email protected]>",
"PackageName": "tar",
"PackageSection": "utils",
"PackageVersion": "1.29b-2ubuntu0.2",
"SHA-1": "C449F470704294CC40DA488C32A4ADB191E2C5DA",
"SHA-256": "6BDBB90C9C073F8E8F92DA231A2E553410CE397489F2F1F77D1AE8DDBD0C7BC4"
}
]
}A compromised Linux server needs to be analyzed. How can I quickly find the binaries from the original distribution from others?
sha1sum * | cut -f1 -d" " | parallel 'curl -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .
hashlookup-forensic-analyser is a simple Python script to analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.
If you don't want to share your lookups online, hashlookup provides a bloom filter to download.
A bloom filter (a compact representation of the dataset) is available at https://cra.circl.lu/hashlookup/hashlookup-full.bloom (~700MB) with all the SHA-1 value known in hashlookup. The format of the bloom filter is the DCSO bloom library and cli.
How to use locally the bloomfilter (just don't forget to install the DCSO bloom cli):
find /usr/bin/ -type f -print0 | xargs -0 sha1sum | awk '{ print $1 }' | tr a-f A-F | bloom c /home/adulau/hashlookup-full.bloom
The bloom filter doesn't contain any metadata. It's just the SHA-1 hash value. You check the statistics and info with the following command
adulau@kolmogorov ~/hashlookup $ bloom s hashlookup-full.bloom
File: /home/adulau/hashlookup/hashlookup-full.bloom
Capacity: 296893697
Elements present: 296890922
FP probability: 1.00e-04
Bits: 5691486835
Hash functions: 14
- Report bugs or ideas about the CIRCL hashlookup service.
- New data sources, let us know where we can find it and we will add it.
- The current database includes more than 4.4 billion hashes.
- Internet-Draft for the hashlookup format.
- Contact [email protected] .