convert nmap XML to JSON

nmap-xml-json.md

Step 1:

Do nmap scan of the target

# nmap -sV -oX nmap_out.xml example.com 1>/dev/null 2>/dev/null

Step 2:

Convert nmap's XML output to JSON so that it can be fed to ELK stack.

#!/usr/bin/env python

import json,xmltodict

"""
Converts Nmap XML Output to JSON 
"""
#!/bin/env python
def xml2json(xml): 
    xml_file = open(xml)
    xml_content = xml_file.read()
    xml_file.close()
    xmljson = json.dumps(xmltodict.parse(xml_content), indent=4, sort_keys=True)
    json_data = json.loads(xmljson)
    return json_data

nmap_json = xml2json(nmap_out.xml)

Step 3

Create elastic index for nmap scan

#!/bin/bash

INDEX="$1"    #ES index name
JSONFILE="$2" #JSON file path name containing the settings for the index

HOST="http://localhost:9200"
DOCID=1

echo "Creating index $INDEX"

curl -XPUT "$HOST/$INDEX" --data-binary @$JSONFILE

echo "Done"

Step 4

Now, index nmap json data into elasticsearch

#!/bin/bash

# To prep a file for this script:
# - take a list of docs orig.json with one json doc per line
# - run: split -l 1000 orig.json orig-split


export ESINDEX="$1"    #ES index name
export ESTYPE="$2"     #ES document type name
JSONFILE="$3" #JSON file path name. One doc per line.

export HOST=""
DOCID=1

DOCS=`wc -l $JSONFILE | awk {'print $1'}`
echo "Indexing $DOCS $ESTYPE documents to $ESINDEX in 5 sec"
sleep 5

echo "Prepping bulk data"
rm tmp-bulk/bulk* #cleanup

awk ' {print "{\"index\":{}}"; print;}' $JSONFILE | split -a 4 -l 3000 - tmp-bulk/bulk-

echo "Indexing..."

# we aren't worried about losing data and setting consistency to 1 to speed this up
ls tmp-bulk/bulk* | xargs -L1 -I 'FILE' sh -c 'curl --silent -XPOST "http://localhost:9200/$ESINDEX/$ESTYPE/_bulk?consistency=one" -H 'Content-Type: application/json' --data-binary @FILE -o /dev/null; echo ".";'

Note: Not working at the moment! Will update soon!

References

• How to do bulk insert of json document in elasticsearch - https://kb.objectrocket.com/elasticsearch/how-to-bulk-index-elasticsearch-documents-from-a-json-file-using-python-753
• How to insert json data into elasticsearch - https://www.educative.io/answers/how-to-insert-data-to-elasticsearch
• How to do bulk json inserts into elasticsearch - https://onexlab-io.medium.com/elasticsearch-bulk-insert-json-data-322f97115d8d
• Utility to create an Elasticsearch bulk request from JSON data - https://github.com/mradamlacey/json-to-es-bulk
• Streaming json data insert into elasticsearch - https://gist.github.com/icamys/4287ae49d20ff2add3db86e2b2053977
• https://towardsdatascience.com/how-to-index-elasticsearch-documents-with-the-bulk-api-in-python-b5bb01ed3824
• https://stackoverflow.com/questions/71889063/bulk-index-create-documents-with-elasticsearch-for-python
• https://dylancastillo.co/elasticsearch-python/

@PSJoshi I've stumbled across this because I was looking into how xmltodict handles the multiple 'host' keys generated in the nmap XML file. It seems that xmltodict strips the many host keys when doing the conversion, leaving one list containing 'host', leaving you with something like:

{
    "nmaprun": {
        "@args": "nmap -sV -T4 -A -O -vv -oX homenetwork-new.xml 192.168.192.0/24",
        "@scanner": "nmap",
        "@start": "1711948208",
        "@startstr": "Mon Apr  1 16:10:08 2024",
        "@version": "7.94",
        "@xmloutputversion": "1.05",
        "debugging": {
            "@level": "0"
        },
        "host": [
            {
                "address": {
                    "@addr": "192.168.192.0",
                    "@addrtype": "ipv4"
                },
                "status": {
                    "@reason": "no-response",
                    "@reason_ttl": "0",
                    "@state": "down"
                }
            },
            {
                "address": {
                    "@addr": "192.168.192.2",
                    "@addrtype": "ipv4"
                },
                "status": {
                    "@reason": "no-response",
                    "@reason_ttl": "0",
                    "@state": "down"
                }
            }
        ]
    }
}

Have you considered this when using xmltodict? If so, is it an issue for you and how did you overcome it?

I see there's some debate here about how to handle this.