data storage and query

gistfile1.md

Defining platform to process large amount of data

Requirement

• [1. Transfer data -> 2. Store data -> 3. Parse data -> 4. Query data] -> 5. Use the result in a playbook environment (ex. in Jupyter notebook)
• hot storage (no longer than 2 months)

Potential technologies

• From Azure there are 7 types of storage (https://azure.microsoft.com/en-us/product-categories/storage/). The main ones are Azure Blob Storage and Azure Data Lake Storage. A comparison between both are (here!)[https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage]. To query both, Azure Data Explorer can be used.
• Comparison between Azure, Google, Amazon here: https://www.parkmycloud.com/blog/cloud-storage-cost-comparison/ If this is true, then Azure Data Lake is the chosen one.

Partial conclusion

• Azure Data Lake + Azure Data Explorer (https://docs.microsoft.com/en-us/azure/data-explorer/data-lake-query-data)
• Azure Data Explorer from Jupyter Notebook (https://docs.microsoft.com/en-us/azure/data-explorer/kqlmagic)