Are You aware of the Data Storage Challenge? If your not as Old as me at time of writing 39 Yeaars Your Probally are not aware of the fact that data can get lost really quick and unrestore able. There are Unlimited Reallife Examples already. And its not their fault. The Failiure is that they are not aware about the Facts what matters for storage as storage is more complex the Industrie is trying to hide the facts and simply offer solutions for diffrent scenarios and then let the customer find out the hard facts. This Is the Motivation to write this Documentation about How Storage Really Works and how to address the problems.
- Write Speed
- Read Speed
- Access/Lookup Speed
- Query Abilitys Search Speed
- Kompression
- Overall data Size
- Data Persistence and Health Managment
- Data Reuse Scenarios
- Encryption.Scenarios Decryption Scenarios
- Data Processing Pipelines.
Yes the Filesystem it self is simply a DB! With a so called Block Storage Interface on top. Block Storage means that every file has at last one block. A Block can not be shared if a file needs more then one Block it will use additional blocks but will never share a block. On some Blocks it will store meta data the inital block which knows where the other blocks are is often refered as the Superblock.
when you formart your harddrive your creating the database on top of the Physical Medium you tell it how big the Blocks Should be and many other options like in a database. Also your filesystem choice is like to chooose a database.
If your a Company Owner or a Engineer we often get lost into Software marketing which claims hey i am super fast faster then all and Other Promotions. But they are hidding the Facts. The so called day 2 costs and the down sides. And when you don't know or understand the Facts it will be impossible to choose the right solutions out of the massiv amount of good software for your Scenario.
and you will come to the conclusion that it would be lean and slick if you would get predictable results from your Storage Solutions. As most companys can't effort a Data Loose!
Now you could complain there is the cloud and hell yes your right thats what the cloud does it offers Predictable Calculate able Results but for a massiv price and i don't mean The Running Costs i am talking about Speed and Privacy For Example in most scenarios you want to keep some data internal in your company thats why Hybrid Cloud Solutions exists. But they are Quit Complex beasts as they are designed to run on Big Scale with Big Hardware so thats where it comes again to Facts about Products like s3 Storage.
There is no one Fits All Solution on the Market we want to change that.
Note for my self
i should expand on this and tell about Resource usage and Administration Costs of diffrent Solutions. Log Managment and Monitoring can easy kill the budged.