webmat

Filebeat modules are all either open source, or provided via the Elastic License. You can look at them all, to understand how the parsing, the conversion and the mapping to ECS are done.

Looking for the modules

• All Filebeat modules are listed here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html
• The code for Apache 2 open source modules is here: https://github.com/elastic/beats/tree/master/filebeat
• The code for Elastic Licensed modules is here: https://github.com/elastic/beats/tree/master/x-pack/filebeat/module

Looking at the code of the pipelines

webmat

The file schema-explorer.ndjson is a dashboard called "Schema Explorer". This dashboard lets you view ECS field definitions from within Kibana.

This requires that you import the ECS csv (found here generated/csv/fields.csv) into Elasticsearch via the Machine Learning Data Visualizer.

webmat

PATTERNS_GLOB = ENV['PATTERNS_GLOB'] || '~/work/elastic/plugins/logstash-patterns-core/patterns/*'

FIELD_MATCHER = /{\w+:([^}]+)}/
ECS = %w(labels agent client cloud container destination ecs error event file
         geo group host http log network observer organization os process
         related server service source url user user_agent)

field_names = {}
puts "File name\tField\tLine\tPosition\tConflict"
Dir[PATTERNS_GLOB].each do |file|

webmat

{
  "@timestamp": "2018-12-07T11:05:07.000Z",
  "agent" : {
    "type": "filebeat",
    "version": "7.0.0-beta1"
  },
  "ecs": { "version": "1.0.0" },
  "event": {
    "action": "get",
    "dataset": "apache.access",

webmat

Keybase proof

I hereby claim:

• I am webmat on github.
• I am webmat (https://keybase.io/webmat) on keybase.
• I have a public key ASBxf8fg0XaY8to2x3Ujd9jBhZIDzp_fjMMI33Dlg9Vq-Qo

To claim this, I am signing this object:

webmat

Understanding the concepts

  - The Git Parable, but Tom Preston-Werner
    http://tom.preston-werner.com/2009/05/19/the-git-parable.html
  - git is a bucket of commits
    - most commits have one parent, and a blob
    - some commits have 2 parents (merges)
    - some commits have 1 parents (first commit!)
  - refs
    - branches: pointers to a commit - moves over time

webmat

I recently had to detect whether a MySQL server had any tables that didn't support transactions (mainly MyISAM).

The only problem is that this server had close to 5000 indivitual databases, and my usual query didn't cut it:

SELECT  information_schema.TABLES.TABLE_SCHEMA,
        information_schema.TABLES.TABLE_NAME,
        information_schema.TABLES.ENGINE,
FROM information_schema.TABLES
WHERE information_schema.TABLES.TABLE_SCHEMA not in ('performance_schema', 'mysql', 'information_schema')

webmat

webmat

# Gemfile
group :production do
  gem 'pg'
end

group :development, :test do
  gem 'sqlite3'
end

# config/application.rb

webmat

alias   j='jump'
alias   m='mark'
alias  mm='marks'
alias  um='unmark'