Logstash Multiline parsing

Config

input {
        stdin {codec => multiline{

                 pattern => "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601}"
                 negate => true
                 what => "previous"
        }  }}

filter {
    grok {
     match => [ "message", "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:time} \[%{WORD:main}\] %{LOGLEVEL:loglevel}%{SPACE}%{SPACE}\(%{JAVACLASS:class}\) %{DATA:mydata}((?m)\n\t%{GREEDYDATA:stack}) " ]
 }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
 }
}

output {
  elasticsearch {
    host => "localhost"
  }
  stdout { codec => rubydebug}
}

Expected output

{
    "@timestamp" => "2015-02-12T09:55:16.056Z",
       "message" => " -  2014-04-29 13:04:23,700 [main] INFO  (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/
	   config/threadpoolworker.properties\n",
      "@version" => "1",
          "host" => "myHost",
          "time" => "2014-01-14 11:09:38,623",
          "main" => "main",
      "loglevel" => "ERROR",
         "class" => "support.context.ContextFactory",
        "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle"
         "stack" => ""
}
{
    "@timestamp" => "2015-02-12T09:55:16.056Z",
       "message" => " -  2014-01-14 11:09:38,623 [main] ERROR  (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n",
      "@version" => "1",
          "host" => "myHost",
          "time" => "2014-01-14 11:09:38,623",
          "main" => "main",
      "loglevel" => "ERROR",
         "class" => "support.context.ContextFactory",
        "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle"
         "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n"
}

Input

 -  2014-04-29 13:04:23,700 [main] INFO  (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciel/splapp/standalone/config/threadpoolworker.properties
 -  2014-01-14 11:09:38,623 [main] ERROR  (support.context.ContextFactory) Error getting connection to database jdbc:oracle, with user cisuser and driver oracle.jdbc.driver.OracleDriver
	at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)
	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)
	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)
	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)

output

{
    "@timestamp" => "2015-02-12T09:55:16.056Z",
       "message" => " -  2014-04-29 13:04:23,700 [main] INFO  (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/config/threadpoolworker.properties\n -  2014-01-14 11:09:38,623 [main] ERROR  (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n",
      "@version" => "1",
          "host" => "myHost",
          "time" => "2014-01-14 11:09:38,623",
          "main" => "main",
      "loglevel" => "ERROR",
         "class" => "support.context.ContextFactory",
        "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle"
         "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n"
}

There is still several issues:

1. You need to put (?m) at the beginning of the regex for it to work
2. You need 2 different grok pattern for matching multiline or simple line
3. From your output, I suppose you copy-pasted the complete input content in one go, but if you copy-paste one line at a time, you get the expected result. This is a strange difference of behaviour, I will further investigate and report a bug if necessary

The config I used for testing is here https://gist.github.com/wiibaa/c47e5f79d45d58d05121

Side note: repeating the %{SPACE} pattern twice is not needed because it already means any spaces with the regex \s* https://github.com/elasticsearch/logstash/blob/1.4/patterns/grok-patterns#L13

Hi wiibaa, Thank you so much this helped me get the results i want. I added the "break_on_match => false" parameter so that grok won't break on the first match. and I used the file input so i won't have to copy-paste one line at a time.
Can you tell me what are the bugs I may encounter while using the input multiline codec? I believe you mentioned something about it before. Thank you again.

When using the multiline codec, you cannot currently define an array or use a wildcard in the path => configuration due to logstash-plugins/logstash-input-file#10
The workaround is to use the multiline filter

Okay I tested the filter multiline with a wildcard it actually works. I hope there are no bugs related to it because when I'll start parsing millions of log lines I won't be able to tell whether lines were merged from different files.
Thank you wiibaa!