Perl One-Liner to Extract Data Using a Regular Expression

extract.markdown

Extract top 1000 slowest requests for a access log file

cat /opt/epoq/wildfly/standalone/log/accesslog | grep -a getRec | perl -nle 'print "$1 $_" if m/ D(.*?) /;' | sort -n | tail -n 1000

This command is designed to:

Filter log entries related to getRec requests.
Extract specific numerical or timestamp-like data following the D character.
Sort and display the top 1000 results based on the extracted numerical data.

Details:

This Bash command performs a series of operations on an access log file to extract, filter, process, and sort specific entries. Here's a breakdown of each step:

Step-by-Step Explanation: cat accesslog

Reads the contents of the access log file located at /opt/epoq/wildfly/standalone/log/accesslog.
This serves as the input for the rest of the pipeline.
| grep -a getRec

Filters lines containing the string getRec.
The -a option ensures binary files are treated as text, which is useful if the log contains non-text characters.
| perl -nle 'print "$1 $_" if m/ D(.*?) /;'

Executes a Perl script to process each line:
m/ D(.*?) / : Matches a pattern D... in the log entry, capturing the substring between D and the next space ( ).
"$1 $_" Prepends the captured group ($1) to the entire log line ($_) and prints it.
This effectively extracts and attaches the relevant data from the pattern for sorting.
sort -n

Sorts the output numerically based on the prepended captured data. tail -n 1000

Displays the last 1000 lines of the sorted output, since we have sorted be number, it's the 1000 slowest requests.

snippet.markdown

Do a statistical analysis of the response time from the other snipped:

All examples with the same dataset

Bins (with maphimbu from sta package)

cat access2024-12-01* | grep -a getX |  perl -nle 'print "$1" if m/ D(.*?) /;' | maphimbu -s 1 -d 1000 -C
500            440030
1500                80
2500                12
3500                12
4500                 3
5500                 2
6500                 1
7500                 2
8500                 2
9500                 4
10500                 3
11500                 3
13500                 3
16500                 1
19500                 2
21500                 1
23500                 1
33500                 1
42500                 1

median with datamash package

cat access2024-12-01* | grep -a getX |  perl -nle 'print "$1" if m/ D(.*?) /;' | datamash median 1
10

99% percentile

cat access2024-12-01* | grep -a getX |  perl -nle 'print "$1" if m/ D(.*?) /;' | datamash perc:99 1
350

snippet.sh

#This Perl command reads through a file line by line, and for each line, it searches for a pattern that matches "orderId":"(.*?)". If it finds this pattern, it extracts the value of orderId and prints it.

#-n loops through each line of the input.
#-l automatically handles newlines.
#-e allows the code to be executed directly from the command line.
#print $1 if /orderId":"(.*?)"/; prints the captured value inside orderId.
#In summary, it extracts and prints orderId values from the input.

cat log.file | perl -nle 'print $1 if /orderId":"(.*?)"/;'