When creating your rules for YARA keep in mind the following guidelines in order to get the best performance from them. This guide is based on ideas and recommendations by Victor M. Alvarez and WXS.
- Revision 1.2.1, February 2019, applies to all YARA versions higher than 3.3
Global rules are evaluated first. Only if they are satisfied non-global rules are evaluated. This may be useful if all samples exhibit the same characteristics. Use them combined with the "private" statement to suppress a match notification on the global rules.
Examples:
All rules of the ruleset should match on Windows executables
global private rule EXE {
meta:
description = "Executable File"
condition:
uint16(0) == 0x5A4D and uint32(uint32(0x3C)) == 0x00004550
}
All rules of the ruleset should match on JAR/ZIP files of a certain size
global private rule Small_JARZIP {
meta:
description = "Small JARZIP File"
condition:
uint16(0) == 0x4B50 and filesize < 800KB
}
Consider the us of the "filesize" variable in global rules. If all the files that you want to analyze are smaller than 3MB than set this limit in a global rule to skip all files of a bigger size before the strings of every rule of the string set get evaluated.
global private rule malware_size {
meta:
description = "Size of all samples is lower than 1MB - setting limit to 3MB"
condition:
uint16(0) == 0x4B50 and filesize < 3MB
}
- Slow: Regular Expressions
- Fast: Strings
- Fastest: Bytes at offset or virtual address
YARA extracts from the strings short substrings up to 4 bytes long that are called "atoms". Those atoms can be extracted from any place within the string, and YARA searches for those atoms while scanning the file, if it finds one of the atoms then it verifies that the string actually matches.
For example, consider this strings:
/abc.*cde/
=> posible atoms are abc and cde, either one or the other can be used
/(one|two)three/
=> posible atoms are one, two and three, we can search for three alone, or for both one and two
YARA does its best effort to select the best atoms from each string, for example:
{ 00 00 00 00 [1-4] 01 02 03 04 }
=> here YARA uses the atom 01 02 03 04, because 00 00 00 00 is too common
{ 01 02 [1-4] 01 02 03 04 }
=> 01 02 03 04 is preferred over 01 02 because it's longer
So, the important point is that strings should contain good atoms. These are bad strings because they contain either too short or too common atoms:
{00 00 00 00 [1-2] FF FF [1-2] 00 00 00 00}
{AB [1-2] 03 21 [1-2] 01 02}
/a.*b/
/a(c|d)/
The worst strings are those that don't contain any atoms at all, like:
/\w.*\d/
/[0-9]+\n/
This regular expression don't contain any fixed substring that can be used as atom, so it must be evaluated at every offset of the file to see if it matches there.
Another good import recommendation is to avoid for loops with too many iterations, specially of the statement within the loop is too complex, for example:
strings:
$a = {00 00}
condition:
for all i in (1..#a) : (@a[i] < 10000)
This rule has two problems. The first is that the string $a is too common, the second one is that because $a is too common #a can be too high and can be evaluated thousands of times.
This other condition is also inefficient because the number of iterations depends on filesize, which can be also very high:
for all i in (1..filesize) : ($a at i)
It is known practice to select files by the first bytes at offset 0. (magic header)
strings:
$mz = { 4d 5a }
condition:
( $mz at 0 )
But using MZ as string or { 4d 5a } would cause a string search for this short atom in the whole file, which could produce a lot of matches before checking the location of those matches with position 0. The best way to do this is by using the uint16(0) or uint32(0) statements. Alternatively you can use the big endian forms uint16be(0) and uint32be(0), which are available since YARA v3.2.0.
Here is an example. Note the little ending order ("Z" = "0x5a" and "M" = "0x4d").
condition:
uint16(0) == 0x5a4d
Also consider using the "magic" module which is not available on the Windows platform. Using the "magic" module makes it much easier to apply complex magic header checks and the resulting rules are easiert to read.
Custom GIF magic header definition:
rule gif_1 {
condition:
(uint32be(0) == 0x47494638 and uint16be(4) == 0x3961) or
(uint32be(0) == 0x47494638 and uint16be(4) == 0x3761)
}
Using the "magic" module:
import "magic"
rule gif_2 {
condition:
magic.mime_type() == "image/gif"
}
Avoid defining too short strings. Any string with less than 5 or 6 bytes will probably appear in a lot of files.
Example:
Looking for a string that has a .pdb extension.
- BAD:
$s = ".js" - OK:
$s = /[^\.\\]{1,40}\.js/ - BEST: just avoid too short strings (atoms)
Some strings are long enough but shouldn't be used due to a different reason - uniformity. These are some examples for strings that shouldn't be used as they could cause too many matches in files.
$s1 = "22222222222222222222222222222222222222222222222222222222222222" ascii
$s2 = "\x00\x20\x00\x20\x00\x20\x00\x20\x00\x20\x00\x20\x00\x20" ascii // wide formatted spaces
Try to describe string definitions as narrow as possible. Avoid the "nocase" attribute if possible, because many atoms will be generated and searched for. Remember, in the absence of modifiers "ascii" is assumed by default. The posible combinations are:
FAST - only one atom is generated
$s1 = "cmd.exe" // (ascii only)
$s2 = "cmd.exe" ascii // (ascii only, same as $s1)
$s3 = "cmd.exe" wide // (UTF-16 only)
$s4 = "cmd.exe" ascii wide // (both ascii and UTF-16) two atoms will be generated
$s5 = { 63 6d 64 2e 65 78 65 } // ascii char code in hex
SLOW - many atoms will be generated
$s5 = "cmd.exe" nocase (all different cases, e.g. "Cmd.exe", "cMd.exe", "cmD.exe" ..)
Use expressions only when necessary. Regular expression evaluation is inherently slow and consumes a significant amount of memory. Don't use them if hex strings with jumps and wild-cards can solve the problem.
If you have to use regular expressions avoid greedy .* and even reluctant quantifiers .*?. Instead use exact numbers like .{1,30} or even .{1,3000}.
Also try to include long sequences of strings that could serve as ankers in the matching progress. Again, the longer the better.
BAD
$s1 = /http:\/\/[.]*\.hta/
BETTER
$s1 = /http:\/\/[a-z0-9\.\/]{3,70}\.hta/
BEST
$s1 = /mshta\.exe http:\/\/[a-z0-9\.\/]{3,70}\.hta/
Try to write condition statements in which the elements that are most likely to be "False" are placed first. The condition is evaluated from left to right. The sooner the engine identifies that a rule is not satisfied the sooner it can skip the current rule and evaluate the next one. The speed improvement caused by this way to order the condition statements depends on the difference in necessary CPU cycles to process each of the satements. If all statements are more or less equally expensive, reordering the statements causes no noticeable improvement. If one of the statements can be processed very fast it is recommended to place it first in order to skip the expensive statement evaluation in cases in which the first statment is FALSE.
Changing the order in the following statement does not cause a significant improvement:
$string1 and $string2 and uint16(0) == 0x5A4D
However, if the execution time of the statements is very different, reordering in order to trigger the short-circuit will improve the scan speed significantly:
SLOW
EXPENSIVE and CHEAP
math.entropy(0, filesize) > 7.0 and uint16(0) == 0x5A4D
FAST
CHEAP and EXPENSIVE
uint16(0) == 0x5A4D and math.entropy(0, filesize) > 7.0
Short-circuit evaluation was introduced to help optimizing expensive sentences, particularly "for" sentences. Some people were using conditions like the one in the following example:
strings:
$mz = "MZ"
...
condition:
$mz at 0 and for all i in (1..filesize) : ( whatever )
Because filesize can be a very big number, "whatever" can be executed a lot of times, slowing down the execution. Now, with short-circuit evaluation, the "for" sentence will be executed only if the first part of the condition is met, so, this rule will be slow only for MZ files. An additional improvement could be:
$mz at 0 and filesize < 100K and for all i in (1..filesize) : ( whatever )
This way a higher bound to the number of iterations is set.
This "short-circuit" evaluation is applied since YARA version 3.4.