Orien is correct, it is the fork() system call triggered by ProcessBuilder or Runtime.exec or other means of the JVM executing an external process (e.g. another JVM running ant, a git command, etc.).
There have been some posts on the Jenkins mailing lists about this: Cannot run program "git" ... error=12, Cannot allocate memory
There is a nice description of the issue on the SCons dev list: fork()+exec() vs posix_spawn()
There is a long standing JVM bug report with solutions: Use posix_spawn, not fork, on S10 to avoid swap exhaustion. But I'm not sure if this actually made it into JDK7 as the comments suggest was the plan.
In summary, on Unix-like systems, when one process (e.g. the JVM) needs to launch another process (e.g. git) a system call is made to