Orien is correct, it is the fork() system call triggered by ProcessBuilder or Runtime.exec or other means of the JVM executing an external process (e.g. another JVM running ant, a git command, etc.).
There have been some posts on the Jenkins mailing lists about this: Cannot run program "git" ... error=12, Cannot allocate memory
There is a nice description of the issue on the SCons dev list: fork()+exec() vs posix_spawn()
There is a long standing JVM bug report with solutions: Use posix_spawn, not fork, on S10 to avoid swap exhaustion. But I'm not sure if this actually made it into JDK7 as the comments suggest was the plan.
In summary, on Unix-like systems, when one process (e.g. the JVM) needs to launch another process (e.g. git) a system call is made to fork() which effectively duplicates the current process and all its memory (Linux and others optimize this with copy-on-write so the memory isn't actually copied until the child attempts to write to it). The duplicate process then makes another system call, exec() to launch the other process (e.g. git) at which point all that copied memory from the parent process may be discarded by the operating system. If the parent process is using large amounts of memory (as JVM processes tend to do), the call to fork() may fail if the operating system determines it does not have enough memory+swap to hold two copies, even if the child process will never actually use that copied memory.
There are several solutions. One is to simply add more memory. Another is to add more swap space to trick the fork() into working, even though the swap space is not strictly needed for anything.
On Linux, a third solution is to enable overcommit_memory option of the vm system (/proc/sys/vm/overcommit_memory). With overcommit, the call to fork() would always succeed, and since the child process isn't actually going to use that copy of the memory, all is well. Of course, it's possible that with overcommit, your processes will actually attempt to use more memory than is available and will be killed by the kernel.
A fourth option is to change the JVM to not use fork()+exec() but to use posix_spawn() when available. This is the solution requested in the JVM bug report above and mentioned on the SCons mailing list. It is also implemented in java_posix_spawn.
I'm trying to find out if that fix made it into JDK7. If not, I wonder if the Jenkins people would be interested in a work around such as java_posix_spawn. There seem to have been attempts to integrate that into Apache commons-exec.
Related - http://comments.gmane.org/gmane.comp.java.openjdk.core-libs.devel/22954