Increase Cloud analysis memory map

Increase AWS Cloud oshell analysis memory maps to avoid job crashes with large NGS files

In some cases, very large RNA-seq files (150 million reads+) may crash in certain memory-intensive steps when analyzed in OmicSoft Server Cloud analysis (or any linux environment), such as with Ngs_FilterNgsFiles.pdf and CombinedFusionAnalysis.

Typical error message

With very large NGS analyses, using certain memory intensive procs, a user may notice one or more files failing with a message in the job log such as

[02:52:00] Finished processing 100000 reads from batch number 1324.
[02:52:02] Started processing 100000 reads. Processing batch number 1325.
[02:52:09] Finished processing 100000 reads from batch number 1325.
[02:52:11] Started processing 100000 reads. Processing batch number 1326.
[02:52:18] Finished processing 100000 reads from batch number 1326.
[02:52:20] Started processing 100000 reads. Processing batch number 1327.
[02:52:20] * Assertion at mini-exceptions.c:3172, condition `tls->signal_stack'
not met
[02:52:20] =================================================================
[02:52:20] Native Crash Reporting
[02:52:20] =================================================================
[02:52:20] Got a SIGABRT while executing native code. This usually indicates
[02:52:20] a fatal error in the mono runtime or one of the native libraries
[02:52:20] used by your application.
[02:52:20] =================================================================
[02:52:20] =================================================================
[02:52:20] Native stacktrace:
[02:52:20] =================================================================
[02:52:20] 0x4b0dfc - /app/mono/bin/mono :

The instance size and parallel job number will not impact the analysis, since a limit to the number of "memory maps" is set by the Linux operating system, and not all memory available is being utilized.

Solution

In ArrayServer.cfg, place the line

CustomCommand=sudo sysctl -w vm.max_map_count=655350

and restart the OmicSoft Server application.

This command changes the number of memory maps available to the cloud instance, and should allow analysis of files with 1 billion+ reads.