Build AWS Ubuntu AMI for OmicSoft Cloud Computing
WARNING: This page is deprecated. Please visit Install OmicSoft v12 on Ubuntu20 for Omicsoft Cloud Computing for updated information on newer versions.
AWS AMI for OmicSoft Cloud Computing
OmicSoft Cloud uses customized Amazon Machine Images (AMIs) as the template for creating compute instances for running cloud-based analyses. These AMIs require a few key components, including the Oshell engine, packages for transferring data to and from AWS S3 storage, and Docker.
OmicSoft provides default AMIs for cloud computing, based on Ubuntu. Alternatively, users can build their own AMI then follow the configuration steps below.
On-demand compute instances to analyze ArrayServer jobs (i.e. AMIs used for OmicSoft Cloud analysis) must use an Ubuntu image. We recommend explicitly specifying the preferred AMI from the list below, in ArrayServer.cfg in the Ami and AmiSnapshot parameters, or by building your own Ubuntu AMI with the configuration steps outlined below.
If you build your own AMI, the AMI should be built following the instructions below exactly, including the file paths to software tools. Please be aware the OmicSoft support can only provide limited advice on troubleshooting custom AMIs, because of the large number of configuration options that can impact performance.
Pre-built OmicSoft Ubuntu images
Pre-built OmicSoft Server Ubuntu20 images
OmicSoft recommends using our pre-configured Ubuntu20 AMI for installing OmicSoft Server in the AWS environment. Find the AMI for your region here.
Pre-built cloud analysis AMIs
For both onsite and cloud-based installations of OmicSoft server, cloud NGS analysis add-on works best with mono6-based AMIs for Cloud analysis: OmicSoft Cloud Analysis AMIs.
Legacy information: Configure your own Ubuntu 14 AWS image
Previous releases of OmicSoft Suite were compatible with pre-built mono4-based AMIs: Pre-built_OmicSoft_Ubuntu_AMIs#Mono4-based_AMIs_.28previous_generation.29
We recommend using a pre-built AMI (above) or building an analysis AMI on a modern OS such as Ubuntu20.
The following information is for reference purposes.
Install Required Linux packages
Make sure you have packages installed:
sudo apt-get install -f sudo apt-get update sudo apt-get install gcc g++ bison make libtool gettext libgdiplus zlib1g-dev libglib2.0-dev sudo apt-get install libpng-dev libjpeg-dev libtiff5-dev libgif-dev libgs-dev libexif-dev libX11-dev fontconfig
Install Mono
Install Mono 4.0.4
- Download Mono 4.0.4
wget -c http://download.mono-project.com/sources/mono/mono-4.0.4.1.tar.bz2
The bz2 file can be saved to a temporary location, e.g. ~/temp/
- Compile and install. On the command line, type
tar jxvf mono-4.0.4.1.tar.bz2
cd mono-4.0.4
./configure --prefix=/opt/mono-4.0.4 --with-large-heap=yes
make
make install
Note:
- The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
- The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)
Double check mono installation and version
ls /opt/mono-4.0.4/bin/mono* -all /opt/mono-4.0.4/bin/mono --version /opt/mono-4.0.4/bin/mono-sgen --version
Install Mono 4.8.1
- Download Mono 4.8.1
wget -c http://download.mono-project.com/sources/mono/mono-4.8.1.0.tar.bz2
The bz2 file can be saved to a temporary location, e.g. ~/temp/
- Compile and install. On the command line, type
tar jxvf mono-4.8.1.0.tar.bz2
cd mono-4.8.1
./configure --prefix=/opt/mono-4.8.1 --with-large-heap=yes
make
make install
Note:
- The location of mono installed is set by the option "--prefix" in the configure step, which can be changed to another location.
- The option --with-large-heap=yes is to enable support for GC heaps larger than 3gb, which is required for NGS alignment, as well as some Array Server functions)
- if you get an error like "processes.c:1672:12: error: implicit declaration of function ‘makedev’ [-Werror=implicit-function-declaration]":
- add
- #include <sys/sysmacros.h>
- to mono/io-layer/processes.c
Double check mono installation and version
ls /opt/mono-4.8.1/bin/mono* -all /opt/mono-4.8.1/bin/mono --version /opt/mono-4.8.1/bin/mono-sgen --version
To review previous instructions on installing mono-2.10.9 (previous version), visit Install mono 2.10.9
use sudo in ubuntu.
Install libgdiplus
sudo apt-get install libgdiplus
Sqlite
Install SQLite v3.7.14.1 (2012-10-04) by building it from source:
# Download SQLite source archive: $ cd /opt $ sudo wget -c http://www.sqlite.org/sqlite-autoconf-3071401.tar.gz $ sudo tar zxvf sqlite-autoconf-3071401.tar.gz $ sudo mv sqlite-autoconf-3071401 sqlite $ cd sqlite # Build and install: $ sudo ./configure --prefix=/opt/sqlite $ sudo make $ sudo make install # Check SQLite was installed successfully: $ sudo ./sqlite3 --version # Clean up: $ sudo rm /opt/sqlite-autoconf-3071401.tar.gz $ cd ~
use sudo in ubuntu.
Set ulimit
Make sure the ulimit for "max user processes" and "open files" are set to the max value: 65536. You can check the values by typing: ulimit -a.
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515184 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Modify ulimit in two config files following ulimit setup wiki.
Make sure ulimit for root (create user root for ubuntu) has the same ulimit.
Install Oshell
cd /opt sudo mkdir oshell cd oshell sudo wget -c http://omicsoft.com/software_update/OmicsoftUpdater.exe sudo touch oshell.exe sudo /opt/mono-4.8.1/bin/mono /opt/oshell/OmicsoftUpdater.exe sudo /opt/mono-4.8.1/bin/mono /opt/oshell/oshell.exe --version
Install AWS CLI
sudo apt-get install python-pip sudo pip install awscli
Optional installation for dev features
Multiple Python versions
sudo apt-get install python3.6 sudo apt-get install python3-pip
Then different version of python can be called:
ubuntu@ip-172-31-12-154:/opt/oshell$ python -V Python 2.7.6 ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6 -V Python 3.6.3 ubuntu@ip-172-31-12-154:/opt/oshell$ python3.6 Python 3.6.3 (default, Oct 6 2017, 00:00:00) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>>
R basics and a few useful packages
sudo apt-get install r-base sudo R
Then install packages
source("http://bioconductor.org/biocLite.R")
install.packages("samr");
install.packages("Rtsne");
biocLite("pvca")
biocLite("Biobase")
biocLite("lme4")
biocLite("vsn")
biocLite("limma")
biocLite("edgeR")
biocLite("impute")
biocLite("lumi")
Install Docker
Following https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-repository
For Ubuntu 14.04, we did
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual sudo apt-get install apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update sudo apt-get install docker-ce
Test
sudo docker run hello-world ubuntu@ip-172-31-12-154:~$ sudo docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 5b0f327be733: Pull complete Digest: sha256:07d5f7800dfe37b8c2196c7b1c524c33808ce2e0f74e7aa00e603295ca9a0972 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly.
Related Articles
- Cloud Online Help
- Example of running 100 CCLE samples on cloud
- Omicsoft aligner wiki and publication
