Build AWS Amazon Linux 2 AMI for OmicSoft Cloud Computing

From Array Suite Wiki

AWS AMI for OmicSoft Cloud Computing

OmicSoft Cloud uses customized Amazon Machine Images (AMIs) as the template for creating compute instances for running cloud-based analyses. These AMIs require a few key components, including the Oshell engine, packages for transferring data to and from AWS S3 storage, and Docker (with v10.2).

OmicSoft provides default AMIs for cloud computing, based on Amazon Linux 2. Alternatively, users can build their own AMI then follow the configuration steps below.

Tips.pngOmicSoft generally recommends using our pre-built Compute AMIs


On-demand compute instances to analyze ArrayServer jobs (i.e. AMIs used for OmicSoft Cloud analysis) must use an Amazon Linux 2 image. We recommend explicitly specifying the preferred AMI from the list below, in ArrayServer.cfg in the Ami and AmiSnapshot parameters, or by building your own Ubuntu AMI with the configuration steps outlined below.

If you build your own AMI, the AMI should be built following the instructions below exactly, including the file paths to software tools. Please be aware the OmicSoft support can only provide limited advice on troubleshooting custom AMIs, because of the large number of configuration options that can impact performance.

Configure your own Amazon Linux 2 image

From AWS Console, start a new instance using Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0742b4e673072066f (64-bit x86) (Region: US East 1 (N. Virginia)

  • instance type: m4.xlarge (not so important in this phase)
  • auto-assign public IP: enabled
  • storage: 30 GiB, volume type: General Purpose SSD (gp2)
  • the 30GiB initial size must be maintained for backwards compatibility
  • Tags: Name: Oshell2021AmazonLinux2Mono6Dev
  • Security group: define a new one that allows you to access the instance
  • when launching it, select an existing key pair for which you have also the private key (for ssh access) - ex.: Omicsoft.Launching.ppk

Connect to the instance using SSH (ex.: using Putty)

  • user: ec2-user

Update and install the required packages

sudo yum update
 
sudo yum install gcc gcc-c++ bison pkgconfig libtool libstdc++-devel \
       glib2-devel gettext make freetype-devel fontconfig-devel \
       libXft-devel libpng-devel libjpeg-devel libtiff-devel giflib-devel \
       ghostscript-devel libexif-devel libX11-devel
 
sudo yum install cmake
 
sudo systemctl daemon-reload

Build and install mono 6.12.0.122 from source

(compile needed only for --with-large-heap=yes --enable-big-arrays=yes option ; latest mono 6 version when the AMI was build)


1. Build and install mono 6.12.0.122 from source: 

$ cd ~
$ wget -c https://download.mono-project.com/sources/mono/mono-6.12.0.122.tar.xz
$ tar xvf mono-6.12.0.122.tar.xz
$ cd mono-6.12.0.122/
$ ./configure --prefix=/opt/mono-6.12.0 --with-large-heap=yes --enable-big-arrays=yes
$ sudo make
$ sudo make install
$ export PATH=$PATH:/opt/mono-6.12.0/bin/

Specify mono6 for spawned jobs

Create a symlink to the mono directory: 

$ sudo ln -s /opt/mono-6.12.0/ /opt/mono-4.0.4

This is needed for backwards compatibility - OmicSoft Server maintains hard-coded paths to mono-4.0.4 for spawned-off jobs.

Remember to update ArrayServer.cfg with paths to mono6:

2. Check the mono version: 

$ cd /opt/mono-6.12.0/bin
$ ./mono --version

The output should look like this: Mono JIT compiler version 6.12.0.122 (tarball Mon Sep 7 13:38:50 UTC 2020)

3. Install the certs in mono store (mono has a separate cert store than the one used by Ubuntu; compiling mono does not populate the mono cert store) 

sudo /opt/mono-6.12.0/bin/certmgr --ssl -m https://resources.omicsoft.com
sudo /opt/mono-6.12.0/bin/cert-sync /etc/ssl/certs/ca-certificates.crt

//additional steps to manually retrieve the public certificates, if the above steps did not work 

wget -c https://resources.omicsoft.com/downloads/certs/resources.omicsoft.com.zip
unzip resources.omicsoft.com.zip
/path/to/mono/certmgr -add -c Trust Trust.crt


Libgdiplus

  • Install:
sudo amazon-linux-extras install epel
sudo yum install libgdiplus
  • The mono config file must be edited to explicitly specify the location of libgdiplus.so
  • Find where libgdiplus was installed:
$ ldconfig -p | grep libgdiplus
libgdiplus.so.0 (libc6,x86-64) => /lib64/libgdiplus.so.0
  • Edit the mono config file, usually located at (ex.):
 sudo vim /opt/mono-6.12.0/etc/mono/config

and add the following entry at the end of the file before </configuration>: 

...
        <dllmap dll="gdiplus" target="libgdiplus.so" os="!windows"/>
        <dllmap dll="gdiplus.dll" target="libgdiplus.so"  os="!windows"/>
        <dllmap dll="gdi32" target="libgdiplus.so" os="!windows"/>
        <dllmap dll="gdi32.dll" target="libgdiplus.so" os="!windows"/>
 
        <dllmap dll="gdiplus.dll" target="/lib64/libgdiplus.so.0"/>
</configuration>

Add appropriate SSL certificates

(mono has a separate cert store than the one used by Ubuntu; compiling mono does not populate the mono cert store) 

certmgr --ssl https://resources.omicsoft.com
sudo /opt/mono-6.12.0/bin/certmgr --ssl -m https://resources.omicsoft.com
 
sudo /opt/mono-6.12.0/bin/cert-sync /etc/ssl/certs/ca-bundle.crt

Verify with: 

certmgr -list -c -m Trust

Sqlite

  • Install Sqlite 3.7.14.1 2012-10-04
cd ~
wget -c http://www.sqlite.org/sqlite-autoconf-3071401.tar.gz
tar zxvf sqlite-autoconf-3071401.tar.gz
sudo mv sqlite-autoconf-3071401 sqlite
cd sqlite/
./configure --prefix=/opt/sqlite
make
sudo make install
 
 
sudo ./sqlite3 --version

Ulimit

Increase ulimit as described at Setting up ulimit for ArrayServer

Install OShell

Assume mono (regardless of version) has been installed in sudo /[path where mono was installed]/bin/mono. We install oshell in the home directory. 

cd /opt
sudo mkdir oshell
cd oshell
sudo wget -c http://omicsoft.com/software_update/OmicsoftUpdater.exe
sudo touch oshell.exe
sudo /[path where mono was installed]/bin/mono /opt/oshell/OmicsoftUpdater.exe
sudo /[path where mono was installed]/bin/mono /opt/oshell/oshell.exe --version

Make sure AWS CLI is installed (currently aws-cli/1.18.147)

aws --version

Install R

$sudo yum -y install R
$sudo R
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
 
Type 'q()' to quit R

Install Docker

Install Docker (20.10.*, see: https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository ) 

$ docker --version
  • Create the docker group.
$ sudo amazon-linux-extras install docker
$ sudo service docker start
  • Add user to the docker group.
$ sudo usermod -aG docker ${USER}
  • Log out and log back in so that the group membership is re-evaluated, by typing the following command (if you know the user pass):
$ su -s ${USER}

OR (alternatively) open a new ssh session.

  • Docker auto-start:
$ sudo chkconfig docker on
  • Verify that you can run docker commands without sudo.
$ docker run hello-world

Create a new AMI

  • On the EC2 Instance created above, in AWS console right click on the instance, go to Images and templates and press Create image
  • Give it a name different than the EC2 instance name
  • If an SSH is open to the EC2 Instance you will be disconnected for a while
Retrieved from "https://wiki.arrayserver.com/wiki/index.php?title=Build_AWS_Amazon_Linux_2_AMI_for_OmicSoft_Cloud_Computing&oldid=30969"