Steps to build AWS ArrayServer with cloud plugin

From Array Suite Wiki

Setup OmicSoft Server in AWS with OmicSoft Cloud plugin

Overview

As part of your OmicSoft Suite installation, OmicSoft Server can be installed on a physical machine or on a Cloud Virtual machine.

Steps in this article will focus on deploying OmicSoft Server to an Amazon EC2 instance, including configuring the AWS Cloud analysis add-on

This page is an overview of the configuration process; please contact ts-bioinformatics@qiagen.com for advice on your specific setup.

AWSFigure02.png


Steps for configuring and Launching OmicSoft Server in AWS

These steps from top-to-bottom describe the general steps to set up your cloud-based OmicSoft Server instance; e-mail ts-bioinformatics@qiagen.com for advice on your specific setup.

Launch and Configure Cloud-based OmicSoft Server

Launch an instance into subnet

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
  • On the dashboard, choose Launch Instance.
    • As a convenience, we have generated Ubuntu images, which already have required packages installed. Alternatively, you can select a minimal Linux or Windows image and follow instructions on Getting Started.
      • The OmicSoft Red Hat and Ubuntu AMI can be found by choosing "Community AMIs", then searching for the AMI names mentioned on the relevant pages.
  • Select appropriate instance type, e.g. "r4.xlarge".
  • Select appropriate VPC, Subnet.
    • Enable "Auto-assign Public IP"
    • Enable "Protect against accidental termination"
  • Attach EBS storage, for example 1000G "General Purpose SSD"
    • Uncheck "Delete on Termination"
  • Create tag with key "Name" and Value "ArrayServer"
  • Instance Security settings
    • This is highly dependent on user needs, but these are some common settings
      • TCP Ports 8064-8066 (for core OmicSoft Studio-OmicSoft Server connections, corresponding to ArrayServer.cfg parameters Port,Port2, and Port3)
      • TCP Port 22 for SSH to the VM
      • TCP ports 60660-60680 (for multi-threaded FTP connections, corresponding to ArrayServer.cfg parameters DataPortBegin and DataPortEnd)
    • Generally, the IP ranges will be restricted to a predefined set of IPs from the company network.
  • Launch with the "Omicsoft.Launching" key pair
  • Make a note of the public IP address.

Access AWS Instance

  • Read: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html
  • Start PuTTYgen (for example, from the Start menu, click All Programs > PuTTY > PuTTYgen).
  • Under Type of key to generate, select SSH-2 RSA
  • Click Load. By default, PuTTYgen displays only files with the extension .ppk. To locate your .pem file, select the option to display files of all types.
  • Click Save private key to save the key in the format that PuTTY can use. PuTTYgen displays a warning about saving the key without a passphrase. Click Yes.
  • Key passphrase: "omicsoft"
  • Connect with Putty
  • In the Host Name box, enter user_name@public_dns_name. Be sure to specify the appropriate user name for your AMI.
    • For an Amazon Linux AMI, the user name is ec2-user.
    • For a RHEL5 AMI, the user name is either root or ec2-user.
    • For an Ubuntu AMI, the user name is ubuntu.
    • Here I use ec2-user@xxxxxxx.compute-1.amazonaws.com
  • Category -> Connection -> SSH -> Auth, browse and load the .ppk file.

Mount EBS Storage

sudo file -s /dev/xvdb
  • If the output of the previous command shows simply data for the device, then there is no file system on the device and you need to create one.
  • Format the EBS drive if necessary
sudo mkfs -t ext4 /dev/xvdb
  • mount the EBS drive
sudo mkdir /arrayserver
sudo mount /dev/xvdb /arrayserver
  • To mount the EBS drive on every system reboot. Find the device uuid
sudo file -s /dev/xvdb
    • Add the following entry to /etc/fstab to mount an ext4 file system at mount point /arrayserver
UUID=90878434-dad9-4723-829a-xxxxxx /arrayserver	ext4	defaults,nofail	0	2
    • To check wheter the entry will work as expected:
sudo mount -a

Set Up ArrayServer

If an OmicSoft Red Hat or Ubuntu AMI was used, all Linux packages, including mono, libgdiplus, and Sqlite, have been installed in the AMI in /opt directory. Otherwise, please follow the steps in Install ArrayServer dependencies to install required packages.

To set up the ArrayServer application, follow steps in Install ArrayServer to get Omicsoft Updater and ArrayServer with configuration template.

Open Ports

ArrayServer uses certain ports for communications between client and server. These ports must be opened.

Start ArrayServer and get connected

Follow Typical way to update/restart ArrayServer to update/start/stop ArrayServer.

Password protect the ArrayServer Access

Set the following in ArrayServer.cfg

AutoCreateNewUsers=True
UserAuthorization=False

Then enforce CFRPart11 (optional)



Set AWS Credentials

Set correct region

Be sure to create S3 buckets, EC2 instances and key pairs in an AWS region close to your company.

Create key pair

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
  • In the navigation pane, under NETWORK & SECURITY, choose Key Pairs.
    • Example Key pair name: Omicsoft.Launching
    • The browser should automatically prompt to download/save the private key file "OmicsoftLaunching.pem"

Create VPC

  • Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
  • Choose desired VPC configuration. Create a simple VPC with a single public subnet by using the "VPC Wizard".
  • Remember the subnet ID to be used in AWS config

Create S3 Bucket

Create a storage location for user data that will be processed by OmicSoft. OmicSoft functions that use data stored in this S3 bucket and output to this S3 bucket will be run on spawned-off EC2 instances, allowing dynamic parallelization of data processing.

  • Sign into the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3
  • Click Create Bucket.
  • Create a Bucket Name, such as east.companyname.omicsoft; choose same region as the ArrayServer machine.
  • Remember to add access to the S3 bucket, and its folders, to the arrayserver policy.

Create IAM Users

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation pane, choose Users and then choose Create New Users
    • Example User name: arrayserver
    • "Access Key Id" and "Secret Access Key" will be generated for the new user. Be sure to make a note of these!

Create Policy

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation column on the left, choose Policies.
  • At the top of the page, choose Create Policy.
  • In "Create Your Own Policy", create policy name "arrayserver", and use the example policy to specify all of the required permissions.

Create Role

  • Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/
  • In the navigation pane, choose Roles, and then choose Create New Role.
  • Give the Role name "arrayserver"
  • Select "Amazon EC2"
  • Attach "arrayserver" policy created above

Attach Policy To User

  • Select "arrayserver" user created above
  • Under "Permissions" tab, attach policy

Related Articles

EnvelopeLarge2.png