EnableAWSSpot

From Array Suite Wiki

Definition

Oscript parameter that tells the application whether or not AWS Spot Instances should be used. When Spot Instances are used, you pay the Spot price that's in effect for the time period your instances are running (more details here).

Possible values: 

   False (default - OnDemand instances are used)
   True (Spot instances are used)

AWS Spot Instance best practices

AWS Spot instances can deliver considerable savings over On-demand, because it is launching from a pool of unused instances. However, Spot instances carry the risk of termination at any point by AWS if the pool of unused instances is depleted. For this reason, we recommend the following Best Practices:

  • If you are routinely running many smaller AWS jobs, enable Spot Instances through ArrayServer.cfg; these are unlikely to fail after launch
    • use the new parameter "FallbackToOnDemand=True" within your launched oscript to automatically launch the job on an on-demand instance if no Spots are available at the start
  • If you are routinely running longer analyses (>2 hours per job), we recommend using On-demand instances by default, and using the Oscript parameter "EnableAWSSpot=True" to activate Spot instances.
    • EnableAWSSpot is only active as an oscript parameter for certain procs (see below); e-mail omicsoft.support@qiagen.com if you need additional proc support.
    • We have observed an increase in AWS terminating individual instances after 8+ hours running, because of sudden increase in demand from the system.


AWS recommends spot instances for (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-best-practices.html) : 

"stateless, fault-tolerant, flexible applications"
"not suitable for workloads that are inflexible, stateful, fault-intolerant, or tightly coupled between instance nodes. They're also not  recommended for workloads that are intolerant of occasional periods when the target capacity is not completely available. We strongly warn  against using Spot Instances for these workloads or attempting to fail-over to On-Demand Instances to handle interruptions"
"Spot does not guarantee that you can keep your running instances long enough to finish your workloads"


Enable spot instances as default setting for all jobs

One method to enable AWS spot instances through OmicSoft Server is to add this parameter to the [Cloud] options section in ArrayServer.cfg. Simply setting this parameter will request spot instances at the default max bid price (same as on-demand price) for any job that will run on Cloud. For more Cloud settings and options, please see ArrayServer.cfg#Cloud_section.

Enabling spot instances in script

If the user wants to enable spot instances for an individual analysis, the user can set /EnableAWSSpot=True in the oscript's /Options section. This will overwrite the default option specified in ArrayServer.cfg for the duration of these jobs. Please note, Oscripts supporting this flag can be found at the bottom of this page. Specifying spot instance price in script

Warning.png WARNING: We recommend using default price as the on demand price as max price to minimize job termination risk


Additionally, only in the oscript, in the Options section, when /EnableAWSSpot is set to True, user can also specify a spot price she is willing to pay, with parameter /SpotBidMaxPrice=x.xxxx. This will have an effect only for the duration of these jobs. If spot instances are more expensive at the time of the run, the oscript will fail.

e.g. /EnableAWSSpot=True /SpotBidMaxPrice=0.045


Example script:

Begin SentieonDnaSeqPipeline /Namespace=NgsLib /RunOnServer=True;

Files

"..";

Reference Human.B37.3;

Haplotyper /emit_mode=gvcf;

Options /PairedEnd=True /OutputFolder="/CloudFolderSupport/Users/lucy.williams/ARRS_526_SpotTest/June26" /ImportSentieonResults=True /ThreadNumberPerJob=4 /UseDev=True /EnableAWSSpot=True /SpotBidMaxPrice=0.60 /InstanceType=c4.8xlarge /ParallelJobNumber=50;

Output Test;

End;

No spots are available: FallbackToOnDemand

A request for an AWS spot instance can fail for various reasons - insufficient available capacity, max price for the bid price too high, etc...

A new parameter (FallbackToOnDemand) was be added to all procs that support now the EnableAWSSpot=True parameter.

The new parameter is ignored if EnableAWSSpot is False or missing.

When EnableAWSSpot parameter is set to True, in a script or in ArrayServer.cfg, the user can also specify the FallbackToOnDemand parameter.

When FallbackToOnDemand=True, if the request to run the AWS EC2 spot instances fails after 5 retries for any reason (caused by AWS, not Array Suite), the instance creation request will be send again, this time for a standard on-demand instance (https://aws.amazon.com/ec2/pricing/on-demand/).

This does not cover the situation when AWS decides to stop an already running EC2 spot instance. An on-demand instance will incur higher costs than using a spot instance, but at least the processing won't be terminated due to causes beyond user's control.

Pricing Examples

We explore two scenarios, starting from the assumption that user runs one spot instance for two hours, and after one hour, the current spot price doubles.

Scenario 1

Description: User sets a valid spot price (i.e. higher than the current spot price). Parameters /EnableAWSSpot=True /SpotBidMaxPrice=0.045 Your spot price = $0.045 Spot price at start of run = $0.025 OnDemand price = $0.125 Spot price after one hour of run = $0.05 Outcome: You pay $0.025 for the first hour of run, then the instance is terminated by Amazon because your spot price is lower than Spot price after one hour run.

Scenario 2

Description: User does not set any spot price. Parameters /EnableAWSSpot=True Your spot price = $0.125 (not specified by user, but Amazon will assume it is OnDemand price) Spot price at start of run = $0.025 OnDemand price = $0.125 Spot price after one hour of run = $0.05 Outcome: You pay $0.025 for the first hour of run and $0.05 for the second hour of run.


One has to consider that no matter what spot price you specify (or don't specify at all), you always pay the current spot price at the time of your run. So, it does not matter that in Scenario 1 user specified a smaller spot price than the one in Scenario 2, because in both situations, for the first hour of run, the charges were the same. More to this, in Scenario 1, user is at risk of having its instances terminated by Amazon and losing all work, if current spot price rises above user's initial spot price. On the other hand, if you do not specify any spot price, you are going to pay at most the OnDemand price.

Supported Procs

Please note that all procs that can run on AWS Cloud will benefit from spot instances if EnableAWSSpot is set to True in ArrayServer.cfg.

In addition, the following list of procs support /EnableAWSSpot and /SpotBidMaxPrice as oscript parameters.


  • ProcConvertNgsFiles
  • ProcCountErcc
  • ProcDemultiplexNgsFiles
  • ProcExportNgsData
  • ProcFilterNgsFiles
  • ProcMapLongRnaSeqReads
  • ProcMergeNgsFiles
  • ProcPreviewDnaSeqReads
  • ProcPreviewLongDnaSeqReads
  • ProcPreviewLongRnaSeqReads
  • ProcPreviewMirnaSeqReads
  • ProcPreviewRnaSeqReads
  • ProcPreviewRnaSeqReadsToTranscriptome
  • ProcReportPairedEndFusionGenes
  • ProcRestoreUnoptimizedBamFiles
  • ProcSampleNgsFiles
  • ProcSCLandTools
  • ProcSearchNgsAdapters
  • ProcSearchNgsTags
  • ProcSentieonDnaSeqPipeline
  • ProcSentieonTNSeqPipeline
  • ProcSplitNgsFileByTag
  • ProcSummarizeFlagStatistics

Common Errors

Price too low: if you see an instance creation error, you will be notified if attempt was unsuccessful because your bid was too low. 

[00:00:40] Error occurred from OJobProcess.Run. Error message=Instance creation is not successful.
Error=Your Spot request price of 0.3 is lower than the minimum required Spot request fulfillment price of 0.563.

Price is zero or negative: if you set the price to /SpotBidMaxPrice=0.0000 or /SpotBidMaxPrice=-0.5000, Amazon will set the default, OnDemand price for you.

Potentially useful page for Spot Instance Pricing 

“The provided credentials do not have permission to create the service-linked role for EC2 Spot Instances.”

Solution: Ask admin to request a spot instance once through AWS Console, and all required roles will be created. Admin can cancel the request and terminate the spot instance once it is launched. After the first request, all ArrayServer request of spot will be working.

More details see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-requests.html#service-linked-roles-spot-fleet-requests


Further search terms: enable aws spot, aws spot, enableawsspot

Retrieved from "https://wiki.arrayserver.com/wiki/index.php?title=EnableAWSSpot&oldid=31545"