ArrayServer folder mapping and management

From Array Suite Wiki

Overview

Genomics data can be very large, and users want to avoid moving/copying/duplicating large data files unless necessary. ArrayServer provides folder mapping to map mounted folders into the ArrayServer file system. This allows folders/files outside of the ArrayServer BaseDirectory hierarchy to be accessible within ArrayServer.

Basic Usage

ArrayServer admins can add folder mapping lines in the ArrayServer.cfg [Folder] section like the one below:

Linux-based Server folder

[Folder]
IData_TestDataSets=/mnt/IData/TestDataSets/
IData_Users=/mnt/IData/Users
IData_OmicTest=/mnt/IData/OmicTest
IData_ArrayLand=/mnt/IData/ArrayLand
InternalData=/InternalData

Windows-based Server folder

[Folder]
IData_TestDataSets=F:\IData\TestDataSets
IData_Users=F:\IData\Users
IData_OmicTest=F:\IData\OmicTest
IData_ArrayLand=F:\IData\ArrayLand
InternalData=G:\InternalData

cloud S3 folder (more details, including how to map S3 buckets from multiple AWS accounts, can be found here.

[CloudFolder]
GaryCloudFolder=/east.test.omicsoft/gary
GaryCloudFolder2=/omicsofttest1

After restarting the server, files and subfolders inside these mapped folder will show up in the ArrayServer root directory:

Foldermapping.png


Limitations

===Virtual folders map to physical folders only===Virtual folders can only be mapped to physical paths; adding a folder mapping where a part of the target folder, was already mapped and represents a virtual path is not supported.

The below example illustrates a use case which is not supported:

[Folder] IData=/mnt/IData/ IData_TestDataSets=/IData/TestDataSets/

Virtual folder mappings to S3 locations

For cloud folder mappings prefixing the AWS S3 paths with 's3://' is not needed and currently not supported. Thus when specifying cloud folder mappings the AWS S3 path would need to start with a leading '/'.


Oscript and Escript mapping to virtual folders

An OScript/EScript/etc.. can be executed only if all folders that appear in the script are virtual mapped folders, system-defined (built-in) folders like /Users/... , /Projects/..., /ServerProjects/... , or absolute paths starting with $$ or %% .

Folders manually created by the user directly under the OmicSoft Server FTP root are not supported.

For example, if /mnt/IData/FtpRoot is the mapped location of FtpRoot, one cannot refer to manually-created /mnt/IData/FtpRoot/MyInputDirectory.

Learn more about virtual folder mapping at Array_Server_Virtual_Path.

Tips

It is better to map the highest level of directory to avoid too many folder mappings. Taking the example above, it is better to map as below:

However if such a path has been specified OmicSoft Server will throw an error on startup.

[Folder]
IData=/mnt/IData/
InternalData=/InternalData

All ArrayServer users are accessing these folders through the user account running ArrayServer, such as "sys_omicsoft".

The account "sys_omicsoft" must have read and write access to these folders in the server machine. Permissions are assigned by folder owners or admins fromServer Tab | Manage | Manage users:

FolderUserManagement.png

OmicSoft Studio deals with mapped folders as they were users. Hence, each mapped folder will be found in the users list, with an ID formatted as "_[parentFolder_mappedFolder]. For example, "_idata_omictest_" is the user for folder mapping "IData_OmicTest=/IData/OmicTest".

Tips.png Admins must assign a group of users or a single user to their folder reader and editor user groups. By default, standard (or above) users can only read files and all editor (or above) users can write files to these mapped folders.



Manage folder mapping

Since ArrayServer 9.0, admin can manage/add folder mapping on the fly without server restart.

ManageFolderMapping.png

Control User Access to Mapped Subfolders

If a mapped folder contains subfolders that have restricted access, ArrayServer admins can control which users or groups can access these data.

  1. Create a user group to define those who should have access to restricted subfolders.
  2. Add Array Studio users to this user group.
  3. Create a pseudo-user with naming pattern _mappedFolder/path/to/RestrictedFolder_.
  4. Add the appropriate user group as a Folder Reader group for the pseudo-user's "folder".

For example, consider a mapped folder "workspace", with three subfolders:

/workspace/
├── Clinicaldata
├── CNSdata
└── Commondata

Only researchers in the CNS group should access CNSdata, while only researchers in the Clinical group should access Clinicaldata; both groups should have access to Commondata.

Step 1: Create user groups to define users with access to folders

Under Manage | Manage User Groups, define groups for users that can have access to CNSdata or Clinicaldata:

AddUserGroup CNSdata.pngAddUserGroup Clinicaldata.png

There is no specific naming scheme required, but consider making the purpose of the user group clear. The new user groups should be listed in Manage User Groups:

ManageUserGroups Window.png

Step 2: Add users to the newly-created user groups

In the Manage User Groups window, select one of the new user groups (e.g. workspace-clinicaldata_access), then click Set Members. Choose users that should belong to this group (e.g. user name clinicalresearcher). Repeat for members of workspace-CNSdata_access.

SetMembers Clinicaldata.pngSetMembers CNSdata.png

Step 3: Create a "pseudo-user" that specifies the subfolders with restricted access

In Manage | Manage Users, click Add User, and create a user in the pattern _mappedFolder/path/to/RestrictedFolder_. This pseudo-user will represent the path to the subfolder that should be restricted.

In this example, two new users are created: _workspace/clinicaldata_ and _workspace/cnsdata_.

AddUser Clinicaldatafolder.pngAddUser CNSdatafolder.png

Tips.pngCapitalization is not important, but exact spelling of the ArrayServer folder path is required.


Step 4: Add user groups to the "pseudo-user's" permitted folder readers

In Manage | Manage Users, select one of the "pseudo-users" that represents a restricted folder (e.g. _workspace/clinicaldata_.

In the right window, click the Privilege | [Folder Reader] User groups tab.

Select the User groups that should be able to see this folder (e.g. workspace-clinicaldata_access). Do the same for _workspace/cnsdata_.

Be sure to click Update User to save your changes!

ManageUser AddClinicaldatagroup.pngManageUser AddCNSdatagroup.png

Tips.pngMake sure that the restricted access user groups are permitted to read the mapped superfolder (e.g. user groups should be added to the "readers" for the _workspace_ pseudo-user), or they will not be able to access the subfolders.


Results

The user "clinicalresearcher" will see the following folders:

Cnsread.PNG

but the access to the CNSData Folder is restricted by the folder's read policy.


Note

ArrayServer file paths use a virtual folder directory system, similar to FTP servers. If you map a folder, such as (ArrayServer folder IData=> physical folder /mnt/IData), it will create an empty folder, IData, in the physical ArrayServerBaseDirectory/FtpRoot folder. When a user browses files in ArrayStudio/ArrayServer, it will display the FtpRoot folder as the root folder, and translate the link IData to the actual folder /mnt/IData.

However, if you map a folder such that your ArrayServer BaseDirectory itself is a subfolder of your mapped folder, it will cause a loop in the FTP.

For example, if the BaseDirectory is /mnt/IData/omicsoft/ArrayServerHome, and /mnt/IData is mapped as a virtual subdirectory, an improper FTP loop will be setup leading to various problems. One tell-tale sign is that clicking the FtpRoot button or navigating to the Root folder displays the full Server path (as opposed to '/'):

FoldermappingLoop.png

If other folders in IData need to be accessed in ArrayServer, the admin should find a better way to add these folders in /mnt/IData, such as

IData_data1=/mnt/IData/data1
IData_data2=/mnt/IData/data2
IData_data3=/mnt/IData/data3


Related Articles

EnvelopeLarge2.png