ArrayServer folder mapping and management
Genomics data can be very large, and users want to avoid moving/copying/duplicating large data files unless necessary. ArrayServer provides folder mapping to map mounted folders into the ArrayServer file system. This allows folders/files outside of the ArrayServer BaseDirectory hierarchy to be accessible within ArrayServer.
ArrayServer admins can add folder mapping lines in the ArrayServer.cfg [Folder] section like the one below:
Linux-based Server folder
[Folder] IData_TestDataSets=/mnt/IData/TestDataSets/ IData_Users=/mnt/IData/Users IData_OmicTest=/mnt/IData/OmicTest IData_ArrayLand=/mnt/IData/ArrayLand InternalData=/InternalData
Windows-based Server folder
[Folder] IData_TestDataSets=F:\IData\TestDataSets IData_Users=F:\IData\Users IData_OmicTest=F:\IData\OmicTest IData_ArrayLand=F:\IData\ArrayLand InternalData=G:\InternalData
cloud S3 folder (more details, including how to map S3 buckets from multiple AWS accounts, can be found here.
[CloudFolder] GaryCloudFolder=/east.test.omicsoft/gary GaryCloudFolder2=/omicsofttest1
After restarting the server, files and subfolders inside these mapped folder will show up in the ArrayServer root directory:
===Virtual folders map to physical folders only===Virtual folders can only be mapped to physical paths; adding a folder mapping where a part of the target folder, was already mapped and represents a virtual path is not supported.
The below example illustrates a use case which is not supported:
[Folder] IData=/mnt/IData/ IData_TestDataSets=/IData/TestDataSets/
Virtual folder mappings to S3 locations
For cloud folder mappings prefixing the AWS S3 paths with 's3://' is not needed and currently not supported. Thus when specifying cloud folder mappings the AWS S3 path would need to start with a leading '/'.
Oscript and Escript mapping to virtual folders
An OScript/EScript/etc.. can be executed only if all folders that appear in the script are virtual mapped folders, system-defined (built-in) folders like /Users/... , /Projects/..., /ServerProjects/... , or absolute paths starting with $$ or %% .
Folders manually created by the user directly under the OmicSoft Server FTP root are not supported.
For example, if /mnt/IData/FtpRoot is the mapped location of FtpRoot, one cannot refer to manually-created /mnt/IData/FtpRoot/MyInputDirectory.
Learn more about virtual folder mapping at Array_Server_Virtual_Path.
It is better to map the highest level of directory to avoid too many folder mappings. Taking the example above, it is better to map as below:
However if such a path has been specified OmicSoft Server will throw an error on startup.
[Folder] IData=/mnt/IData/ InternalData=/InternalData
All ArrayServer users are accessing these folders through the user account running ArrayServer, such as "sys_omicsoft".
The account "sys_omicsoft" must have read and write access to these folders in the server machine. Permissions are assigned by folder owners or admins fromServer Tab | Manage | Manage users:
OmicSoft Studio deals with mapped folders as they were users. Hence, each mapped folder will be found in the users list, with an ID formatted as "_[parentFolder_mappedFolder]. For example, "_idata_omictest_" is the user for folder mapping "IData_OmicTest=/IData/OmicTest".
Manage folder mapping
Since ArrayServer 9.0, admin can manage/add folder mapping on the fly without server restart.
Control User Access to Mapped Subfolders
If a mapped folder contains subfolders that have restricted access, ArrayServer admins can control which users or groups can access these data.
- Create a user group to define those who should have access to restricted subfolders.
- Add Array Studio users to this user group.
- Create a pseudo-user with naming pattern _mappedFolder/path/to/RestrictedFolder_.
- Add the appropriate user group as a Folder Reader group for the pseudo-user's "folder".
For example, consider a mapped folder "workspace", with three subfolders:
/workspace/ ├── Clinicaldata ├── CNSdata └── Commondata
Only researchers in the CNS group should access CNSdata, while only researchers in the Clinical group should access Clinicaldata; both groups should have access to Commondata.
Step 1: Create user groups to define users with access to folders
Under Manage | Manage User Groups, define groups for users that can have access to CNSdata or Clinicaldata:
There is no specific naming scheme required, but consider making the purpose of the user group clear. The new user groups should be listed in Manage User Groups:
Step 2: Add users to the newly-created user groups
In the Manage User Groups window, select one of the new user groups (e.g. workspace-clinicaldata_access), then click Set Members. Choose users that should belong to this group (e.g. user name clinicalresearcher). Repeat for members of workspace-CNSdata_access.
Step 3: Create a "pseudo-user" that specifies the subfolders with restricted access
In Manage | Manage Users, click Add User, and create a user in the pattern _mappedFolder/path/to/RestrictedFolder_. This pseudo-user will represent the path to the subfolder that should be restricted.
In this example, two new users are created: _workspace/clinicaldata_ and _workspace/cnsdata_.
Step 4: Add user groups to the "pseudo-user's" permitted folder readers
In Manage | Manage Users, select one of the "pseudo-users" that represents a restricted folder (e.g. _workspace/clinicaldata_.
In the right window, click the Privilege | [Folder Reader] User groups tab.
Select the User groups that should be able to see this folder (e.g. workspace-clinicaldata_access). Do the same for _workspace/cnsdata_.
Be sure to click Update User to save your changes!
The user "clinicalresearcher" will see the following folders:
but the access to the CNSData Folder is restricted by the folder's read policy.
ArrayServer file paths use a virtual folder directory system, similar to FTP servers. If you map a folder, such as (ArrayServer folder IData=> physical folder /mnt/IData), it will create an empty folder, IData, in the physical ArrayServerBaseDirectory/FtpRoot folder. When a user browses files in ArrayStudio/ArrayServer, it will display the FtpRoot folder as the root folder, and translate the link IData to the actual folder /mnt/IData.
However, if you map a folder such that your ArrayServer BaseDirectory itself is a subfolder of your mapped folder, it will cause a loop in the FTP.
For example, if the BaseDirectory is
/mnt/IData/omicsoft/ArrayServerHome, and /mnt/IData is mapped as a virtual subdirectory, an improper FTP loop will be setup leading to various problems. One tell-tale sign is that clicking the FtpRoot button or navigating to the Root folder displays the full Server path (as opposed to '/'):
If other folders in IData need to be accessed in ArrayServer, the admin should find a better way to add these folders in /mnt/IData, such as
IData_data1=/mnt/IData/data1 IData_data2=/mnt/IData/data2 IData_data3=/mnt/IData/data3