                    INSTALLATION

This document describes the installation  of Spillbox software to utilize the free
compute resources of remote data center. User level software and documentation
is a separate bundle.

Requirements at remote data center:

    1. Need a machine ( 32-64 cores, 128-256Gb, 2TB and above local NVME storage) to run Spillbox web server.  IT needs to create a Unix service account which will be used 
       to run Spillbox web server. Service accounts should have SUDO (super user do ) access without needing a password on the webserver machine.
       Service account should have Docker/Singularity access. This machine should have good amount of storage 
       in /var ( or somewhere local, about 20gb ) to accommodate docker images.
    2. IT needs to have a list of OS patches to create Docker/Singularity images.  
    3. IT needs to allocate an adequate set of machines with enough local storage (32-64 cores, 256Gb, 2TB and above local NVME storage ) for data caching to be used by Spillbox NFS servers. NVME storage should be mount locally with world writable permission.
    4. If using Docker, then IT needs to setup and maintain Docker image orchestration by installing
       docker daemon on all machines. Spillbox can create a container registry server on the web server.
       Need docker-ce and docker-ce-cli containerd.io packages on all machines.
    5. Common storage ( NFS mounted, visible to all hosts ) to install Spillbox software
                - Spillbox installation area creates two directories parallel to its installation
                  ( ../sites and ../customers ) which may grow over time and it keeps per project/user data.
                  This will be a few gb only.
    6. NIS/LDAP support required on host and in container. All Spillbox users must have an account with home directory on cloud hosts machines.


        NFS server cannot be run in a container on azure Cloud due to a bug/limitation in Azure Cloud. For NFS server on run on host without container, followinngs are needed on all NFS servers:
        - patchelf binary/pkg  
        - sudo without password by the service account on the nfs server
        - max number of open files for all users should be 1024000. 
          Entry in /etc/security/limits.conf on webserver and nfs-server
*                soft    nproc           65536
*                hard    nproc           65536
*                soft    nofile          1024000
*                hard    nofile          1024000

        
        - Below kernel configurations on web server and nfs-server ( /etc/sysctl.conf):
sunrpc.tcp_max_slot_table_entries=65536
fs.file-max=10240000
vm.min_free_kbytes=4124416
net.core.somaxconn=8192
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.tun0.disable_ipv6=1

      # please do "sysctl -w ..." for each above entry for already running kernel.


There are two steps:

      In Phase 1: One time installation by an account having docker access. This account
          must have write permission in the area where Spillbox software is installed.
          Better way would be for this account to download and install the software at
          common NFS location.

          This phase will create common docker image and docker registry , if needed.
          This account also configures the config file used by all.


      In Phase 2: This step starts a webserver. Any user can start a webserver and need not
          have docker access if docker container can be spawned through LSF app.

          If docker container cannot be spawned through LSF app then server should be
          run by a common service account which has docker access ( in docker group).
          All users can use this server for their clients on premise. In this case,
          $HOME of the servrice account should be on the local disk on the server machine,
          not on a common NFS storage.

Phase 1 Steps:

     - Login to designated web server machine with account created for Spillbox
     - Install docker and run docker server ( sudo service docker start )
     - On NFS partition ( no. 6 above ), create a directory spillbox
        ( mkdir -p spillbox ) for Spillbox software download and installation.
     - Download software from https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/v1.3/spill-dc.tar.gz
     - unzip and untar it in spillbox
     - copy terraform/etc/spillbox_config.example to terraform/etc/spillbox_config
     - Edit terraform/etc/spillbox_config configuration file as per your site ( described below )
     - Run "terraform/server_image.sh  build"            
            This will create docker image and docker registry server
            
     - To destroy, please run "terraform/server_image.sh destroy"

     - This account should also configure system-wide terraform/etc/spillbox_config.

     - If you are using Spillbox docker registry server then please make Spillbox
       registry server as insecure private registry server in /etc/docker/daemon.json
       on all of your machines.

Entries in terraform/etc/spillbox_config :

    Please note that all entries are not must. Tool will inform about missing must field at runtime.

    disable_server_image=false
            If you have your own docker image, Spillbox can use that.

    disable_registry_server=false
            If you have your own docker orchestration system through LSF etc,
            Spillbox will not create docker registry server used as docker repo
            for docker orchestration.

    registry_server=""
            Please provide host:port here if using own docker registry server.

    cache_dir=/cache_dir
            This is a snapshot area where local data center's data will be cached.
            This should be a large local or NFS partition.
            If value is "" then file system will be kept in docker's shared memory.

    memory_size="" // 100m or  1g ...
            File server is managed by docker. If cache_dir is "" then files are in
            shared memory of the server. Default shared memory of the docker is low.
            Please provide memory_size if using spillbox container.
  
    cache_server=10.1.10.103
             if cache_dir is NAS then the IP address of the NAS server.

    hub_port_min=8090
            For all projects and users, currently hub will run on the web server itself.
            So multiple hub will be on one machines and each hub will use a unique
            port to communicate with the nfs server of that project. Hub will hunt for
            free unix port starting from this port number.

    docker_base_image="centos:7"
            If you have your own base image to be used company wide, please add here.
            Users can override this.

    local_mount_paths="/tools"
            If remote data center has its own tool area and need not cache that from local
            data center, please add here.

    exclude_mount_paths="" // list seperated by spaces
            Force exclusion of these nfs/local mount paths from being exported
            from on-premise.

    disable_docker_image=true
            This disables docker image per user/project.

    disable_docker_privileged=false
            Disable --privileged option in docker being passed.

    cloud_shell="/bin/csh"
            Change default shell in Docker.

    nfs_port_min=10000
            Starting port number for NFS server.

    docker_wrapper=""  // full paths separated by spaces
            User's runtime commands are executed in a docker wrapper such that command runs
            as user. Spillbox has a default wrapper. Customer can bring their own wrapper.
            Multiple approved wrappers can be provided. Only approved wrappers are allowed.

    client_hosts=""
            In case if you want to enforce all clients (onprem) to be run from
            a particular machine.

    container_with_automount=false
            In case you are using your own container and container already mounts
            relevant local file systems and does not need help from Spillbox for
            local_mount_paths. This is not common.

    job_manager="" // slurm, uge, lsf, lsfapp
            Job manager used for running file server.

    job_submit_command="" // sbatch, bsub, qsub ...
            job manager submit command 
            
    job_kill_command="" // scancel,  bkill ...
            job manager' command to cancel a job

    user_setup_file="" // path to a setup file
            If setup file exists, then it will be sourced before spawning the container(running the job/batch job).

    prexec_file="" // on-premise filepath to be executed as first thing in the container 
            If prexec_file exists in the container, then it will be sourced quietly before executing the user command. Please note the file path and file exists on premise.

    container="" // default docker
            "docker" and "singularity" containers are supported. Applies for NFS servers as well
            as for compute servers.

    docker_path=
            In case "docker" in your network is at some other location.
    singularity_path= 
            In case "singularity" in your network is at some other location.

    singularity_base_image=""
            Location of singularity .sif image. Please provide full path.

    runtime_container=   // default is whatever is during NFS server
            At compute time, one could use singularity container even if nfs
            servers are docker container.

    workdir=   // default log/work directory is  .spillbox directory in $HOME
            of the service account which runs the server. User can configure 
            this to any NFS directory, instead of default $HOME.

    batch_command_allowed=  // default is ALL
            This should be either left "" ( disallow) or filled with set of commands,
            separated by spaces. These are usually job manager's batch commands 
            like sinfo, qsub, bsub, nc. Please provide full path also. 

    docker_option =    // default is ""
            Instead of users providing options through "docker run -docker_option" for
            each command line, this adds globally for all commands.

    uge_dockerized_cluster =  // default is false
            UGE is configured to spin Spillbox docker.

    lsf_dockerized_cluster =  // default is false
            LSF is configured to spin Spillbox docker. Please see 
            spillbox_lsf_docker_app_profile_configuration.docx.

    docker_register_port =  // default is 5000
            Port used by Spillbox docker registry server.

    server_base_image  =    // default is docker_base_image 
            If NFS servers are configured to use container then container image
            to be used by it.

    remote_command_allowed =  // default is "", list seperated by spaces
            <remote_cmd> in "splrun run -remote <remote_cmd>" will run by server
            with server userid, not non-prem user's userid.

    rootless_docker  == // default is false
            Enable rootless docker.

    nameserver = // default is nameservers on webserver host
            List of nameservers seperated by spaces, to be used in container.

    uge_docker_option  = // default is ""
            In case of UGE spawning docker container, extra docker option to be supplied.

    setup_bapp = // default is false
	    Set this to true if the spillbox specific lsf bapp setup guide needs to be enabled.

    bapp_name = // default is none
	    Set this to the name of lsf application profile. This in effect if setup_bapp is true.

    bapp_image_name= // default is none
	   Set this to the docker image that has to be used. This in effect if setup_bapp is true.

    lsf_cluster_name= // default is none
	    Set this to the lsf clustername that is needed. This in effect if setup_bapp is true.

    nc_setup_bapp = // default is false
	    Set this to true if the spillbox specific nc bapp setup guide needs to be enabled.

    nc_bapp_name = // default is none
	    Set this to the name of nc resource name. This in effect if nc_setup_bapp is true.

    nc_bapp_image_name = // default is none
	   Set this to the docker image that has to be used. This in effect if nc_setup_bapp is true.


    nc_vov_swd= // default is none
       Set this to the main vov server's swd path. This in effect if nc_setup_bapp is true.

    nc_vov_project= // default is none
       Set this to the vovproject for which the setup to be done. This in effect if nc_setup_bapp is true.

    checkpoint_objstore=    // true/false
        Provide a object store file system (juicefs with S3) to keep the checkpoint data
        To enable object store (To enable object store, set these variable in spillbox_config:
            checkpoint_objstore, checkpoint_image_dir, checkpoint_bucket, checkpoint_objport )

    checkpoint_image_dir= // no default
       Provide a world writable NFS directory with enough space for checkpoint
       to keep data in this directory.

    checkpoint_bucket=     // (only alphabet, number and - are allowed, and the length should be 3 to 63 characters)
        Bucket used for storing juicefs data, only empty buckets is allowed to be configured for juicefs storage
        If bucket doesn't exist, new bucket will be created.

    checkpoint_objport=     // port number
        Currently used fo redis database, necessary for juicefs to track checkpoint metadata

    hybrid_mount_paths = "" // list of mount paths separated by spaces
       Enable fully synchronized data sync between on-premise and Cloud
       for set of mount paths hierarchically. This is equivalent to 
       enabling -download-hier, missing-hier and sync-hier in the snapshot option
       for these directories. Special keyword "ALL" enables globally on all mount paths. 

    "hybrid_mount_paths_exclude" = "" // list of mount paths separated by spaces
       Diasble fully synchronized data sync between on-premise and Cloud
       for set of mount paths hierarchically. This will be applied after
       hybrid_mount_paths Environment variable SPILLBOX_HYBRID_MOUNT_PATHS_EXCLUDE
       also does the same.

    boostmode=  // default is false
        Set this to true to enable boostmode

    boost_server_cache_size=""  // default will be 500M, possible values K|M|G
        Configures the cache size on the server. Setting this variable will allocate
        the specified amount of host RAM for the spillbox project

    boost_client_cache_size=""  // default will be 500M, possible values K|M|G
        Configures the cache size on the client. Setting this variable will allocate
        the specified amount of host RAM for the spillbox project

    "wan_timeout" = 900 // set WAN timeout in min ( 0 for no timeout )

Phase 2 Steps:

        Can be run as any user if docker orchestration does not need
        user to be in docker group 
     - Run the webserver on any machine through LSF/ssh:
         terraform/run_server.sh  <projectname>  

       This starts webserver and returns SPILLBOX_SERVER value as host:port.
       Clients onprem should set this value.
     
Q. How do I use multiple container images or bring my own set of container images?
Ans:   Create  container image aliases in file terraform/etc/spillbox_image_map having entry
        of form:
             <short_name>=<host>:<port>/<image>

       It is expected that the admin will  push these images  on registry servers (<host>:<port>)
       on their own.  Users can select image at runtime:
            splrun run -image <short_name> <cmd> <args>
Q. How do I preconfigure the job manager system configuration paramters mapped for NFS server?
Ans:   Create a NFS Model mapped config parameters in file terraform/etc/spillbox_nfsMode_config 
        having the entry of form :
        <machine Model short name>="<config paramaters>"
        example: medium="-n 4" allow the LSF job manager to allocate 4 slots of CPU to the NFS server
        
       To configure the default config parameter IT team shall provide the default parameters config as 
       example: default="small"
                small="-n 2"
		medium=-n 4"

       However the client shall override the default with other predifined config options given by IT team. 
       By providing the "nfs_model" config paramter in project.config file from client end
       example: nfs_model="medium"
       With nfs_model defined , the NFS server shall allocate 4 slots of CPU to NFS server , 
       eventhough the default is small

Q.  How to enable Checkpoint for Spot Instances? 
Ans:   If Checkpoint is not enabled, job will restart from beginning on spot preemption.
    To enable running before the preemption, checkpoint needs to be done at regular intervals,
    controlled by user job by job. To enable checkpoint container should have Spillbox
    version of crui executable. Please add below lines in your container Dockerfile:
         RUN yum -y install asciidoc xmlto libbsd-devel libnet libnet-devel protobuf protobuf-c protobuf-c-devel protobuf-compiler protobuf-devel protobuf-python libnl3-devel libcap-devel python3 nftables
         RUN (cd /tmp/; wget https://s3.us-west-1.amazonaws.com/spillbox.io-public/criu/criu.tar; tar xvf criu.tar; cd /tmp/criu-criu-dev/; make clean ; make  install)

    criu will need NFS space to keep images. Please provide the value for checkpoint_image_dirin config file (see above). 

Q. How to enable/create objectstore filesystem ?
Ans:  Set these values : checkpoint_image_dir, checkpoint_objstore, checkpoint_bucket, checkpoint_objport (terraform/etc/spillbox_config)
      Run the script :: terraform/objstore.sh <create/delete>
            - It will create S3 bucket to store data and will create redis database to store its metadata.

Q.  How can I get help from Spillbox team?
Ans:    Please email support@spillbox.io.

