                README - SPILLBOX DATACENTER TO DATACENTER (SPL-DC)


Spillbox is a hybrid Cloud product which can also be used to burst your on-premise
compute job on a remote Datacenter to use its extra available resources. 
This is done in a few simple steps:

    1. Download the executable and untar it.
            curl -o splfs.tar.gz https://s3-us-west-1.amazonaws.com/spillbox.io-public/spill-dc/v1.3/splfs.tar.gz
            tar xvfz splfs.tar.gz

       The above steps should create an executable for splrun, along 
       with a few more executables.

    2. Default value in spillbox may be sufficient but if you want to customize or 
       override a few defaults, please create a config file. Following config file creates
       docker network with an overlay driver, default is host driver, needed for jobs 
       where dockers need to communicate among itself.
cat > ./myconfig_file <<eoj
"docker_driver" = "overlay"
eoj

       More details about the config file will be described in later as part of Q&A.

    3. Apply the config file. This should be done only once per project and 
       on a host or VM having at least 4 cores and 32Gb RAM.
       The concept of the project will be described in later steps. The lines below create
       a project, myproject, using a config file, myconfig_file, created
       in step 2.
            export SPILLBOX_VPN=1   # Not a Cloud flow but a Datacenter flow
            export SPILLBOX_SERVER=10.10.10.10 # IP address of web server at remote Datacenter
            ./splrun project apply  myproject ./myconfig_file 


    4. You are now ready to run your existing executable/script at a top/leaf
       level in the remote Datacenter. Above apply step will print the Docker command.
       You can submit this
       
       Syntax: ./splrun run  <existing exe/script> <arg> 

       Below runs the script /depot/openssh-7.9p1/build in the other Datacenter.

            export SPILLBOX_PROJECT=demo
            ./splrun run /depot/openssh-7.9p1/build

       Since you may have multiple projects, you need to set
       SPILLBOX_PROJECT to a project. Here SPILLBOX_PROJECT is set to
       myproject so that script /depot/openssh-7.9p1/build is run in
       project demo.

            ./splrun brun /depot/openssh-7.9p1/build
       runs in batch mode with return status of the job. run option may not
       return job exit status in all cases.


COMMON QUESTIONS AND THEIR ANSWERS
     
Q. 	How can I get help?
Ans. 	At any level, you can type  help to get a help message. For example:

           splrun help
           splrun run help

Q. 	What is the format and possible entries in the config file?
Ans. 	The format of the config file is "key" = "value", both in quotes.
        If the value is in the form of a list, then the list syntax is ["value1", "value2"] .
        Possible keys and their default values are:

    "mount_paths" = // default is derived automatically by Spillbox
        This gives the list of NFS export paths on local Datacenter to be made
        available in the remote Datacenter.
        Spillbox automatically adds by default the  list of top level 
        directories that are not part of Linux OS. User home directories
        are not added if they are inside a Linux OS distribution path like /home.
        If the Spillbox automatic NFS mount point finder is not suitable to
        you, please override it by providing the list. You can add a full path
        of subdirectories also. For example ["/depot", "/home/user"].

    "exclude_mount_paths" = [] // no default exclude 
        Provide the list of mount_paths to be excluded.

    "cloud_user" = // default is on-prem unix username
    "cloud_group" = // default is on-prem unix  group
    "cloud_uid" = // default is on-prem unix uid
    "cloud_gid" = // default is on-prem unix gid
    "cloud_shell" = // default is /bin/bash
    "exclude_filter_path" = "" // default no file/directories are excluded
        Excludes file/directory from getting uploaded to the remote Datacenter.

    "docker_base_image=   // default is centos:7
        You can put your own docker image on the docker repo server ( web server is
        docker repo server ) and select the image for the project using this variable.
     "docker_driver" =  // default is host
        You can choose overlay/macvlan as long as web server supports overlay/macvlan
        networking and sysadmin has configured the web server. 
     "disable_docker_image" = // default is false
        User can bring their own docker image and disable per project docker image.
        Env SPILLBOX_DISABLE_DOCKER_IMAGE set to any value will also disable spillbox image.
     "wan_timeout" = 901 // Change WAN timeout
     "nfs_model"= // default is empty 
       Based on the terraform/etc/spillbox_nfsModel_config file entry , the job manager is allowed to 
       define the addition options like cpu , memeory etc which will be appeneded with the job submit command
       example, "nfs_model"="small" will be mapped to the entry small="-n 2" 
       which will allocate 2 Slot(s) on Host(s) of job getting submitted byt the lsf job manager
     "hybrid_mount_paths" = "" // list of mount paths separated by spaces
                Enable fully synchronized data sync between on-premise and Cloud
                for set of mount paths hierarchically. This is equivalent to
                enabling -download-hier, missing-hier and sync-hier in the snapshot option
                for these directories. Environment variable SPILLBOX_HYBRID_MOUNT_PATHS
                also does the same. Special keyword "ALL" enables globally on all
                mount paths and "NONE" disables it if enabled globally by sysadmin.
    "hybrid_mount_paths_exclude" = "" // list of mount paths separated by spaces
                Diasble fully synchronized data sync between on-premise and Cloud
                for set of mount paths hierarchically. This will be applied after
                hybrid_mount_paths Environment variable SPILLBOX_HYBRID_MOUNT_PATHS_EXCLUDE
                also does the same. Special keyword "PATH" excludes all paths in $PATH.

   Spillbox provides wrapper script for LSF/UGE/RTDA/... in lsf/uge/rtda/... sub-directory
   so that users do not need to change their job submission command line embedded 
   in their script/flow. Just picking up Spillbox job submission command by modifying $PATH
   will do the trick. There are few environment variables used in these wrappers:
        SPLAPP_SKIP_DOCKERIZED_CLUSTER
		Let Spillbox spawn container, instead of UGE/LSF/RTDA/...
		even if the cluster has container support.
        SPLAPP_BATCH_RUN_OPTION
		Add extra options to the command allowed in the batch mode.
	SPLAPP_IMAGE
		To utilize user own image for submitting a job in SGE.
	SPLAPP_INTERACTIVE_RUN
		To submit a Job in an interactive mode
	SPLAPP_SKIP_OUTPUT_REDIRECTION
		It ignores the -o and -e option. The output will be 
		on the host instead of the container, user can view
		it by executing the command "splrun run -remote cat ~/<file>".
	SPLAPP_NCPATH
		To use nc which is installed in a different path. 
		Set this variable.
	SPLAPP_DEBUG
		It will enable more debugging messages to be printed.

     

Q. 	What are other important commands to know?
Ans. 	Other important commands are:

    Syntax: splrun project destroy|destroy_all|snapshot|backup|restore <projectname> 

    destroy           -- To destroy compute machine 
    destroy_all       -- To destroy compute machine and file cache   
    snapshot          -- To create snapshot at current time 
    backup            -- To backup Cloud data // not applicable and disabled for Datacenter 
    restore           -- To restore Cloud  // not applicable and disabled for Datacenter 

    Once you are done running all your jobs, please run destroy_all to delete
    all resources in the other Datacenter. If your data upload is large and you
    want to come back again to use the remote Datacenter in the near future,
    please run destroy only. This keeps the file server running and the cache intact.
    This does, however, cost resources.

    If you make changes on premise after creating a project and accessing data on the
    remote Datacenter, these changes are not updated in the remote Datacenter.
    To make your changes visible in the remote Datacenter, you have to run snapshot.
    snapshot creates a file instance in the remote data center based on time of the 
    snapshot command.  Each time you create changes on local data center and need 
    the changes to be available at the remote data center, you need to update the
    file system instance time by running the snapshot command.

    Syntax: splrun show <projectname> [<variable>]

    In the case that you need to login to a remote Datacenter compute machine,
    you will need to know information such as the ssh key and the IP address of
    the machine.  These and other important variables can be printed with their
    values using the show command.

    Syntax: splrun precache <dir1> <dir2>...<dirN>
            splrun download <src_file> <dst_dir>
            splrun download_all [<dst_dir>]

    The requirements of your job may be such that it will sync TBs of data to
    the remote Datacenter before the job starts running. Although Spillbox will do this
    automatically for you, you may not want to waste compute resources while
    data is being uploaded automatically by Spillbox. You may want to help
    the system by uploading some of the larger data yourself before running
    the job. splrun upload is a very useful tool for this. You can sync multiple
    directories at a time. Spillbox does not sync data from the remote Datacenter 
    to local automatically. You will need to use the above download command to 
    sync the selected result back to the local Datacenter.
    Upload/download can exclude/include selected files/directories and 
    detailed usage can be available with command:
         splrun precache/download help

    There are few important features for data synchronization between on-premise
    and remote datacenter which can be activated through snapshot features:

    splrun  project snapshot  help|<project_name> [-missing|-missing-hier|-missing-remove|-missing-hier-remove] [-missing-pattern-include <pattern>] [-missing-pattern-exclude <pattern>] [-sync|-sync-hier|-sync-remove|-sync-hier-remove] [-download|-download-hier|-download-remove|-download-hier-remove] [-download-pattern-include <pattern>] [-download-pattern-exclude <pattern>] [-reset-log <tag>] [-reset-appendlog <tag>] [<file>/<dir>]
        <file>/<dir>              -- Optionally snapshot a file/dir only
        -missing                  -- Search for missing files/dirs at current level in on-prem
        -missing-hier             -- Search for missing files/dirs at current and below level in on-prem
        -missing-remove           -- Remove search for missing files/dirs at current level in on-prem
        -missing-hier-remove      -- Remove search for missing files/dirs at current and below level in on-prem
        -missing-pattern-include  -- Pattern to include in handling missing files/dirs
        -missing-pattern-exclude  -- Pattern to include in handling missing files/dirs
        -sync                     -- Sync files/dirs at current level with on-prem all the time
        -sync-hier                -- Sync files/dirs at current level and below with on-prem all the time 
        -sync-remove              -- Remove -sync
        -sync-hier-remove         -- Remove -sync-hier
        -download                 -- Download new files/dirs at current level automatically at close
        -download-hier            -- Download new files/dirs at current and below level automatically at close
        -download-remove          -- Remove download new files/dirs at current level automatically at close
        -download-hier-remove     -- Remove download new files/dirs at current and below level automatically at close
        -download-pattern-include -- Pattern to include in handling download of files/dirs
        -download-pattern-exclude -- Pattern to exclude in handling download of files/dirs
        -reset-log                -- Create full snapshot and recreate log file for tracking data transfer
        -reset-appendlog          -- Create full snapshot and recreate log file in append mode for tracking data transfer 


        With the activation of -missing*, -sync* and -download* features in the beginning of the project, 
        Spillbox's cloud bursting can be made to look like two way fully synchronized hybrid cloud.
    
Q. 	How can I ssh out to spillbox worker machines when the ssh port is blocked at corporate level?
Ans. 	You can use spillbox data pipe to ssh out.
     
        After project is created, you need to run the following command to ssh:

           splrun ssh <worker_hostname> 

        where <worker_hostname> is the spillbox machines you want to ssh to.
        No need to provide ssh private key. It is automatically taken care by splrun.

        Implementation internally uses a local port, 2222 and above. You can control
        the port with an environment variable SPILLBOX_SSH_PORT by giving the starting port number.

Q. 	How can I pass environment variable to "splrun run"
Ans. 	Please see below usage:
           splrun run [-env <var>=<value>] help|<command>  <argument>

Q. 	splrun run is not sourcing my .cshrc ( .bashrc ) in the container.
Ans:  	Please do "splrun run /bin/csh -c \"<command> <args> ...\"  (or similar for bash) 
        in places where you want to source .cshrc in the container. It will be usually needed
        at the top level script, and not when calling splrun from the middle of the script.

Q. 	How can I use Spillbox for test case archival and replication at a later time or at a third party?
Ans. 	If user runs into complex bug and need to send the test case to the third party
        for debugging by replicating the test case at third party then please follow the
        below steps:

           splrun run -archive <dir1> <cmd> <args> // archive in dir1 at the end of running
        Inspect dir1 for sensitive data as suggested by splrun at runtime and ship it to the third party.

        Third party can restore  using Spillbox during project apply phase:
           splrun project apply <projectname> -load <dir2>  // dir2 is directory where shipped
                                                         // data were untar'ed by the vendor.

        Then run as
           splrun run -rerun   // if no arguments are given, it runs the command used during archiving

Q. How do I use multiple container images?
Ans:    Users can select container image using short name ( or alias) from a set of
        containers available on the server:  

            splrun run -image <short_name> <cmd> <args>

Q. 	How can I get an interactive terminal
Ans:  Please run "splrun run $SHELL -i" to get an interactive terminal.	
        Example : splrun run /bin/bash -i

    Please type control-D to end the session. Typing "exit" in the shell will also end the session. 

Q. 	How can I change the location of Spillbox config file $HOME/.config/spillbox ?
Ans: 	Please set environment variable SPILLBOX_CONFIGDIR to a location other than $HOME.

Q.      How can I run multiple splrun in single command line?
Ans:    Use multi_splrun splrun run <cmd1> ++ splrun run <cmd2> . Using "++" user can concatenate any number of splrun commands to the need.
        To run multiple splrun commands in background use only brun
        multi_splrun splrun brun <cmd1> ++ splrun brun <cmd2> &

        Example : multi_splrun splrun run hostname ++ splrun run whoami
                  multi_splrun  splrun brun hostname ++ splrun brun whoami &

Q.  How do I enable support for Spot Instances?
Ans:    Support for Spot Instances are done on the server by your sysadmin following the
        Spillbox documentation in INSTALLATION.txt. In case support for the spot instances
        are enabled, you can choose the job to be automatically restarted on spot preemption
        ( default ) suitable for short jobs, or enable checkpoint at regular intervals.
        On preemption, if checkpoint exits, job will start from the last checkpoint.
        Following environment variables are used to control Checkpoint:
            SPLAPP_CR_ENABLE     // not set for default
                                 set it to 1 to enable checkpoint
            SPLAPP_CR_INTERVAL  //  default is 60 
                                set it to a number (in minutes) to checkpoint at that regular internal.

Q.  How do I enable spillbox boostmode?
Ans:    export SPILLBOX_BOOSTMODE=1
        export SPILLBOX_BOOST_MOUNT_PATHS="DIR1 DIR2"     # Directory to be in boost mode

Q. How can I login to the container where my job (job_id) is running??
Ans:    - For LSF jobs: Use the Spillbox bsub wrapper
          bsub -jobattach <job id>

        - For UGE jobs: Use the Spillbox qsub wrapper
          qsub -jobattach <jobid>

        - For slurm jobs: Use the Spillbox sbatch wrapper
          sbatch -jobattach <jobid>

        - For nc jobs: Use the Spillbox nc wrapper
          nc -jobattach <job id>
Note: This will be useful for debugging purposes.

Q.  How can I map a remote queue name to an on-premises queue name?
    - To map your on-premises queue name with a remote queue name, create a setup.sh script in the slurm directory on your client side.
    - Provide your on-premises queue name in the setup.sh file in the following format:

        queue_map={"queue_name1"="compute-queue","queue_name2"="nfs-queue","queue_name3"="spot-queue"}

        - "queue_name1" , "queue_name2" and "queue_name3" are your on-prem queue names (Replace them with your actual queue names).
        - "compute-queue" , "nfs-queue" and "spot-queue" are remote queue names.
    NOTE: The format should be the same as provided, without any extra spaces or characters.

Q. How to Automatically Destroy an Idle Project ?
Ans: Set SPILLBOX_IDLE_TIMEOUT to the desired number of hours before creating a project. It specifies how long a project can remain idle before being automatically destroyed.

Q.  How can I get help from Spillbox team?
Ans:    Please email support@spillbox.io or call 1-833-SPILLBX(774-5529).
