Created
Feb 23, 2024 10:26 PM
Status
Complete
Category
BioinformaticsImaging
- TL;DR
- Preparing your BOSE account
- Perform tracking
- Transfer files to be analyzed
- Prepare and run the Snakemake script
TL;DR
Before tracking:
- Make sure you have the most recent commit of the
invision-tools
repository in your BOSE home directory. - Make sure the experiments to be analyzed have been uploaded from the InVision computer to the Wheeler Lab NAS.
- Make sure that snakemake is installed into your
invision-env
conda environment. - To install snakemake to the environment, run the following commands:
# if you don't have it already...
cd && mkdir GitHub && git clone git@github.com:wheelerlab-uwec/invision-tools.git
# if you do have it but need to update
cd ~/GitHub/invision-tools && git pull
conda activate invision-env
conda config --add channels bioconda
conda config --add channels conda-forge
# the above commands won't print any messages after completing
mamba install bioconda::snakemake=8.14.0
mamba install bioconda::snakemake-executor-plugin-cluster-generic
# type "Y" and hit Enter when asked to confirm changes
- While logged into the BOSE command line, transfer the videos from the NAS to BOSE:
- In the command below, replace
wheelenj
with your username,20240711-a02-CNN_20240711_140720.24568709
with the experiment you want to analyze, andmiracidia
with the proper BOSE directory. - Use your your UWEC password.
- Navigate to the organisms directory:
- Activate the
invision-env
and submit the Snakemake script, which will run tracking on all the experiments from the provided date.
rsync -Ppvrz wheelenj@wheeler.nas.uwec.edu:/volume1/WheelerLab_data/InVision/20240711-a02-CNN_20240711_140720.24568709 /data/groups/wheelenj/miracidia
cd /data/groups/wheelenj/miracidia/
conda activate invision-env
./run_snakemake.sh 20240711
Preparing your BOSE account
It is assumed that the user has an active BOSE account, has received training from Tyler Bauer from LTS and Dr. Wheeler, and has been added to the Wheeler Lab group, which will allow the user to access the data at
/data/groups/wheelenj/
. The following steps only need to be followed the first time a user performs an analysis.
In all code blocks below, commands that should be run in your terminal follow a $
sign - copy and paste the command (not including the $
) into your terminal and press enter. Lines representing example output following a command do not start with a $
.- Clone the
invision-tools
GitHub repository into your home folder. - Login to OnDemand in the browser and click the “BOSE Cluster Shell Access” card
- The repository should be cloned into your home directory (
/data/users/{username}
): - If you cloned the repo previously but haven’t updated in awhile, pull the newest version:
- Create the conda environment required for
invision-tools
. This environment will contain all the software and Python libraries required for running the tracking scripts. - This environment can be activated at anytime with the command:
conda activate invision-env
.
$ cd # navigate to your home directory
$ mkdir GitHub # make a new GitHub directory (if you haven't already)
$ cd GitHub # navigate into the new directory
$ git clone https://github.com/wheelerlab-uwec/invision-tools.git # clone the repo
$ cd ~/GitHub/invision-tools # navigate to the repo
$ git pull # get the latest version
$ cd # navigate to your home dir
$ module load python-libs # load the required modules from the BOSE shared software library
$ conda init bash # initialize conda
$ mamba env create -f ~/GitHub/invision-tools/environment.yml # create the environment
Perform tracking
Transfer files to be analyzed
- Login to OnDemand in the browser and click the “BOSE Cluster Shell Access” card.
- Transfer videos from the Wheeler Lab’s server (https://wheeler.nas.uwec.edu:5501) to BOSE Note: When accessing the server via a browser, you may get a warning about the connection not being private. Click Advanced and Proceed.
- On the server, all videos should be stored in the shared folder found at
/volume1/WheelerLab_data/InVision/
. Here’s what it looks like using the File Station application (click the above link to access the sign-in page): - On BOSE, all videos should be stored in the Wheeler Lab’s group folder found at
/data/groups/wheelenj
. Mosquito videos are in themosquitoes/
subfolder, miracidia videos are in themiracidia/
subfolder, and planaria videos are in theplanaria/
subfolder (here’s what it looks like when using the OnDemand file system): - Use rsync to transfer the files, for example. The following shows the command run (on the first line) and the resulting output:
- General rsync syntax:
rsync username@server:/path/to/source /path/to/destination
- Explanation of options:
-P
- show progress-p
- preserve permissions-v
- be verbose-r
- sync recursively (i.e., copy everything in the source directory)-z
- compress while transferring
$ rsync -Ppvrz wheelenj@wheeler.nas.uwec.edu:/volume1/WheelerLab_data/InVision/20240301-a01-MRB_20240301_144112.24568709 /data/groups/wheelenj/mosquitoes
Could not chdir to home directory /var/services/homes/wheelenj: No such file or directory
receiving incremental file list
20240301-a01-MRB_20240301_144112.24568709/
20240301-a01-MRB_20240301_144112.24568709/000000.extra_data.json
274,356 100% 261.65MB/s 0:00:00 (xfr#1, to-chk=4/6)
20240301-a01-MRB_20240301_144112.24568709/000000.hd5
1,118,607,211 100% 85.02MB/s 0:00:12 (xfr#2, to-chk=3/6)
20240301-a01-MRB_20240301_144112.24568709/000000.mp4
1,615,335,187 100% 24.12MB/s 0:01:03 (xfr#3, to-chk=2/6)
20240301-a01-MRB_20240301_144112.24568709/000000.npz
274,972 100% 262.23MB/s 0:00:00 (xfr#4, to-chk=1/6)
20240301-a01-MRB_20240301_144112.24568709/metadata.yaml
875 100% 213.62kB/s 0:00:00 (xfr#5, to-chk=0/6)
sent 123 bytes received 1,647,720,372 bytes 16,728,126.85 bytes/sec
total size is 2,734,492,601 speedup is 1.66
Prepare and run the Snakemake script
Each organism directory (i.e., miracidia
, mosquitoes
, or planaria
) has a Snakemake runner in it - run_snakemake.sh
. This runner will run all the command necessary to analyze all experiments recorded on a given date. Al you have to do is run ./run_snakemake.sh {date}
from the organisms directory, where {date}
is the experiment date you want to track. Follow the steps below for more details.
- Navigate to the organism directory:
cd /data/groups/wheelenj/mosquitoes/
. This step is crucial; the job will not run properly if you are not in the correct directory at the time of submission. - Run the Snakemake runner to start the job(s):
./run_snakemake.sh {date}
. Snakemake will automatically submit Slurm jobs for you, and it will submit them in order (i.e., it won’t submit the linking job until tracking has been completed for all videos). - Verify the job(s) have been started by running the
sacct
command:
$ sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
71371_0 tracking highmemory wheelenj_+ 64 RUNNING 0:0
71371_0.bat+ batch wheelenj_+ 64 RUNNING 0:0
71371_0.ext+ extern wheelenj_+ 64 RUNNING 0:0
You should see a JobID for each job that you are running.