Tracking InVision videos

Created

Feb 23, 2024 10:26 PM

Status

Complete

Glossary

💡

BOSE - The UWEC high-performance computer that we use to track InVision videos OnDemand - a browser-based tool for interacting with BOSE Slurm - popular job manager for high-performance computing GitHub - a website that stores repositories of version-controlled code git - the software that allows version control of code NAS - network-attached storage, where all our InVision videos are stored long term Snakemake - software for managing multi-step analytical workflows Anaconda/conda - software for managing computing environments

TL;DR

💡

Before tracking:

Make sure you have the most recent commit of the invision-tools repository in your BOSE home directory.

# if you do have it but need to update
cd ~/GitHub/invision-tools && git pull

Make sure the experiments to be analyzed have been uploaded from the InVision computer to the Wheeler Lab NAS.

To track:

While logged into the BOSE command line, transfer the videos from the NAS to BOSE:

In the command below, replace wheelenj with your username, 20240711-a02-CNN_20240711_140720.24568709 with the experiment you want to analyze, and miracidia with the proper BOSE directory.
Use your your UWEC password.

rsync -Ppvrz wheelenj@wheeler.nas.uwec.edu:/volume1/WheelerLab_data/InVision/20240711-a02-CNN_20240711_140720.24568709 /data/groups/wheelenj/miracidia

Navigate to the organisms directory:

cd /data/groups/wheelenj/miracidia/

Activate the invision-env and submit the Snakemake script, which will run tracking on all the experiments from the provided date.

module load python-libs
conda activate invision-env
./run_snakemake.sh 20240711

Preparing your BOSE account

💡

It is assumed that the user has an active BOSE account, has received training from Tyler Bauer from LTS and Dr. Wheeler, and has been added to the Wheeler Lab group, which will allow the user to access the data at /data/groups/wheelenj/ . The following steps only need to be followed the first time a user performs an analysis. It is also assumed that you have been added as a member of the wheelerlab-uwec GitHub organization, which will give you access to the invision-tools repository, and that you’ve generated an SSH key on BOSE and added it to your GitHub account. In all code blocks below, commands that should be run in your terminal follow a $ sign - copy and paste the command (not including the $) into your terminal and press enter. Lines representing example output following a command do not start with a $.

Clone the invision-tools GitHub repository into your home folder.

Login to OnDemand in the browser and click the “BOSE Cluster Shell Access” card
The repository should be cloned into your home directory (/data/users/{username}):

$ cd # navigate to your home directory
$ mkdir GitHub # make a new GitHub directory (if you haven't already)
$ cd GitHub # navigate into the new directory
$ git clone https://github.com/wheelerlab-uwec/invision-tools.git # clone the repo

If you cloned the repo previously but haven’t updated in awhile, pull the newest version:

$ cd ~/GitHub/invision-tools # navigate to the repo
$ git pull # get the latest version

Create the conda environment required for invision-tools. This environment will contain all the software and Python libraries required for running the tracking scripts.

$ cd # navigate to your home dir
$ module load python-libs # load the required modules from the BOSE shared software library
$ mamba env create -f ~/GitHub/invision-tools/environment.yml # create the environment

Activate the environment by running conda activate invision-env.

Perform tracking

Transfer files to be analyzed

Login to OnDemand in the browser and click the “BOSE Cluster Shell Access” card.
Transfer videos from the Wheeler Lab’s NAS (https://wheeler.nas.uwec.edu:5501) to BOSE Note: When accessing the server via a browser, you may get a warning about the connection not being private. Click Advanced and Proceed.

On the server, all videos should be stored in the shared folder found at /volume1/WheelerLab_data/InVision/ . Here’s what it looks like using the File Station application (click the above link to access the sign-in page):

On BOSE, all videos should be stored in the Wheeler Lab’s group folder found at /data/groups/wheelenj. Mosquito videos are in the mosquitoes/ subfolder, miracidia videos are in the miracidia/ subfolder, and planaria videos are in the planaria/subfolder (here’s what it looks like when using the OnDemand file system):

Use rsync to transfer the files, for example. The following shows the command run (on the first line) and the resulting output:

$ rsync -Ppvrz wheelenj@wheeler.nas.uwec.edu:/volume1/WheelerLab_data/InVision/20240301-a01-MRB_20240301_144112.24568709 /data/groups/wheelenj/mosquitoes

Could not chdir to home directory /var/services/homes/wheelenj: No such file or directory
receiving incremental file list
20240301-a01-MRB_20240301_144112.24568709/
20240301-a01-MRB_20240301_144112.24568709/000000.extra_data.json
        274,356 100%  261.65MB/s    0:00:00 (xfr#1, to-chk=4/6)
20240301-a01-MRB_20240301_144112.24568709/000000.hd5
  1,118,607,211 100%   85.02MB/s    0:00:12 (xfr#2, to-chk=3/6)
20240301-a01-MRB_20240301_144112.24568709/000000.mp4
  1,615,335,187 100%   24.12MB/s    0:01:03 (xfr#3, to-chk=2/6)
20240301-a01-MRB_20240301_144112.24568709/000000.npz
        274,972 100%  262.23MB/s    0:00:00 (xfr#4, to-chk=1/6)
20240301-a01-MRB_20240301_144112.24568709/metadata.yaml
            875 100%  213.62kB/s    0:00:00 (xfr#5, to-chk=0/6)

sent 123 bytes  received 1,647,720,372 bytes  16,728,126.85 bytes/sec
total size is 2,734,492,601  speedup is 1.66

General rsync syntax: rsync username@server:/path/to/source /path/to/destination
Explanation of options:

-P - show progress
-p - preserve permissions
-v - be verbose
-r - sync recursively (i.e., copy everything in the source directory)
-z - compress while transferring

Prepare and run the Snakemake script

💡

Each organism directory (i.e., miracidia , mosquitoes , or planaria) has a Snakemake runner in it - run_snakemake.sh. This runner will run all the commands necessary to analyze all experiments recorded on a given date. All you have to do is run ./run_snakemake.sh {date} from the organism directory, where {date} is the experiment date you want to track. Follow the steps below for more details.

Navigate to the organism directory: cd /data/groups/wheelenj/mosquitoes/ . This step is crucial; the job will not run properly if you are not in the correct directory at the time of submission.
Run the Snakemake runner to start the job(s): ./run_snakemake.sh {date}. Snakemake will automatically submit Slurm jobs for you, and it will submit them in order (i.e., it won’t submit the linking job until tracking has been completed for all videos).
Verify the job(s) have been started by running the sacct command:

$ sacct
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
71371_0        tracking highmemory wheelenj_+         64    RUNNING      0:0
71371_0.bat+      batch            wheelenj_+         64    RUNNING      0:0
71371_0.ext+     extern            wheelenj_+         64    RUNNING      0:0

You should see a JobID for each job that you are running.