SLEAP tracking InVision videos

Created

Jun 17, 2025 1:19 PM

Status

In progress

Testing

Desktop session (139020):

Jobeff: metrics.hpc.uwec.edu

Commands:

# from BOSE
rsync -Ppvrz wheelenj@wheeler.nas.uwec.edu:/volume1/WheelerLab_data/InVision/20250616* .

# from desktop session terminal
module load sleap/1.4.1
sleap-label

File > Open Project… /data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp

File > Add Videos… /data/groups/wheelenj/mosquitoes/20250616a01controlard_20250616_140037.24568709/000000.mp4

Predict > Run Inference…

Save configuration files… `/data/groups/wheelenj/mosquitoes/models/

inference-script.sh

#!/bin/bash
sleap-track /data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp --video.index 26 --frames 0,-2331 -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none -o L1_L2_L4.labels.v001.slp.predictions.slp --verbosity json --no-empty-frames

jobs.yaml

inference:
- data_path: L1_L2_L4.labels.v001.slp
  models:
  - /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json
  only_suggested_frames: false
  output_path: L1_L2_L4.labels.v001.slp.predictions.slp
  tracking:
    tracking.tracker: none
  type: video
training: []

train-script.sh

#!/bin/bash

Inference processed at ~22 FPS

Terminal STDOUT:

Using already trained model for single_instance: /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json
Command line call:
sleap-track /data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp --video.index 26 --frames 0,-2331 -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none --controller_port 9000 --publish_port 9001 -o /data/groups/wheelenj/mosquitoes/predictions/L1_L2_L4.labels.v001.slp.250617_083316.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2025-06-17 08:33:21.720874
Args:
{
\u2502   'data_path': '/data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp',
\u2502   'models': [
\u2502   \u2502   '/data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json'
\u2502   ],
\u2502   'frames': '0,-2331',
\u2502   'only_labeled_frames': False,
\u2502   'only_suggested_frames': False,
\u2502   'output': '/data/groups/wheelenj/mosquitoes/predictions/L1_L2_L4.labels.v001.slp.250617_083316.predictions.slp',
\u2502   'no_empty_frames': True,
\u2502   'verbosity': 'json',
\u2502   'video.dataset': None,
\u2502   'video.input_format': 'channels_last',
\u2502   'video.index': '26',
\u2502   'cpu': False,
\u2502   'first_gpu': False,
\u2502   'last_gpu': False,
\u2502   'gpu': 'auto',
\u2502   'max_edge_length_ratio': 0.25,
\u2502   'dist_penalty_weight': 1.0,
\u2502   'batch_size': 4,
\u2502   'open_in_gui': False,
\u2502   'peak_threshold': 0.2,
2025-06-17 08:33:23.158584: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
\u2502   'max_instances': None,
\u2502   'tracking.tracker': 'none',
\u2502   'tracking.max_tracking': None,
\u2502   'tracking.max_tracks': None,
\u2502   'tracking.target_instance_count': None,
\u2502   'tracking.pre_cull_to_target': None,
\u2502   'tracking.pre_cull_iou_threshold': None,
\u2502   'tracking.post_connect_single_breaks': None,
\u2502   'tracking.clean_instance_count': None,
\u2502   'tracking.clean_iou_threshold': None,
2025-06-17 08:33:23.698307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30978 MB memory:  -> device: 0, name: Tesla V100S-PCIE-32GB, pci bus id: 0000:27:00.0, compute capability: 7.0
\u2502   'tracking.similarity': None,
\u2502   'tracking.match': None,
\u2502   'tracking.robust': None,
\u2502   'tracking.track_window': None,
\u2502   'tracking.min_new_track_points': None,
\u2502   'tracking.min_match_points': None,
\u2502   'tracking.img_scale': None,
\u2502   'tracking.of_window_size': None,
\u2502   'tracking.of_max_levels': None,
\u2502   'tracking.save_shifted_instances': None,
\u2502   'tracking.kf_node_indices': None,
\u2502   'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Auto-selected GPU 0 with 32491 MiB of free memory.
2025-06-17 08:33:30.169369: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
Versions:
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-4.18.0-348.el8.x86_64-x86_64-with-centos-8.5.2111

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
       Initialized: False
     Memory growth: True

Finished inference at: 2025-06-17 08:35:16.655386
Total runtime: 114.9345314502716 secs
Predicted frames: 2332/2332
Provenance:
{
\u2502   'model_paths': [
\u2502   \u2502   '/data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json'
\u2502   ],
\u2502   'predictor': 'SingleInstancePredictor',
\u2502   'sleap_version': '1.4.1a2',
\u2502   'platform': 'Linux-4.18.0-348.el8.x86_64-x86_64-with-centos-8.5.2111',
\u2502   'command': '/data/software/python-libs/3.0.0/envs/sleap-1.4.1/bin/sleap-track /data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp --video.index 26 --frames 0,-2331 -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none --controller_port 9000 --publish_port 9001 -o /data/groups/wheelenj/mosquitoes/predictions/L1_L2_L4.labels.v001.slp.250617_083316.predictions.slp --verbosity json --no-empty-frames',
\u2502   'data_path': '/data/groups/wheelenj/mosquitoes/L1_L2_L4.labels.v001.slp',
\u2502   'output_path': '/data/groups/wheelenj/mosquitoes/predictions/L1_L2_L4.labels.v001.slp.250617_083316.predictions.slp',
\u2502   'total_elapsed': 114.9345314502716,
\u2502   'start_timestamp': '2025-06-17 08:33:21.720874',
\u2502   'finish_timestamp': '2025-06-17 08:35:16.655386'
}

Saved output: /data/groups/wheelenj/mosquitoes/predictions/L1_L2_L4.labels.v001.slp.250617_083316.predictions.slp




Process return code: 0

In general very good performance with one consistent outlier/error:

Resource usage:

Now to try running from command line instead of GUI:

#!/bin/bash
sleap-track {input_data.mp4} -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none --verbosity json --no-empty-frames

sleap-track /data/groups/wheelenj/mosquitoes/20250616a01cueard_20250616_140456.24568709/000000.mp4 -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none --verbosity json --no-empty-frames

Processed at ~24 FPS

Finished inference at: 2025-06-17 08:49:46.518150
Total runtime: 334.90364265441895 secs
Predicted frames: 7630/7903
Provenance:
{
\u2502   'model_paths': [
\u2502   \u2502   '/data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json'
\u2502   ],
\u2502   'predictor': 'SingleInstancePredictor',
\u2502   'sleap_version': '1.4.1a2',
\u2502   'platform': 'Linux-4.18.0-348.el8.x86_64-x86_64-with-centos-8.5.2111',
\u2502   'command': '/data/software/python-libs/3.0.0/envs/sleap-1.4.1/bin/sleap-track /data/groups/wheelenj/mosquitoes/20250616a01cueard_20250616_140456.24568709/000000.mp4 -m /data/groups/wheelenj/mosquitoes/models/250603_091255.single_instance.n=171/training_config.json --batch_size 4 --tracking.tracker none --verbosity json --no-empty-frames',
\u2502   'data_path': '/data/groups/wheelenj/mosquitoes/20250616a01cueard_20250616_140456.24568709/000000.mp4',
\u2502   'output_path': '/data/groups/wheelenj/mosquitoes/20250616a01cueard_20250616_140456.24568709/000000.mp4.predictions.slp',
\u2502   'total_elapsed': 334.90364265441895,
\u2502   'start_timestamp': '2025-06-17 08:44:11.614541',
\u2502   'finish_timestamp': '2025-06-17 08:49:46.518150'
}

Decent performance with some outliers

This produces: /data/groups/wheelenj/mosquitoes/20250616a01cueard_20250616_140456.24568709/000000.mp4.predictions.slp

To get the CSV, use sleap-convert --format analysis.csv -o 000000.predictions.analysis.csv 000000.mp4.predictions.slp

>head 000000.predictions.analysis.csv 
track,frame_idx,instance.score,head.x,head.y,head.score,thorax.x,thorax.y,thorax.score,abdomen_middle.x,abdomen_middle.y,abdomen_middle.score,abdomen_tail.x,abdomen_tail.y,abdomen_tail.score
,123,0.3270174562931061,,,0.0,1447.37255859375,2351.1376953125,0.20003756880760193,,,0.0,,,0.0
,141,0.3234536945819855,,,0.0,1447.395263671875,2351.110107421875,0.20008735358715057,,,0.0,,,0.0
,152,0.3307470381259918,,,0.0,1447.404052734375,2351.099609375,0.20525634288787842,,,0.0,,,0.0
,153,0.32632553577423096,,,0.0,1447.4530029296875,2351.09228515625,0.20249448716640472,,,0.0,,,0.0
,169,0.3285437822341919,,,0.0,1447.3699951171875,2351.196533203125,0.20188017189502716,,,0.0,,,0.0
,178,0.3275035321712494,,,0.0,1447.35009765625,2351.18310546875,0.20127855241298676,,,0.0,,,0.0
,180,0.32908669114112854,,,0.0,1447.377197265625,2351.163330078125,0.20046330988407135,,,0.0,,,0.0
,185,0.33619993925094604,,,0.0,1447.4140625,2351.2451171875,0.20938457548618317,,,0.0,,,0.0
,189,0.3285723626613617,,,0.0,1447.4677734375,2351.217529296875,0.20510120689868927,,,0.0,,,0.0