CRAM → FASTQ conversion: efficiency & cost comparison

All 4 approaches convert the same CRAM inputs to paired FASTQ (12 array tasks each). Metrics from the Slurm accounting database; cost from the SIT job-cost API.

Summary

What the numbers show

• Approach A requests 8 CPUs but reaches only 12% CPU efficiency — effectively 1.0 core(s). It runs single-threaded, so the other allocated cores sit idle while it also pins ~90 GB RAM for the whole run.
• Approach B reaches 68% efficiency across 6 CPUs (~4.1 cores) with 6 GB, finishing in 00:05:28 (mean) at a cost of 0.0708.
• Approach C reaches 93% efficiency across 3 CPUs (~2.8 cores) with 3 GB, finishing in 00:07:40 (mean) at a cost of 0.0496.
• Approach D reaches 89% efficiency across 4 CPUs (~3.5 cores) with 4 GB, finishing in 00:05:58 (mean) at a cost of 0.0515.
• The leanest approach, Approach C, costs 100x less than Approach A (0.0496 vs 4.96) for identical output, using 33x less memory and 18x less wall time. Because the gap is per-sample, it grows in absolute terms with bigger datasets.

Cost vs speed

• Approach B has the fastest per-sample turnaround (00:05:28) — prefer it when single-sample latency matters (e.g. one urgent sample).
• Approach C is the cheapest per sample (0.0041) and packs the most work into a core budget (261 samples per 100 core-hours) — best for large batches where total cost dominates.
• Approach D is the best all-rounder: within ~4% of Approach C's cost but ~22% faster (00:05:58 vs 00:07:40).
• See Notes for how each approach threads samtools fastq.

Notes

• Approach A runs samtools fastq without -@ (single-threaded), so the 8 requested cores cannot be used — hence ~1/8 CPU efficiency.
• Approach B threads samtools fastq with -@ (SLURM_CPUS_PER_TASK - 1): one worker per allocated core, minus one for overhead.
• Approach C overprovisions samtools fastq threads with -@ (SLURM_CPUS_PER_TASK * 2 - 1), about twice the allocated cores, so the CPUs stay busy through I/O waits — this is why efficiency stays high at a lower core count.
• Approach D overprovisions samtools fastq threads with -@ (SLURM_CPUS_PER_TASK * 2 - 1), about twice the allocated cores, so the CPUs stay busy through I/O waits — this is why efficiency stays high at a lower core count.

Scaling projection

Per-sample cost and wasted compute are essentially fixed (each CRAM is similar), so the differences scale linearly with cohort size.

• A 1,000-sample cohort would cost ~413 with Approach A versus ~4 with the leanest Approach C (~100x more, ~409 wasted), while Approach A alone burns ~15,823 CPU-hours on idle cores that never do any work.
• A 10,000-sample cohort would cost ~4,134 with Approach A versus ~41 with the leanest Approach C (~100x more, ~4,093 wasted), while Approach A alone burns ~158,232 CPU-hours on idle cores that never do any work.

Approach A - 8 CPU / 100 GB requested — per task

Approach B - 6 CPU / 6 GB requested — per task

Approach C - 3 CPU / 3 GB requested — per task

Approach D - 4 CPU / 4 GB requested — per task