biopipen

• deps: update pipen-runinfo dependency to version 0.8.1
• feat(scrna): add CellCellCommunication and CellCellCommunicationPlots
• fix(scrna.SeuratMap2Ref): fix report and add stats to report
• fix(utils.single_cell.R): fix categorical data when converting seurat to anndata format
• refactor(scrna.Seurat2AnnData): abstract seurat_to_anndata() for reuse
• enh(tcr.TCRClustering): make GIANA compatible with latest BioPython (v1.84)
• fix(tcr.TCRClstering): fix clusTCR error due to scipy update (v1.14)

What's Changed

• 0.32.0 by @pwwang in https://github.com/pwwang/biopipen/pull/154

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.7...0.32.0

• deps: bump pipen-args to 0.16
• chore: update pyright configuration to include biopipen/**/*.py
• feat(bam): add BamSubsetByBed process for subsetting bam file by regions in a bed file
• feat(bed): add BedtoolsMakeWindows process for generating windows from a BED file or genome size file

What's Changed

• 0.31.7 by @pwwang in https://github.com/pwwang/biopipen/pull/153

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.6...0.31.7

• deps: pin the channels of conda dependencies for tests
• feat(vcf): adopt truvari v4+ for related processes
• feat(regulatory): add VariantMotifPlot to plot motif and surrounding sequences with mutations
• refactor(regulatory.MotifAffinityTest): optimize code base
• ci: add verbosal output for tests

What's Changed

• 0.31.6 by @pwwang in https://github.com/pwwang/biopipen/pull/152

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.5...0.31.6

• deps: update pipen to version 0.15.3 and xqute to version 0.5.2
• feat(bam): add BamSampling process for sampling a fraction of reads from a bam file
• feat(protein): add the protein module and Prodigy and ProdigySummary to calculate the binding affinity of a complex structure
• ci: do not print verbose logs for tests
• chore(bam.BamMerge): use logger instead of print for logging

What's Changed

• 0.31.5 by @pwwang in https://github.com/pwwang/biopipen/pull/151

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.4...0.31.5

• deps: bump pipen-report to 0.20.1 (pipen to 0.15.2)
• fix(plot.VennDiagram): update default devpars and fix issues with computed data
• fix(scrna.SeuratMap2Ref): fix identifying the normalization method of reference

What's Changed

• 0.31.4 by @pwwang in https://github.com/pwwang/biopipen/pull/150

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.3...0.31.4

• test: fix test not failing when tests failed
• test: fix gene name conversion tests due to external API change
• fix(tcr.CDR3AAPhyschem): fix when chain is not available
• fix(tcr.TCRClustering): fix when chain is not available

What's Changed

• 0.31.3 by @pwwang in https://github.com/pwwang/biopipen/pull/149

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.2...0.31.3

• fix(tcr.CDR3AAPhyschem): use sequence from TRB chain only
• fix(tcr.TCRClustering): fix for multi-chain TCRs, use TRB only if on_multi is false

What's Changed

• 0.31.2 by @pwwang in https://github.com/pwwang/biopipen/pull/148

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.1...0.31.2

• enh(scrna.SeuratMap2Ref): check if reference has SCTModel if SCTransform'ed (likely prepared by old Seurat)

What's Changed

• 0.31.1 by @pwwang in https://github.com/pwwang/biopipen/pull/147

Full Changelog: https://github.com/pwwang/biopipen/compare/0.31.0...0.31.1

• deps: bump pipen to 0.15.0

What's Changed

• 0.31.0 by @pwwang in https://github.com/pwwang/biopipen/pull/146

Full Changelog: https://github.com/pwwang/biopipen/compare/0.30.0...0.31.0

scrna/tcr

• BREAKING(scrna): move clustree plots from SeuratClustering/SeuratSubClustering to SeuratClusterStats
• feat(scrna.CellTypeAnnotation): allow to merge/not to merge (envs.merge) the clusters with the same labels predicted
• feat(scrna.SeuratPreparing): add scDblFinder to detect doublets
• feat(scrna.SeuratMap2Ref): add envs.skip_if_normalized option to skip normalization if query is already normalized using the same method as the reference
• refactor(tcr.Immunarch): source the files for Immunarch scripts for better debugging
• refactor(scnra.SeuratClustering): refactor the script for better debugging
• refactor(scnra.SeuratPreparing): refactor the script for better debugging
• fix(scrna): fix resolution expansion for SeuratClustering and SeuratSubClustering
• fix(cellranger.CellRangerCount): fix falsy envs.create_bam not working for cellranger v7
• fix(scrna): Fix generating PrepSCTFindMarkers command when no previous commands present
• tests(scrna.ScFGSEA): fix unavailable urls to GMT files
• chore(scrna.SeuratMap2Ref): optimize memory usage
• chore(scrna.MetaMarkers): remove plugin_opts.poplog_max
• chore(tcr.CloneResidency): improve logging when handling subjects

other

• fix(stats.Mediation): fix when NAs in the data
• feat(plot): add Scatter for scatter plots
• tests: use single conda env for tests
• ci: fix CI due to conda env changes
• docs(web): update docs of envs.tool for Download/DownloadList
• feat(web): add GCloudStorageDownloadFile and GCloudStorageDownloadBucket to download files from GCP
• chore(regulatory.MotifAffinityTest): use template filter source_r to source R files
• tests(regulatory.MotifAffinityTest): rename regulation to regulatory
• chore: use template filter source_r to source R files
• fix(stats): handle case when p-value is 0 for MetaPvalue and MetaPvalue1

What's Changed

• 0.30.0 by @pwwang in https://github.com/pwwang/biopipen/pull/145

Full Changelog: https://github.com/pwwang/biopipen/compare/0.29.2...0.30.0

• chore(stats.Mediation): make better logging strategy for various number of cases
• chore(scrna.SeuratClusterStats): use ident label length to adjust default height for feature plots
• fix(scrna.MetaMarkers): fix seurat object not updated when expanding cases and run PrepSCTFindMarkers when necessary before calling meta-markers
• fix(scrna.MarkersFinder): fix fetching command when composing the PrepSCTFindMarkers command
• fix(scrna_metabolic_landscape): handle null values in for loop in MetabolicFeatures and MetabolicFeaturesIntraSubset for report generation

What's Changed

• 0.29.2 by @pwwang in https://github.com/pwwang/biopipen/pull/144

Full Changelog: https://github.com/pwwang/biopipen/compare/0.29.1...0.29.2

• BREAKING: rename namespace regulation to regulatory
• choir(plot.Manhattan): default envs.title to None (don't add title to the plot by default)
• enh(plot.Manhattan): give warnings instead of errors about zooming chromosomes not existing
• fix(plot.Manhattan): fix envs.ylabel not working
• feat(stats): add Mediation for mediation analysis
• feat(plot.QQPlot): add support for custom theoratical values
• tests(plot.QQPlot): add tests
• chore(snp.MatrixEQTL): allow pvalue cutoffs to be greater than 1 (but 1 will be used anyway)
• fix(snp.PlinkIBD): add --keep-allele-order to keep the allele order
• fix(delim.SampleInfo): fix numbers not split up when each is specified.
• enh(delim.SampleInfo): make sizes of pie charts proportional to number of samples when each is specified
• enh(scrna.MarkersFinder): run PrepSCTFindMarkers when necessary before calling FindMarkers
• feat(scrna.SeuratPreparing): add option to cache Seurat object at different steps
• feat(scrna.SeuratPreparing): allow doubletfinder to run with a different number of cores
• chore(scrna.SeuratClustering): record PrepSCTFindMarkers command in sobj@commands
• tests(scrna.SeuratClusterStats): use less stringent p-value cutoff for DEG/MarkersFinder
• tests(scrna.SeuratPreparing): add doubletfinder in tests

What's Changed

• Dev by @pwwang in https://github.com/pwwang/biopipen/pull/143

Full Changelog: https://github.com/pwwang/biopipen/compare/0.29.0...0.29.1

Depedencies

• deps: update pipen-filters to version 0.13
• deps: add meme to env_bio.yml for tests
• deps: bump pipen-board to 0.15.2
• ci: update GitHub Actions versions and dependencies

BREAKING

• BREAKING: merge namespace bcftools to vcf

New Features/Processes

• feat(snp): add PlinkFromVcf to convert VCF files to PLINK format
• feat(snp): add Plink2GTMat for converting PLINK files to genotype matrix
• feat(snp): add PlinkIBD analysis for identity by descent
• feat(snp): add PlinkHWE for Hardy-Weinberg Equilibrium test
• feat(snp): add PlinkHet for calculating sample heterozygosity
• feat(snp): add PlinkCallRate for calculating call rate of samples and variants
• feat(snp): add PlinkFilter process for filtering PLINK files
• feat(snp): add PlinkFreq process for calculating and filtering by allele frequencies
• feat(snp): add PlinkUpdateName proc to update variant names in PLINK files
• feat(gene): add GenePromoters for retrieving gene promoter regions and tests for it
• feat(bed): add BedtoolsIntersect process for finding the intersection of two BED files
• feat(regulation): add the namespace and MotifScan to use fimo for motif scanning
• feat(regulation): add MotifAffinityTest to test the affinity of motifs to the sequences and the affinity change due the mutations.
• feat(cnv.AneuploidyScore): allow BED and VCF files as in.segfile
• feat(cnv.TMADScore): allow BED and VCF files as in.segfile
• feat(cnvkit): allow user home directory (~) to be used in envs.ref in mulitple processes
• feat(plot): add ManhattanPlot to for support for plotting Manhattan plots
• feat(plot): add QQPlot proc for generating QQ-plot or PP-plot
• feat(vcf): add BcftoolsView process for viewing, subsetting, and filtering VCF files
• feat(vcf): add run_bcftools function for running bcftools with given arguments
• feat(vcf.BcftoolsSort): allow sorting contigs based on a chrom size file
• feat(vcf.BcftoolsFilter): allow indexing output file
• feat(vcf.BcftoolsAnnotate): allow providing annotation file as input file and allow indexing output file
• feat(stats): add MetaPvalue1 to combine pvalues from the same file
• feat(stats.MetaPvalue): add envs.keep_single flag to keep the single p-values
• feat(utils.misc.R): add run_command function for R
• feat(utils.reference): allow tabix_index to index infile directly
• feat(snp.MatrixEQTL): add envs.match_samples flag to subset snp, expr and cov data with common samples
• feat(snp.MatrixEQTL): fix cov data being wrongly transposed
• feat(snp.MatrixEQTL): allow extra columns when snp and gene position file is BED
• feat(tests): add lazy loading for reference data download and --local flag for downloading more references locally

Refactors/Improvements

• refactor(utils.gene): redesign gene_name_conversion functions for both python and R
• refactor(gene.GeneNameConversion): use R for implementation
• refactor(misc.Shell): save envs.cmd to a file and run it to fix the escaping issues of the command
• enh(snp.MatrixEQTL): use rtracklayer to read the position files

Minor

• choir(misc.Str2File): add default for in.name
• choir(gene.GeneNameConversion): allow envs.notfound to be ignore or skip when envs.output is append or replace
• choir(utils.misc.py): flush output for command printing in run_command function
• choir(utils.misc.py): print command with a new line for run_command function
• choir(snp.PlinkFromVcf): indicate sex is to be handled
• choir(cellranger_pipeline): remove unused import
• choir(bam.CNVpytor): implement cnvnator2vcf directly instead of using cnvnator2vcf.pl

Tests

• tests(utils.gene): update tests for gene_name_conversion
• tests(gene.GeneNameConversion): use right environment for tests
• tests(snp): add tests for plink related processes
• tests(snp): disable report generation for plink related tests
• tests(regulation): specify envs.genome for MotifAffinityTest
• tests: add python package mygene to conda environment biopipen-r
• tests: add tests for bcftools processes
• tests: do not download reference data for hg38 at CI
• tests: update bioconductor-ggmanh dependency to version 1.9.6
• tests: add bcftools to conda environment dependencies for tests

Docs

• docs(MatrixEQTL): fix choice items of envs.model
• docs(cellranger_pipeline): fix types of some items in docs, which should be 'list', instead of 'type=list'

Fixes

• fix(utils.reference): avoid index file to be created again for the same infile for tabix_index function
• fix(utils.reference): pass -f to bgzip or gunzip to overwrite the output if exists
• fix(vcf): fix passing vcffile as a string in fix_vcffile in VcfFix_utils.py
• fix(cnvkit_pipeline): fix sex in process channels
• fix(cnv.AneuploidyScoreSummary): fix when Sample column is already in metafile
• fix(cnv.TMADScoreSummary): fix when Sample column is already in metafile

Immunopipe-related

• fix(tcr.TCRClusterStats): fix envs.shared_clusters.heatmap_meta being broken by envs.shared_clusters.sample_order
• choir(scrna.SeuratMap2Ref): present better error message when envs.use or values of envs.MapQuery.refdata not in reference
• fix(scrna.MarkersFinder): run PrepSCTFindMarkers when needed
• choir(scrna.SeuratClustering): use FindClusters to run for multiple resolutions
• choir(scrna.SeuratSubClustering): use FindClusters to run for multiple resolutions
• feat(scrna.SeuratClustering): add clustree plot
• feat(scrna.SeuratSubClustering): add clustree plot
• tests(scrna.SeuratClusterStats): add assertion for clustree plot generation

What's Changed

• 0.29.0 by @pwwang in https://github.com/pwwang/biopipen/pull/142

Full Changelog: https://github.com/pwwang/biopipen/compare/0.28.1...0.29.0

• fix(scrna.CellsDistribution): fix devpars and hm_devpars not working
• tests: add r-sccatch to conda environment biopipen-r

• tests(scrna.CellTypeAnnotation): add tests for CellTypeAnnotation using scCATCH
• feat(cellranger_pipeline): add docker image building for cellranger pipeline
• chore(cellranger.CellRangerCount): add envs.create_bam to control whether create bams (supporting cellranger v8)
• chore(cellranger): add in.id to CellRangerCount and CellRangerVdj to specify sample ids
• chore(cellranger.CellRangerSummary): set the default value of report_paging to 8

What's Changed

• 0.28.0 by @pwwang in https://github.com/pwwang/biopipen/pull/140

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.9...0.28.0

• feat(tcr.TCRClusterStats): add sample_order to set sample order on heatmap and cluster_rows to switch row clustering on/off

What's Changed

• 0.27.9 by @pwwang in https://github.com/pwwang/biopipen/pull/139

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.8...0.27.9

• fix(scrna.SeuratClusterStats): fix selected columns not unique for stats
• feat(scrna.SeuratMap2Ref): allow non-SCTransform'ed reference
• feat(scrna.SeuratMap2Ref): allow splitting query object for mapping (pwwang/immunopipe#61)
• deps: update pipen-board to version 0.15.1

What's Changed

• 0.27.8 by @pwwang in https://github.com/pwwang/biopipen/pull/138

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.7...0.27.8

• fix(utils.gsea): fix gsea table not being printed for runFGSEA
• fix(core.filters): fix slugified pathway plot file name in report
• fix(scrna_metabolic_landscape): fix mutaters not working
• fix(scrna_metabolic_landscape.MetabolicFeatures/MetabolicFeaturesIntraSubset): skip groups with less than 5 cells in do_one_group and save a warning file under the case
• fix(utils.gsea): do not switch 1st and 2nd columns when 2nd column is numeric for localizeGmtfile
• chore: fix typo in class name ExprImpution to ExprImputation
• choir(tests): remove KEGG_metabolism.gmt for prep_reference.py
• tests(scrna_metabolic_landscape): fix tests

What's Changed

• 0.27.7 by @pwwang in https://github.com/pwwang/biopipen/pull/137

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.6...0.27.7

• fix(scrna_metabolic_landscape.MetabolicFeatures): fix return value of groups with less than 5 cells in do_one_group
• choir(utils.gsea): avoid printing NULL for runFGSEA
• tests: use the return value of pipen.run() to test the success

What's Changed

• 0.27.6 by @pwwang in https://github.com/pwwang/biopipen/pull/136

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.5...0.27.6

• fix(scrna.Subset10X/SeuratTo10X): correct the paths to the scripts
• feat(testing): allow to enable report for testing pipelines
• feat(scrna.SeuratPreparing): add envs.cell_qc_per_sample to filter cells before merging instead after
• test: add tests to scrna.SeuratTo10X and scrna.SeuratPreparing
• fix(scrna.SeuratClusterStats): fix color palette for ridge plots

What's Changed

• 0.27.5 by @pwwang in https://github.com/pwwang/biopipen/pull/135

Full Changelog: https://github.com/pwwang/biopipen/compare/0.27.4...0.27.5