vg

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.21.0

Buildable Source Tarball: vg-v1.21.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Changes to how vg paths can be used to remove paths from a graph.
• Use of a new toil-vg version in testing.
• Changes to minimizer seed handling algorithms in vg gaffe.
• Some paired-end clustering logic to be used in vg gaffe.
• Improvements to the build system around the libsdsl build.
• No-hit read counting and better S3 support in Giraffe Wrangler script.
• A new subcommand, vg depth.
• Refactoring and race condition removal in vg pack.
• Probabilistic calling support.
• Improvements to snarl traversal enumeration.
• Improved ability of the build system to handle switching compilers.
• Bugfixes to the vg RDF ontology.
• vg augment now cuts off softclips by default.
• vg explode functionality now subsumed by vg chunk.
• Internal VGSet object now supports HandleGraphs.
• Some manpages available via make man.
• Better handling of reverse-strand cycles in vg call.
• Better gtf/gff parsing in vg rna.
• Snarl computation now parallel.
• Subpaths output by vg chunk -p now contain offsets.
• Fix a race condition when reading from the Paths object.
• Ability to chunk non-indexed GAM files by graph component.
• Bugfixes to kmer enumeration.
• Deprecated some vg mod flags, with warning messages.
• vg find can produce more information about kmers.
• vg construct should no longer drop the end breakpoints of deletions.

New and Updated Submodules

The BBHash, libbdsg, libhandlegraph, pinchesAndCacti, and xg submodules have been updated.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.20.0

Buildable Source Tarball: vg-v1.20.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Faster and more efficient minimizer finding and clustering in vg gaffe
• Memory usage optimizations in vg rna
• vg call bugfixes for cyclic graphs
• Improved giraffe-wrangler.sh script
• Improvements to tail anchor synthesis in vg mpmap that should resolve some slowdowns
• vg rna bugfixes for variants at the ends of exons
• Better use of overlays in vg call
• BED-based extraction in vg find
• High-degree node removal in vg prune
• Bugfix for vg call -b parsing issue
• New -s option to vg augment to ignore out-of-graph alignments
• Binaries and Dockers now built for Nehalem architecture and above; AVX1 instruction support should no longer be required
• Faster packing with vg pack
• Ability to build a GBWT from sampled local haplotypes rather than a whole haplotype database
• Coverage threshold for faster vg augment
• Distance indexing with vg index now supports any graph filetype, not just .vg.

New and Updated Submodules

The BBHash submodule has been added.

The gbwt, gbwtgraph, libbdsg, libhandlegraph, and libvgio submodules have been updated.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.19.0

Buildable Source Tarball: vg-v1.19.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• The experimental vg mcmc command, a Markov-chain-Monte-Carlo genotyper prototype

• Improved clustering and intermediate scoring logic in vg gaffe, and the elimination of MultipathAlignments from that pipeline

• vg gaffe now emits MAPQ 0 for unmapped reads

• Handle-graph-ification of vg map, phase unfolding, vg vectorize, vg find, vg chunk, vg call`, and other operations using the xg index. Many commands now support more file types as arguments.

• Phase unfolder path verification no longer runs in parallel

• A new and improved XG index implementation. Note that backward compatibility is available only for XG indexes created with vg 1.18 or newer

• Giraffe Wrangler script now counts only mapped reads for identity computation, can take its inputs from Amazon S3, and binds BWA to a single NUMA node

• GBWT accesses now cached

• Giraffe Facts now reports statistics by filter instead of by mapping stage

• Vendored Protobuf submodule has been removed. vg can now build with (and indeed now requires) system-installed Protobuf 3 from a package manager

• vg call algorithm has been revised to use the much more efficient "pack" format, allowing much larger single calling operations

• A paired-end clustering algorithm for use in vg gaffe has been added, but not yet used

• Warnings about deprecated pointer types in lru_cache eliminated

• A new Dockerfile for a multi-stage build has been integrated directly into the vgteam/vg repository. The vgteam/vg_docker repository should no longer be required.

• The GBWTGraph has been moved to a submodule

• Added a magic number system to identify bare XG files as XG files

• Testing of builds of vg on Mac with GNU GCC has been discontinued. It is very difficult to get GNU GCC to build with the libc++ standard library, which packaged Protobuf builds for Mac use. GNU GCC should still work with Protobuf built with GNU GCC against its default libstdc++.

• vg version now reports the C++ standard library used in the build

• CFLAGS and CXXFLAGS are now honored by submodule dependencies in more (but not all) places. Some dependencies still ignore them.

• There is now a make static-docker target which quickly builds a static vg binary in the checked-out source tree and dumps it into a Docker container

• The build system now has its own opinion about whether the current vg binary is statically linked, as recorded by a marker file in lib/.

• User support for vg now ought to happen on Biostars: https://www.biostars.org/t/vg/

• Crash bug in optimal_score_on_genome has been fixed

• Memory leaks in vg map have been plugged

New System Dependencies

Protobuf 3.0 or greater, including libraries and compiler.

New and Updated Submodules

The protobuf submodule has been removed.

The DYNAMIC, gbwt, gcsa2, gfakluge, libbdsg, libhandlegraph, libvgio, lru_cache, and sdsl-lite submodules have been updated.

The gbwtgraph, ips4o, mmmultimap, and xg submodules have been added.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.18.0

Buildable Source Tarball: vg-v1.18.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Completely removed support for gPBWT's stored in XG indexes. Must now use vg gbwt.
• vg mod -g now preserves embedded paths.
• vg call and vg snarls can now use any HandleGraph implementation, which can significantly reduce memory usage.
• Fixed a bug that made it impossible to make distance indexes using XG files.
• Output files are now produced in uncompressed form, but bgzipped files can still be read.
• Improved speed in experimental gaffe mapper through caching and altered alignment algorithms.
• Introduced serialization for GBWTGraphs that can replace the XG used gaffe, significantly lowering memory usage.
• Transitioned to generic interface for XG to facilitate alternate backend data structures.
• Improved input checking in vg gamcompare.
• Fixed a bug that made invalid alignments in vg mpmap.
• Fixed a bug in path editing for VG graphs.
• Improved logic in vg deconstruct.
• Various build system improvements.

New System Dependencies

None

New and Updated Submodules

The FlameGraph submodule has been added.

The sglib submodule has been replaced by libbdsg.

The libvgio submodule has been updated.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.17.0

Buildable Source Tarball: vg-v1.17.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• VG annotate edited to search further for path hits.

• Single-ended Dijkstra algorithm factored out into its own HandleGraph algorithm.

• HandleGraphs can be loaded from files with dynamic implementation selection

• Fixed bugs in minimum distance clustering

• Fixed bugs in AlignmentEmitter

• vg gaffe improved speed and memory

• GBWT manipulation tools in vg gbwt

• vg mod -i deprecated, replaced by vg augment -i

• vg edit mostly refactored into augment.cpp, except for the interface with lists of paths.

• 'vg find -E' extracts nodes with an id range defined by a path interval.

• vg convert allows graph type conversion between graph implementations.

• vg gbwt now considers unphased homozygous variants as phased to prevent phase breaks. This can be disabled with option -z.

• Improved vg compilation on MacOS with gcc 9

• pack -q allows weighting support by mapping quality.

• Bug fixed in split strand overlay

• Bug fixes with compilation errors using G++ 9.1.0.

• Bug fix in tcmalloc.

• Speed up vg filter

• vg call now supports pack graph format.

• xg moved into vg namespace.

• vg call can now call sites in parallel.

• Names added to transcripts written to the gbwt index. Transcript naming has been simplified.

• gam output option from gbwt is removed. Gam files can be generated from gbwt transcript index using vg paths.

• Small speed-up for merging packer objects in vg pack.

• Options added for adding either reference paths or non-reference paths as embedded paths in a graph.

• vg deconstruct now works with more input graphs.

• PackedGraph and HashGraph now moved to sglib.

• vg paths -F now writes embedded paths in FASTA format.

• Factored out path-to-component index from XG

• Bug fix in rewrite_segment.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.16.0

Buildable Source Tarball: vg-v1.16.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Improvements to the snarl-based clusterer
• Support for thread names stored in the GBWT, and removal of xg thread name storage
• Removal of gPBWT construction from vg index
• GBWT maniputaltion tools in vg gbwt
• VCF-based snarl traversal finding, for re-calling VCF variants
• Recall option for vg call
• GaplessExtension support in the GaplessExtender and in vg gaffe
• Restored support for loading empty files, most obvious when working with snarls files
• "Funnel" statistics in vg gaffe
• Additional vg gaffe heuristics
• A graph equivalence check algorithm
• A random handle graph construction algorithm
• Bugfixes for PackedGraph
• Serialization support in PackedGraph and HashGraph
• Improved GBWT thread name syntax
• Disjoint splice graph support and other improvements in vg rna
• An updated internal handle graph interface with circular path support, path editing, serialization, and subsequence access
• Refactoring of IO functionality to go through the new libvgio.
• Removal of vg.proto, now provided by libvgio.
• Ability to trim softclips/tips when embedding paths into the graph
• Distance index improvements
• properly defined numerical class constants to fix unoptimized builds
• Giraffe Facts script for evaluating vg gaffe performance
• Snarl depth in SnarlManager
• Multiple GAM support in vg chunk
• Hash-based minimizer index
• Activation of compiler optimization in libstructures
• Bugfixes for vg call -f
• Bugfixes for vg snarls -m
• Fix for a crash caused by inconsistent handling of Ns.
• Single-orientation minimizers
• Fixed multipath alignment validation with self loops
• Xdrop aligner refactoring and documentation
• Xdrop support in vg gaffe
• Permit empty GBWT in vg paths, with a warning
• Building the maximum distance index is now optional
• Case-insensitive variant ids
• Improved PackedGraph storage backend with greater compression
• Bugfixes to and visualizations for MEMChainModel
• Bugfixes to chunked alignment
• Long read mapping preset in vg map
• Bugfixes to vg construct SV handling and alt path generation
• Bugfixes to Eades Algorithm sort

New System Dependencies

None

New and Updated Submodules

The libvgio submodule has been added.

The gbwt, gcsa2, libhandlegraph, protobuf, sonLib, structures, and vcflib submodules have been updated.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.15.0

Buildable Source Tarball: vg-v1.15.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Bugfixes for mpmap's clusterer
• Usage of libhandlegraph to provide the HandleGraph API
• Removal of broken MAF importer
• Fixes to VPKG backward compatibility, for reading files created with older versions of vg
• Fixes to SAM flag generation
• vg rna improvements
• A new functioin for converting handle graphs between implementations
• A GBWTGraph for "haplotype-consistent traversal"
• Fixes to unmapped read placement in vg surject BAMs
• More robust macOS builds with more include directories
• A snarl-based seed clusterer for the vg minimizer mapper
• A fixed vg::VG move constructor that actually works
• Improved inversion SV calling
• Support for Windows line endings in FASTA
• An AlignerClient base class for everything that uses Aligners
• Unified, multithreaded read output logic via AlignmentEmitter
• Distance index bugfixes and VPKG loading/saving
• TLEN support in SAM/BAM output and proper pairing computation.
• Handling of . ALTs in input VCF
• A prototype minimizer-based mapper, vg gaffe
• New default mpmap parameters
• Fix to Eades' Algorithm segfault bug
• Preliminary work on graph annotations

New System Dependencies

None

New and Updated Submodules

The libhandlegraph submodule has been added.
The fastahack, gbwt, and gssw submodules have been updated.

Make sure to git submodule update --init --recursive if building from source.

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.14.0

Buildable Source Tarball: vg-v1.14.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• A minimizer index can now be build with vg minimizer.
• vg surject now performs forced full-length alignments by default instead of with a command line flag.
• Better logic for marking properly paired reads in vg surject.
• Bug fix in VCF parsing for graph construction
• Improved accuracy of vg call on deletions.
• Fix a bug in vg call that caused crashes on some graph topologies.
• Fix a bug in vg chunk that would produce graph chunks that were inappropriate for variant calling.
• Improved selection of sites where vg call should call variants.
• Improved serialization that can encapsulate multiple indexes and recognize mismatched Protobuf file types.
• Fix help documentation error in vg index.
• Lighter weight graph implementations for improved memory efficiency.
• Improved speed in vg mpmap on graphs with tight cycles.
• Improved alignment performance in vg mpmap's anchoring heuristics.
• Improved speed on some embedded path manipulations.
• Additional GAM filtering options in vg filter.
• More semantically consistent behavior when trying to use haplotypes to score mappings with unphased variants.
• vg msga can be directed to align specific regions.
• Bug fixes in vg validate and vg mod
• Source tar balls now properly document the VG version used.

New System Dependencies

None

New and Updated Submodules

The structures submodule has been updated. Make sure to git submodule update to get it if building from source.

Static Binary:

Don't forget to mark the static binary executable:

chmod +x vg

Docker Image: quay.io/vgteam/vg:v1.13.0

Buildable Source Tarball: vg-v1.13.0.tar.gz

Includes source for vg and all submodules. Use this instead of Github's "Source Code" downloads; those will not build as they do not include code for bundled dependencies that the vg build process needs.

This release includes:

• Reduction of default node size for constructed graphs from 1000 to 32
• Some functionality moved from vg mod to vg augment
• GBWT improvements, including metadata
• Haplotype count override for haplotype scoring
• More permissive haplotype-based population scoring of read alignments in mpmap
• vg annotate now reports correct positions for the reverse strands of nodes
• MAPQ now reported for second multipath alignment when outputting paired multipath alignments from mpmap
• vg validate can validate GAM
• RDF improvements

New System Dependencies

None

New and Updated Submodules

The dozeu and gbwt submodules have been updated. Make sure to git submodule update to get them if building from source.

Static Binary:

Don't forget to mark the static binary executable:

chmod +x vg

Docker image: quay.io/vgteam/vg:v1.12.1

This release includes:

• Fix for an output corruption bug in vg construct

New System Dependencies

None

New and Updated Submodules

None

Docker image: quay.io/vgteam/vg:v1.12.0

This release includes:

• Ability to delay population scoring until after rescue in vg mpmap
• Improvements to Doxygen documentation generation
• Correct parsing of SVLEN and END fields for structural variants in VCF files
• Snarl-based distance indexing for clustering MEMs
• An initial Target Value Search implementation for distance computation in DAGs, with an A* implementation
• Improved mpmap stability
• Support for lower-case characters in the reference given to vg construct
• Ability for vg filter to defray away entire reads
• Ability to index an XG to GCSA without going through a VG file
• Much faster path insertion in VG::edit
• Checks for cyclic VCF reference paths in variant calling
• Ability to simplify out rare variants with vg simplify
• More efficient Clang build process for repeat builds
• Ability to suppress missing variant warnings when building a GCSA index
• Bugfix for calling long indels with multi-node alleles
• Fix for a bug that corrupted alt paths split across chunks when building the xg
• Improved Basic Operations tutorial
• Support for calling inversions and some other well-behaved non-ultrabubbles
• Ability to run vg call without passing the base graph
• SV recall improvements in vg call
• A watchdog to find slow reads in vg mpmap
• Simplifications to the Mac Travis build process
• Suppression of empty calls
• Refactored indexing code that can work on GAM and VG files.

New System Dependencies

None

New and Updated Submodules

The following submodules have been updated: vcflib,

Docker image quay.io/vgteam/vg:v1.11.0-0-gea4aaded-t238-run

This release includes:

• Added functionality in vg surject to preserve long insertions or deletions.
• Added parsing of symbolic variant representation from VCF (INV, CNV, etc.) in vg construct.
• Added population-based MAPQ adjustment in vg mpmap.
• Improvements to the robustness of vg add.
• Added an index into vg index for computing distances.
• Corrections to the online documentation.
• Support for clang compilation.
• Bugfixes in vg construct that led to invalid GBWT indexes.
• Bugfix in make clean's cleaning of some dependencies.
• Bugfix in vg genotype on low-coverage samples.
• Harmonized the BGZF usage in GAM files across the code base.
• Improved the functionality of graph normalization.
• Speed up deserialization of snarls.
• Stabilize the representation of snarls and chains with respect to orientation.
• Optimized GBWT construction for massive VCFs.

New System Dependencies

None

New and Updated Submodules

The following submodules have been updated: gfakluge, vcflib, gbwt, and gcsa2.

Docker image: quay.io/vgteam/vg:v1.10.0-0-ga4811915-t226-run

This release includes:

• A new backwards-compatible file format. Most files, including .vg and .gam, are now blocked-gzip compressed to permit random access.
• GAM sorting and indexing (with vg gamsort) for efficient random access to reads in GAM files.
• A new "xdrop" aligner backend.
• Support for repeated read annotations.
• Improvements to the vg mpmap aligner and its MAPQ computation.
• Fixes to vg paths so paths extracted from a GBWT are now valid.
• Thread name listing support in vg paths.
• An improved vg augment interface.
• Faster VCF parsing during graph construction and GBWT indexing.
• Correct semantics for vg snarls --max_nodes.
• Ability to get edge counts via the xg::XG public interface.

Breaking Changes

• vg find no longer has a -i/--alns-in option. You should use --alns-on instead; it's much simpler to think about.
• vg msga behavior has changed.

New System Dependencies

None

New and Updated Submodules

The vcflib and htslib submodules have been updated.

The dozeu and libdeflate submodules are new.

The htslib submodule has moved to https://github.com/vgteam/htslib. Developers will need to do a git submodule sync.

All Changes

Docker image: quay.io/vgteam/vg:v1.9.0-0-g3286e131-t206-run

This release includes:

• No more CLI tests for vg deconstruct (as the command is still under development)
• GFA output streaming, and a new GFA importer
• A new, as-yet-unused distance estimator intended for paired-end mapping
• mpmap tuning improvements and new options
• Functional support for building graphs from structural-variants in VCFs (vg construct -S)
• An increased Protobuf message size limit, and improved constructed graph chunking, which should solve most message-too-big-related errors
• Mitigations for strand and position bias when choosing between equivalent alignments
• Faster and more accurate haplotype thread generation for GBWT construction
• Random, samtools-compatible downsampling of alignments with vg filter -d
• Support for circular paths in the xg format and tools
• Better error detection for invalid option arguments
• More version and build metadata in vg version
• Bugfixes and improvements to the build process:
  • More correct net graph traversals
  • Addition of missing return statements to alleviate many potential crashes
  • Correct pairing of reads dumped from the buffer in mpmap

Breaking Changes

• This release generates XG version 10. It can read previous versions, but new indexes cannot be read by previous versions.
• The behavior of the new GFA importer differs form previous versions. Also, it validates GFA input more aggressively, and overlaps that are longer than sequences are no longer permitted.
• Previously, all whitespace was accepted as a GFF file separator. Now only tabs (which the spec requires) are accepted.
• vg is more conservative about its argument parsing. Numbers with trailing garbage after them, or decimals on integer parameters, will now generally be rejected with an error message, instead of being silently accepted.
• The vg version output looks different. Anyone who may be parsing it will need to adapt their code.

New Dependencies

None

New and Updated Submodules

The fastahack, gbwt, gcsa2, gfakluge, and vcflib submodules have all been updated.

The fastahack submodule has also moved to https://github.com/vgteam/fastahack. Developers will need to do a git submodule sync.

All Changes

Docker image: quay.io/vgteam/vg:v1.8.0-0-g3cb239e2-t176-run

This release includes:

• Improvements to haplotype-aware mapping
• The vg viz native graph rendering tool
• Improvements to plotting scripts
• Improvements to the build system
• Improvements to CI infrastructure error handling
• Bugfixes

Breaking Changes

• This release generates XG version 9. It can read previous versions, but new indexes cannot be read by previous versions.

New Dependencies

Now vg depends on Cairo, which vg viz uses for drawing.

New and Updated Submodules

The gbwt and raptor submodules have been updated.

All Changes

This release roughly coincides with resubmission of the vg paper. There have been a lot of changes since v1.6.0 and it would be difficult to provide a specific summary, but several items deserve mention.

• We improve vg's mapping runtime,
• integrate the GBWT,
• and refresh some critical components of the indexing process (pruning and kmer generation).
• We explore haplotype-based alignment scoring using the GBWT,
• and enable using it as the base thread storage system for all the tools in vg.
• We greatly improved the accuracy of the long read alignment algorithm in vg map.

This checkpoints the version that was used to generate experimental results in the preprint of our manuscript. (It will be linked here when made public.)

Attached is a stripped static binary that is portable across all 64-bit Linux systems that we have access to.

Recent developments in vg have focused on improving the mapper's performance via simulations.

As a result of this work, our simulations show that when aligning reads simulated from the 1000G variation graph, vg is equivalent to bwa mem in performance on the linear reference, and bests it considerably when aligning against the pangenome:

Experiments are completed with scripts/map-sim. The above example can be reproduced by:

vg/scripts/map-sim human.10M hs37d5.fa ref/index pan/index 32 "-s 571 -n 500000 -e 0.01 -i 0.002 -l 150 -p 500 -v 50" " "

Runtime is between 5 and 10x that of bwa mem.

There have been 286 commits since v1.3.0, so it is hard to summarize everything that's happened. Here are some of the highlights:

• RDF export (@JervenBolleman)
• improvements in build process (@edawson)
• chop/unchop procedures (@ekg)
• maintain paths in graph normalization (@ekg)
• graph unrolling/unfolding/dagification (@ekg)
• new GFA parsing (@edawson)
• improvements to MSGA (@ekg)

Since v1.2.0:

• Improvements in MSGA allow construction of the GA4GH bakeoff regions (however, problems with alignment are breaking the paths that are included).
• Mappings now include ranks, so that it's easy to obtain subsets of paths and merge subsets of graphs without needing to keep the entire path around.
• Better validation of paths by using ranks on mappings.

The changes in format necessitated changes in xg, and any indexes from older versions will be invalidated with this update.