Fast Tensors Packaging library for text, image, video, and audio data compatible with PyTorch, TensorFlow, & NumPy 🖼️🎵🎥 ➡️ 🧠
Three simple things this library does to you:
Where do we use it? To connect the Data Storage layer of UKV to High-Performance Computing libraries like TensorFlow and PyTorch.
Most common file-formats have "signatures" or "magic numbers" embedded into them. Often, as the prefix of the byte-stream.
Libraries implementing the first step have been implemented for other languages:
In fact, TenPack is just a CMake-friendly generalization of those libraries with a C interface and focus on memory reusing.