PyHDX

Features

• Integrates with hdxms-datasets, allows public dataset loading from web interface (#327)

Fixes

• Fixed RFU app file input, improvements to web app peptide input (#318 )
• Fixed changing norms after plotting linear bars (e86bf4ddf3f4b75bdfae0bbab6b85f9f2ee2ca3c)
• Various minor fixes

Dependencies

• Switched to proplot == 0.9.7 version requirement.

Deprecations

• Removed the Protein object (#315).

Packaging and distribution:

• Switched to hatch for packaging (#331).
• Switched to mkdocs for documentation building (#314).

• Added a peptide uptake simulation app (#285 )
• Updated configuration object to Omegaconf, allow users to select different configuration files (#291)
• Adding fitting of D-uptake values per timepoint (#296)
• Perform per-timepoint D-uptake fitting and calculate ΔD-uptake in the web application (#297)
• Added storing and exporting of user settings and logs from web application (#301)
• Refactored how data is loaded into PyHDX (#305)

Fixes:

• Fixed bug where sessions files do not reload graphs (#289)

Bugfix release and some updates to documentation:

• Fixed progress bars (4f557b9e6ac0d42382b9ca0980dfd87e576f1120)
• Moved pyyaml requirement to base requirements (57835ee8c412551493d16d09c5cd13bdb6c5ae0f)
• Updated the readme (57835ee8c412551493d16d09c5cd13bdb6c5ae0f)
• Add resources and citation section to docs (cfc50e062262f5b299ad9221345d96c48834541d)

Release v0.4.0 is the combination of 9 beta versions, spanning a total of ~10 months of PyHDX development.
Please refer to the individual beta releases for more detail.

Major improvements of the new version 0.4.0 release are:

Base PyHDX:

• Batch fitting of multiple large HDX datasets (+ GPU support)
• Batch processing of large HDX datasets through .yaml file spec
• Added $pyhdx process CLI command (early alpha feature)
• Added plotting and pdf output modules
• Major API changes

PyHDX-web:

• Two complete overhauls of the web interface, improving internal logic of the web interface
• Updated the GoldenLayout template to a combination of panel's GoldenLayout and the elvis golden layout
• Addition of the PDBeMolstar protein viewer, allowing direct export of colored structures
• Direct output of publication quality graphs from the web interface
• Create differential HDX datasets (ΔΔG / ΔRFU)

This release:

• Updated docs and docstrings, introduced PEP483 type hints (#271)
• Update to panel version 0.13.0 (#264)
• Added black, mypy, flake8, (isort) linters

• Updated async handling of tasks, removed loop in main controller checking for new futures (#255)
• Fitting of multiple protein states individually through the web interface (#257)
• Allows for either rates or deltaG initial guesses (#257)
• Allows for one-to-one or one-to-many initial guesses (#257)
• Data input in web app in batch through yaml spec file (#258)
• Fixed web application not working in Chromium (#259)
• Added $ pyhdx process cli command (#267)

Changes:

• Added ΔRFU in web interface
• Fixed bug where duplicate entries appeared in controls by using corrected start and end fields
• Initial guesses are now interpolated at fitting function and guesses are now broadcast over sample axis
• Added MultiTransform which takes multiple sources as input and SelectTransform for the protein view (3d21e4d70a96cc686a085a2ed0fc21c085fa5463)
• Reverted template back to GoldenTemplate #249
• Fixed initial guesses by association model (fbe9c23dadc31a2966afd115b7e0b1d9a25d898a)
• Added PDBeMolstar protein viewer #252
• Minor tweaks to web app css #254

Changes:

• Web interface performance improved by using hashes and caches (#242)
• Export of grouped rfu linear bars (f7792015a9d04d964815cc373e7c597ee80c1e35)
• Graph export bugfix (34552b02f1a496bf9e31a799f413acd4317ad53b)
• Added RFU app and index page (#243)

Patch release with bugfixes and packaging updates

This release features mostly an overhaul of the web interface (#234).

Internal changes:

• Created new classes handling data/parameter flow within the web interface: Source, Transform, View, Opts
• Added a Constructor class which generates the web application from a yaml spec file
• Color transforms (colormap + normalize) are now defined on a per-datatype basis (ie one for rfu, dG, ddG, mse). Default color transforms can be changed with the ColorTransformControl control panel.
• The NGL protein viewer is now implemented as ReactiveHTML, obviating the need for custom bokeh models
• Timepoint zero is by default not selected
• The template was updated to the FastGridTemplate.

New features:

• Scatterplot of RFU values per residue.
• Create differential HDX ΔΔG comparisons directly.
• Download .zip session file and restore session from session file.
• Download ΔG/ΔΔG proplot/matplotlib graphs (linear_bars, scatter, rainbowclouds) in .png, .pdf. .svg or .eps format.
• Export data tables as hex color code tables (for supported tables)
• Export of pymol coloring scripts for RFU values

Refactors:

• deltaG was refactored to dG (ddG for deltadeltaG)

Other changes:

• csv_to_dataframe now also accepts BytesIO objects
• local_cluster.default_cluster now accepts additional kwargs
• Various bugfixes

Full Changelog: https://github.com/Jhsmit/PyHDX/compare/v0.4.0b4...v0.4.0b5

Major changes to plotting: (#232)

• Added functions for making standard output plots in the plot.py module

• Added functions for default colormaps and normalizers; defined default colormaps for standard data types (rfu, dG, ddG)

• Added plotting class FitResultPlot which makes it easier to generate plots from a fit result

• Updated the pdf output class (now named FitReport) which allows all standard plots to be combined into a .pdf file

• SingleFitResult and BatchFitResult are now merged into one object.

Changes:

• Switched to pandas DataFrame instead of numpy structured arrays for data object's internal tables (#227)
• Refactored uptake_corrected to d_exp for FD-corrected D-uptake values (#227)
• Properties rfu_residues, rfu_peptides are now pandas Series objects (#227)
• Removed python 3.6 support (#227)
• Added option to filter input data (peptide table data) in batch processing (8fa9c5b542feea97001a8bb8d6e05537f80bdcc5)
• Fitting can be done on a specific device (ie CUDA GPU) and the fitting dtype can be specified (float32/float54) (#231)

Changes:

• Refactored config; moved config files, refactored cluster to scheduler_address where applicable (#220)
• Changed Lagrangian squared error reduction from 'total' to 'mean' (affecting regularization values) (#221)
• Regularization components are treated separately and stored in losses dataframe (#222)
• Internal time units are now in seconds (73c387c5337d5bb7a55f3d0df9d5b0bfdb577e4b)
• Added optimization loop callbacks and checkpoint (#222)
• Allow for saving of HDXMeasurementSet objects to .csv/.txt file (9cca09f4f99ddc964247ccbb52f3802268e33ded)
• Allow for reading of 'HDXMeasurementSet` objects from .csv/.txt file (4e9fcaca80e53b6bb4a448411dcc8a4cd2795cc1)
• Implemented __str__ for HDXMeassurement objects (5122d239f9f58dee003989390e0613ea96abce54)
• Updated yaml batch formatting format (7d94411014d884a69ee0dae69a, 7c71847caa2f8bf5f7339b4c8ef4e3d4a490d74c)

Changes:

• Added additional fit feedback panels; MSE per peptide, D-uptake curves + fit and losses/epoch #212
• Documentation and docstrings (#210, #213, #214)
• Updated pdf module, now requires proplot and generates figure subplots per page instead of per subplot

Fixes:

• Fixed mse and losses for single fits
• Extension for table export (#218 )

Big changes in the new 0.4.0 version!
Major changes include an overhaul of the web interface and refactoring of major parts in the code.
Thanks for all contributions @sebaztiano @sajetan

Major changes:

Web interface

• Users can add multiple datasets ('state') at the same time and perform analysis in batch on these datasets. The datasets, operations on these datasets and the resulting views show in the web interface are now all based on lumen objects (Source, Filter, Transform, View) #188
• NGL viewer was implemented as a custom bokeh extension #201
• Classification control panel can now also apply matplotlib/colorcet colormaps

Refactoring of the PyHDX API

• Fitting API is now functional instead of object oriented; fit_gibbs_global(..), fit_gibbs_global_batch(..) #199
• KineticsSeries object was refactored to HDXMeasurement
• Added HDXMeasurementSet object
• The scores attribute (percentage D-uptake) was refactored to rfu (relative fractional uptake, 0-1)
• Updated fileIO functions, enable saving of kwargs to reload the file as well as saving metadata #205
• Allow saving and loading of fit results (#208) and HDXMeasurement objects #205

Minor changes:

• Versioning by versioneer instead of pbr (#207)
• Added k_obs as output variable of the fit (ce4f58326e18e3e80c64f7a4bf3e19132ddc1c22)
• Added console script entry point to run the web interface (pyhdx serve) #191
• Export of all peptide uptake graphs + fitted curves in pdf format #209

Bugfixes:

• Alignment of exchanges and k_int (8301686c3cda6eed61e68215bdd3378cc6ae8f6c)

Minor release fixing a bug in requirements.txt

Major changes:

• Estimate errors in ΔG fit from covariances obtained from the Hessian
• Allow fitting of multiple 'states' (conditions) in batch
• Allow fitting of 2 aligned (homologous) protein conditions in batch
• Added console entry script such that the pyhdx server can be started with pyhdx serve (#191)

Minor changes:

• Input/output data table formats are now in comma separated values (.csv) format
• Updated support for reloading and fitting data in batch from .yaml files
• Updated rgb_to_hex function to work on arrays (vectorized)
• Added a colorbar to plotting of peptides (in plot.py)
• Refactored k_series to series in KineticsFitting objects
• Refactored cov to coverage in KineticsSeries objects
• Updates to docstrings
• Protein full sequences can optionally be added to Protein objects
• Moved fitting loop to separate function outside of main fitting object
• Protein objects now more closely mimic pandas' dataframes

Web application updates:

• 'Scores' (RFU) can be obtained as a table from the GUI
• User feedback on relative contribution of regularizer to fit result
• Control sidebar is now formatted as accordions where only one can be opened at a time
• Panel requirement bumped to >= 0.11, fixing scrolling issues in sidebar
• Added config file, currently lists default Dask cluster address to use

• Various bug fixes!

Thanks to @sajetan for contributing (#191, #181)

This release accompanies submission to bioRxiv as well as various bugfixes.

• Updates to documentation (available as pdf)
• Refactored how the apps are created in apps.py
• Added all sub-controllers in controllers.py and main controllers in main_controllers.py

Bugfixes:

• Fixed bug where linking the x range of panels would cause an error (610d985)
• Removed unused .data attribute (#108)
• Fixed updates of figures, now via callback (#117)
• Fixed protein view not updating without switching between datasets (#114)

This release features an embedded Protein Viewer and implementation of Panel Themes / Templates.

Main features:

• NGL Protein viewer
• Streamlined layouts with Elvis (https://github.com/LeonvanKouwen/elvis) with Panel Templates (panel >0.10.0)
• Color protein by custom continuous color map (36e7070)
• Added Dark/Light themes
• Separate app for loading single state fitting results and applying classification (coloring)
• Separate app for comparing two protein states

Other changes:

• Added DataSource object which is now used for 'publishing' data
• Control and Figure panels are now supplied to controller as their `class`` rather than a string
• Added setup_hooks() method to couple Figures to Controllers
• Added a Logging panel
• Fixed ignoring prolines to always be True

Fixes:

• read_to_np can read files with two comment lines and with hex color codes (#000000)
• Fixed datapoints without coverage showing up (#89)
• Fixed coverage intervals (#91)
• Various bugfixes
• Improved documentation

This release mostly features updates in the GUI:

• Added publish_data functionality which allows pushing data sources to client graphs
• Linking x-range of coverage and rates/protection factor figures is working again
• Coverage and rates/protection figures are now aligned by introducing left border space
• Different color maps for the coverage figure can be selected
• A color bar was added to the coverage control panel

Other changes:

• Added tests for the GUI
• Removed old code and comments, specifically RateFigureOld and CoverageFigureOld
• Added a script to run a cluster locally
• Added a new function for obtaining initial guesses (needs to be added to GUI)

Fixes

• Fixed thresholding values bug (#86)
• Fixed fit1 in initial guesses by default (#87)
• Fixed classification not working after second iteration of fitting (#71)
• Fixed not re-applying thresholds when threshold values are set (#88)
• Fixed mouse-over hovertools (#73)

The main new feature of this beta release is the change to fitting of the whole protein in one go rather than splitting it into sections and fitting one by one.

Advantages include:

• Significant speed increase
• Regularizer can propagate across gaps of no coverage
• Regularizer now propagates across prolines

Changelog:

• Default value for l1 regularizer is now 2000 because it was changed to average rather than sum
• Default value of epochs from 10000 to 100000
• Added method for getting intersections of KineticSeries
• Added method to get k_int structured array, k_int for prolines and no coverage is now 0.
• Updated internal intervals to [start, end) rather than [start, end]
• Added TFCoverage object to allow fitting over the whole protein, to be merged in future releases
• Added a 'uptake_corrected' field which is absolute D uptake but corrected by FD control
• Added 'half life' initial guesses to GUI
• Added full protein fit to GUI (replaced old fitting procedure)