What's Changed

Mostly big cleanup & deprecations for core.common module https://github.com/karlicoss/HPI/pull/380 https://github.com/karlicoss/HPI/pull/382

New core dependency: typing-extensions

Lots of cleanup/deprecations in core. This mostly shouldn't break any runtime code, but mypy might complain -- so best to migrate the imports appropriately

• core.common: deprecate outdated LazyLogger alias

• core.common: move Json, datetime_aware, datetime_naive, is_namedtuple, asdict to my.core.types

• my.core: deprecate Path/dataclass imports from my.core during type checking

  runtime still works for backwards compatibility

• core.common: move assert_subpackage to my.core.internal

• core.common: move mime-related stuff to my.core.mime

  no backward compat, unlikely it was used by anyone else

• core.common: move stats-related stuff to my.core.stats and add more thorough tests/docs

  deprecate core.common.stat and core.common.Stats with backwards compatibility

• core: cleanup my.core.common.unique_everseen

  • move to my.core.utils.itertools
  • more robust check for hashable types -- now checks in runtime (since the one based on types purely isn't necessarily sound)
  • add more testing

• my.core.common: move warn_if_empty to my.core.utils.itertools, cleanup and add more tests

• core.common: move away import related stuff to my.core.utils.imports

  moving without backward compatibility, since it's extremely unlikely they are used for any external modules

  in fact, unclear if these methods still have much value at all, but keeping for now just in case

• core.common: move listify to core.utils.itertools, use better typing annotations for it

  also some minor refactoring of my.rss

• core: cleanup itertool style helpers

  • deprecate group_by_key, should use itertool.bucket instead
  • move make_dict and ensure_unique to my.core.utils.itertools

• core: cleanup deprecations, exclude from type checking and show runtime warnings

  among affected things:

  • core.common.assert_never
  • core.common.cproperty
  • core.common.isoparse
  • core.common.mcachew
  • core.common.the
  • core.common.tzdatetime
  • core.compat.sqlite_backup

• core: cleanup deprecations, exclude from type checking and show runtime warnings

  among affected things:

  • core.common.assert_never
  • core.common.cproperty
  • core.common.isoparse
  • core.common.mcachew
  • core.common.the
  • core.common.tzdatetime
  • core.compat.sqlite_backup

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.4.20240810...v0.5.20240824

What's Changed

• fix for new instagram gdpr format by @karlicoss in https://github.com/karlicoss/HPI/pull/371
• fix legacy google takeout by @karlicoss in https://github.com/karlicoss/HPI/pull/372
• fbmessenger.android: fix minor issue with processing thread participants by @karlicoss in https://github.com/karlicoss/HPI/pull/373
• fix my.twitter.android by @karlicoss in https://github.com/karlicoss/HPI/pull/374
• tests: use updated conftest from pymplate, this allows to run individual test modules properly by @karlicoss in https://github.com/karlicoss/HPI/pull/375
• improve testing consistency, simplify tox by @karlicoss in https://github.com/karlicoss/HPI/pull/376
• replace deprecated utcnow and utcfromtimestamp by @karlicoss in https://github.com/karlicoss/HPI/pull/378

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.4.20240506...v0.4.20240810

What's Changed

• monzo.monzoexport: initial module by @karlicoss in https://github.com/karlicoss/HPI/pull/346
• time.tz.via_location: more consistent behaviour wrt caching by @karlicoss in https://github.com/karlicoss/HPI/pull/351
• my.codeforces/my.topcoder: move to the top level & cleanup by @karlicoss in https://github.com/karlicoss/HPI/pull/352
• my.body.sleep: massive speedup for average temperature calculation by @karlicoss in https://github.com/karlicoss/HPI/pull/353
• gpslogger: catch broken xml file error by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/356
• initial module for official twitter app for android by @karlicoss in https://github.com/karlicoss/HPI/pull/359
• my.youtube.takeout: fix exception handling by @karlicoss in https://github.com/karlicoss/HPI/pull/361
• fbmessenger.android: expose contacts by @karlicoss in https://github.com/karlicoss/HPI/pull/362
• whatsapp.android: expose all entities extracted from the db by @karlicoss in https://github.com/karlicoss/HPI/pull/363
• google takeout: support multiple locales, resolves #330 by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/364
• my.google.maps: intitial module for extracting saved places data from Android app by @karlicoss in https://github.com/karlicoss/HPI/pull/365
• my.twitter.android: better detection of own user id by @karlicoss in https://github.com/karlicoss/HPI/pull/368
• my.tinder.android: better error handing and fix case with empty db by @karlicoss in https://github.com/karlicoss/HPI/pull/369
• smscalls: parse mms from smscalls export by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/370

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.4.20231102...v0.4.20240506

not ideal that the date in the version is the future date.. but oh well

What's Changed

• ci: fix wrong release command by @karlicoss in https://github.com/karlicoss/HPI/pull/344

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.4.20231101...v0.4.20231102

Big update, even though things should be backwards compatible, it's recommended to install the package again if you use editable git checkout (pip3 install --user -e /path/to/hpi_repo)

Also worth installing optional dependencies (pip3 install --user -e /path/to/hpi_repo[optional])

What's Changed

Core:

• core: implement more methods for ZipPath and better support for get_files by @karlicoss in https://github.com/karlicoss/HPI/pull/285
• query: add --warn-exceptions, dateparser, docs by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/290
• my.core.pandas: rely on typing annotations from types-pandas by @karlicoss in https://github.com/karlicoss/HPI/pull/292
• core: experimental import of my._init_hook to configure logging/warnings/env variables by @karlicoss in https://github.com/karlicoss/HPI/pull/298
• core/logging: overhaul and many improvements -- mainly to deprecate abandoned logzero by @karlicoss in https://github.com/karlicoss/HPI/pull/299
• minor module updates + logging/warnings improvements by @karlicoss in https://github.com/karlicoss/HPI/pull/300
• general: move reddit tests into my/tests + tweak my.core.cfg to be more reliable by @karlicoss in https://github.com/karlicoss/HPI/pull/295
• core/cli: allow user to bypass PEP 668 by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/302
• core/cli: update vendored completion files by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/304
• core/logging: fix issue with logger setup called multiple times when called with different levels by @karlicoss in https://github.com/karlicoss/HPI/pull/310
• my.core.logging: compatibility with HPI_LOGS by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/307
• core/stats: exclude contextmanagers from guess_stats by @karlicoss in https://github.com/karlicoss/HPI/pull/334
• move hpi specific compat stuff to core.hpi_compate + minor updates for hypothesis and kobo by @karlicoss in https://github.com/karlicoss/HPI/pull/335
• core/config: implement a warning if config is imported from the dir other than MY_CONFIG by @karlicoss in https://github.com/karlicoss/HPI/pull/336
• general: improve logging during file processing in various modules by @karlicoss in https://github.com/karlicoss/HPI/pull/339
• general: enhancle logging for various modules by @karlicoss in https://github.com/karlicoss/HPI/pull/340
• general: deprecate some old methods by hiding behind TYPE_CHECKING by @karlicoss in https://github.com/karlicoss/HPI/pull/342
• core: add helper for more_iterable to check that all types involved are hashable by @karlicoss in https://github.com/karlicoss/HPI/pull/343
• core/kompress: move vendorized kompress to _deprecated, use kompress library directly by @karlicoss in https://github.com/karlicoss/HPI/pull/319
• more consistent handling of zip archives in get_files, less kompress boilerplate in modules by @karlicoss in https://github.com/karlicoss/HPI/pull/321
• experimental: add a hacky helper to import "original/shadowed" modules from within overlays by @karlicoss in https://github.com/karlicoss/HPI/pull/322
• core/stats: less duplication in output, report stats of first item, report input stats by @karlicoss in https://github.com/karlicoss/HPI/pull/324
• core.logging: ignore CollapseLogsHandler if we're not attached to a terminal by @karlicoss in https://github.com/karlicoss/HPI/pull/329
• dcotor: make compileall check a bit more defensive by @karlicoss in https://github.com/karlicoss/HPI/pull/332

modules:

• my.whatsapp.android: initial module by @karlicoss in https://github.com/karlicoss/HPI/pull/284
• smscalls: make some fields optional by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/287
• cli: add option to output locations to gpx files by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/286
• smscalls: make checking for keys stricter by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/289
• browser: fix duplicate logs when fetching loglevel by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/309
• new module: Harmonic app for Hackernews by @karlicoss in https://github.com/karlicoss/HPI/pull/313
• location: make accuracy default config floats by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/314
• reddit.rexport: big cleanup + use new types from rexport.dal by @karlicoss in https://github.com/karlicoss/HPI/pull/320
• instagram: better normalising/error handling by @karlicoss in https://github.com/karlicoss/HPI/pull/325
• updates for whatsapp.android and tinder.android by @karlicoss in https://github.com/karlicoss/HPI/pull/326
• twitter.archive: deduplicate results via json.dumps by @karlicoss in https://github.com/karlicoss/HPI/pull/327
• reddit.rexport: experiment with using optional cpu pool (can potentially be used by other HPI modules) by @karlicoss in https://github.com/karlicoss/HPI/pull/328
• small zulip and dogsheep updates by @karlicoss in https://github.com/karlicoss/HPI/pull/331
• my.emfit: cleanup and pass cpu pool by @karlicoss in https://github.com/karlicoss/HPI/pull/333
• my.stackexchange.stexport: small cleanup & stat improvements by @karlicoss in https://github.com/karlicoss/HPI/pull/337
• fbmessenger.android: support processing msys database by @karlicoss in https://github.com/karlicoss/HPI/pull/341

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.3.20230327...v0.4.20231101

What's Changed

• vk_messages_backup updates by @karlicoss in https://github.com/karlicoss/HPI/pull/274
• location fallback by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/263
• add example for denylist, update ci by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/275
• core.logging/core.kompress updates by @karlicoss in https://github.com/karlicoss/HPI/pull/277
• google_takeout: add semantic location history by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/278
• cli completion doc updates, hide legacy import warning by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/279
• some updates for bumble/fbmessenger/twint by @karlicoss in https://github.com/karlicoss/HPI/pull/280
• better unification of my.instagram and various enhancements by @karlicoss in https://github.com/karlicoss/HPI/pull/282
• docs: fix typos by @kianmeng in https://github.com/karlicoss/HPI/pull/281
• telgram_backup updates by @karlicoss in https://github.com/karlicoss/HPI/pull/283

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.3.20230221...v0.3.20230327

What's Changed

• my.telegram: initial module by @karlicoss in https://github.com/karlicoss/HPI/pull/270
• my.fbmessenger.android updates by @karlicoss in https://github.com/karlicoss/HPI/pull/271
• my.telegram.telegram_backup enhancements by @karlicoss in https://github.com/karlicoss/HPI/pull/273

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.3.20230209...v0.3.20230221

What's Changed

• my.fbmessenger.export: use context manager to properly close sqlite connection by @karlicoss in https://github.com/karlicoss/HPI/pull/268
• core: better support for config overrides (tmp_config) by @karlicoss in https://github.com/karlicoss/HPI/pull/269

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.3.20230207...v0.3.20230209

What's Changed

• core/serialize: serialize stdlib Decimal class by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/259
• google.takeout.parser: recreate cache on upgrade by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/261
• switch modules from dataset to use raw sqlite3 by @karlicoss in https://github.com/karlicoss/HPI/pull/267
• twitter archive + instapaper fixes by @karlicoss in https://github.com/karlicoss/HPI/pull/260
• • docs: browser - fix examples for config by @seanbreckenridge in https://github.com/karlicoss/HPI/pull/258

Full Changelog: https://github.com/karlicoss/HPI/compare/v0.3.20220607...v0.3.20230207

mainly so hpi install --parallel is available

mostly fixes for Windows and Python 3.10

new google takeout module!

v0.3.20210220

General/my.core changes:

• a3305677b24694391a247fc4cb6cc1237e57f840 deprecate my.cfg, instead my.config can (and should be) used directly

• 0534c5c57dc420f9a01387b58a7098823e54277e new cli feature: module management

  cli: add hpi module install and hpi module requires

  relevant: https://github.com/karlicoss/HPI/issues/12, https://github.com/karlicoss/HPI/issues/79

• 97650adf3b48c653651b31c78cefe24ecae5ed4f add discovery_pure module to get modules and their dependencies via ast module

• f90599d7e4463e936c8d95196ff767c730207202 make module discovery rely on =ast= module
  Hopefully it will make it more robust & much faster.

• 07f901e1e5fb2bd3009561c84cc4efd311c94733 helpers for automatic dataframes from sequences of NamedTuple/dataclass

• 4012f9b7c2a429170df8600591ec8d1e1407b162 more generic functions to jsonify data

• 746c3da0cadcba3b179688783186d8a0bd0999c5 core.pandas: allow specifying schema; add tests

• 5313984d8fea2b6eef6726b7b346c1f4316acd01 add tmp_config context manager for test & adhoc patching

• df9a7f7390aee6c69f1abf1c8d1fc7659ebb957c core.pandas: add check for 'error' column + add empty one by default

• e81dddddf083ffd81aa7e2b715bd34f59949479c proprely resolve class properties in make_config + add test

Modules:

• some innitial work on filling InfluxDB with HPI data

• pinboard

  • 42399f6250d9901d93dcedcfe05f7857babcf834: breaking backwards compability, use pinbexport module directy

    Use 'hpi module install my.pinboard' to install it

    relevant: https://github.com/karlicoss/HPI/issues/79

• stackexchange

  • 63c825ab81bb561e912655e423c6b332fb6fd1b4 use GDPR data for votes
  • ddea816a49f5da79fd6332e7f6b879b1955838af use proper pip package, add stat

• bluemaestro

  • 6d9bc2964b24cfe6187945f4634940673dfe9c27 populate grafana
  • 1899b006de349140303110ca98a21d918d9eb049 investigation of data quality + more sanity checks
  • d77ab92d8634d0863d2b966cb448bbfcc8a8d565 get rid of unnecessary file, move to top level

• runnerup

  • 6b451336ed5df2b893c9e6387175edba50b0719b Initial parser for RunnerUp data which I'm now using instead of Endomondo

Misc:

• f102101b3917e8a38511faa5e4fd9dd33d284d7e core/windows: fix get_files and its tests

• 56d5587c209dcbd27c7802d60c0bc8e8e2391672 CI: clean up tox config a bit, get rid of custom lint script

• d562f00dca720fd4f6736377a41168e9a796c122

  tests: run all tests, but exclude tests specific to my computer from CI
  controllable via HPI_TESTS_KARLICOSS=true

• improved mypy coverage

General:

• documentation improvements, my.config in the package now serves both as documentation & tests helper
• my.kython: deprecated, moved to my.core
• my.core.error: error handling enhancements
• stat now supports pandas.DataFrame
• minor enhancements to hpi doctor

Module changes:

• new: my.time.tz: provides localize function to attach a timezone to an arbitrary date https://github.com/karlicoss/HPI/pull/96
• my.location: exposes a combined log of user's location, with fallback to 'default' home locations https://github.com/karlicoss/HPI/pull/95 https://github.com/karlicoss/HPI/pull/99
• my.fbmessenger: switch to use a proper PIP package
• new: my.taplog data provider

Misc: more tests & mypy running on CI

See commit changelog for more details.

Special thanks to @seanbreckenridge for discussions!

core:

• core config section, allow enabling/disabling modules
• better module detection
• experimental helpers to supply errors with datetime information
• experimental helpers to make it easier to override config, e.g. see https://github.com/karlicoss/dashboard/blob/ff15295d8118494f986ed13757e022ae2aa18000/src/dashboard/data.py#L99-L117 for example usage
• nicer warnings
• initial core.pandas helpers

cli: hpi doctor enhancements

modules:

• big update for instapaper, reddit, ghexport, hypothesis, rescuetime, pocket.
  DEPRECATED: the old way of specifying dependecies either as

  • config parameters (e.g. hypexport = "/path/to/hypexport_on_disk")
  • symlinks in the config (e.g. my/config/repos/hypexport)

  These dependencies are proper PIP packages now, see https://github.com/karlicoss/HPI/issues/79 for the discussion about the change
  This is backwards compatible, but please reinstall them properly as soon as you can (see the updated instructions in the corresponding repos, e.g. https://github.com/karlicoss/instapexport#setting-up)

• provide fake data providers for some modules

• better mypy coverage for some modules

• for other minor updates, see the commit messages

• core helper to open databases as read-only/immutable (useful for Docker mounts)
• some cachew experiments, including cachew base directory (still work in progress)
• more stat functions and improvements for stats
• initial work on using proper virtualenv for DAL helpers (using emfit to start with)

modules affected:

• reddit
• pdfs
• bluemaestro
• emfit
• twint
• rescuetime