ralph

Added

• Backends: Add Writable and Listable interfaces to distinguish supported
  functionalities among data backends
• Backends: Add max_statements option to data backends read method
• Backends: Add prefetch option to async data backends read method
• Backends: Add concurrency option to async data backends write method
• Backends: Add get_backends function to automatically discover backends
  for CLI and LRS usage
• Backends: Add client options for WSDataBackend
• Backends: Add READ_CHUNK_SIZE and WRITE_CHUNK_SIZE data backend settings
• Models: Implement Pydantic model for LRS Statements resource query parameters
• Models: Implement xAPI LMS Profile statements validation
• Models: Add EdX to xAPI converters for enrollment events
• Project: Add aliases for ralph-malph extra dependencies: backends and
  full

Changed

• Arnold: Add variable to override PVC name in arnold deployment
• API: GET /statements now has "mine" option which matches statements that
  have an authority field matching that of the user
• API: Invalid parameters now return 400 status code
• API: Forwarding PUT now uses PUT (instead of POST)
• API: Incoming statements are enriched with id, timestamp, stored
  and authority
• API: Add RALPH_LRS_RESTRICT_BY_AUTHORITY option making ?mine=True
  implicit
• API: Add RALPH_LRS_RESTRICT_BY_SCOPE option enabling endpoint access
  control by user scopes
• API: Enhance 'limit' query parameter's validation
• API: Variable RUNSERVER_AUTH_BACKEND becomes RUNSERVER_AUTH_BACKENDS, and
  multiple authentication methods are supported simultaneously
• Backends: Refactor LRS Statements resource query parameters defined for
  ralph API
• Backends: Refactor database, storage, http and stream backends under
  the unified data backend interface [BC]
• Backends: Refactor LRS query_statements and query_statements_by_ids
  backends methods under the unified lrs backend interface [BC]
• Backends: Update statementId and voidedStatementId to snake_case,
  with camelCase alias, in LRSStatementsQuery
• Backends: Replace reference to a JSON column in ClickHouse with
  function calls on the String column [BC]
• CLI: User credentials must now include an "agent" field which can be created
  using the cli
• CLI: Change push to write and fetch to read [BC]
• CLI: Change -c --chunk-size option to -s --chunk-size [BC]
• CLI: Change websocket backend name -b ws to -b async_ws along with it's
  uri option --ws-uri to --async-ws-uri [BC]
• CLI: List cli usage strings in alphabetical order
• CLI: Change backend configuration environment variable prefixes from
  RALPH_BACKENDS__{{DATABASE|HTTP|STORAGE|STREAM}}__{{BACKEND}}__{{OPTION}}
  to RALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}
• Models: The xAPI context.contextActivities.category field is now mandatory
  in the video and virtual classroom profiles. [BC]
• Upgrade base python version to 3.12 for the development stack and Docker
  image
• Upgrade bcrypt to 4.1.2
• Upgrade cachetools to 5.3.2
• Upgrade fastapi to 0.108.0
• Upgrade sentry_sdk to 1.39.1
• Upgrade uvicorn to 0.25.0

Fixed

• API: Fix a typo ('attachements' -> 'attachments') to ensure compliance with
  the LRS specification and prevent potential silent bugs

Removed

• Project: Drop support for Python 3.7
• Models: Remove school, course, module context extensions in Edx to xAPI
  base converter
• Models: Remove name field in VideoActivity xAPI model mistakenly used in
  video profile
• CLI: Remove DEFAULT_BACKEND_CHUNK_SIZE environment variable configuration

Install with:
pip install --pre --upgrade ralph-malph

Features

Added

• Backends: Add Writable and Listable interfaces to distinguish supported
  functionalities among data backends
• Backends: Add max_statements option to data backends read method
• Backends: Add prefetch option to async data backends read method
• Backends: Add concurrency option to async data backends write method
• Backends: Add get_backends function to automatically discover backends
  for CLI and LRS usage
• Backends: Add client options for WSDataBackend
• Backends: Add READ_CHUNK_SIZE and WRITE_CHUNK_SIZE data backend settings
• Models: Implement Pydantic model for LRS Statements resource query parameters
• Models: Implement xAPI LMS Profile statements validation
• Models: Add EdX to xAPI converters for enrollment events
• Project: Add aliases for ralph-malph extra dependencies: backends and
  full

Changed

• Arnold: Add variable to override PVC name in arnold deployment
• API: GET /statements now has "mine" option which matches statements that
  have an authority field matching that of the user
• API: Invalid parameters now return 400 status code
• API: Forwarding PUT now uses PUT (instead of POST)
• API: Incoming statements are enriched with id, timestamp, stored
  and authority
• API: Add RALPH_LRS_RESTRICT_BY_AUTHORITY option making ?mine=True
  implicit
• API: Add RALPH_LRS_RESTRICT_BY_SCOPE option enabling endpoint access
  control by user scopes
• API: Enhance 'limit' query parameter's validation
• API: Variable RUNSERVER_AUTH_BACKEND becomes RUNSERVER_AUTH_BACKENDS, and
  multiple authentication methods are supported simultaneously
• Backends: Refactor LRS Statements resource query parameters defined for
  ralph API
• Backends: Refactor database, storage, http and stream backends under
  the unified data backend interface [BC]
• Backends: Refactor LRS query_statements and query_statements_by_ids
  backends methods under the unified lrs backend interface [BC]
• Backends: Update statementId and voidedStatementId to snake_case,
  with camelCase alias, in LRSStatementsQuery
• Backends: Replace reference to a JSON column in ClickHouse with
  function calls on the String column [BC]
• CLI: User credentials must now include an "agent" field which can be created
  using the cli
• CLI: Change push to write and fetch to read [BC]
• CLI: Change -c --chunk-size option to -s --chunk-size [BC]
• CLI: Change websocket backend name -b ws to -b async_ws along with it's
  uri option --ws-uri to --async-ws-uri [BC]
• CLI: List cli usage strings in alphabetical order
• CLI: Change backend configuration environment variable prefixes from
  RALPH_BACKENDS__{{DATABASE|HTTP|STORAGE|STREAM}}__{{BACKEND}}__{{OPTION}}
  to RALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}
• Models: The xAPI context.contextActivities.category field is now mandatory
  in the video and virtual classroom profiles. [BC]
• Upgrade base python version to 3.12 for the development stack and Docker
  image
• Upgrade bcrypt to 4.1.2
• Upgrade cachetools to 5.3.2
• Upgrade fastapi to 0.105.0
• Upgrade sentry_sdk to 1.39.1
• Upgrade uvicorn to 0.24.0.post1

Fixed

• API: Fix a typo ('attachements' -> 'attachments') to ensure compliance with
  the LRS specification and prevent potential silent bugs

Removed

• Project: Drop support for Python 3.7
• Models: Remove school, course, module context extensions in Edx to xAPI
  base converter
• Models: Remove name field in VideoActivity xAPI model mistakenly used in
  video profile
• CLI: Remove DEFAULT_BACKEND_CHUNK_SIZE environment variable configuration

Added

• Allow to ignore health check routes for Sentry transactions

Changed

• Upgrade sentry_sdk to 1.22.2
• Upgrade uvicorn to 0.22.0
• LRS /statements GET method returns a code 400 with certain parameters as per the xAPI specification
• Use batch/v1 api in cronjob_pipeline manifest
• Use autoscaling/v2 in HorizontalPodAutoscaler manifest

Fixed

• Fix the more IRL building in LRS /statements GET requests

Changed

• Upgrade httpx to 0.24.0
• Upgrade fastapi to 0.95.1
• Upgrade sentry_sdk to 1.19.1
• Upgrade uvicorn to 0.21.1

Fixed

• An issue with starting Ralph in pre-built Docker containers
• Fix double quoting in ClickHouse backend server parameters
• An issue Ralph starting when ClickHouse is down

Added

• Implement PUT verb on statements endpoint
• Add ClickHouse database backend support

Changed

• Make trailing slashes optional on statements endpoint
• Upgrade sentry_sdk to 1.16.0

Changed

• Upgrade fastapi to 0.92.0
• Upgrade sentry_sdk to 1.15.0

Fixed

• Restore sentry integration in the LRS server

Added

• Restored python 3.7+ support for library usage (models)

Changed

• Allow xAPI extra fields in extensions fields

Changed

• Relax required Python version to 3.7+

Added

• Add a new auth subcommand to generate required credentials file for the LRS
• Add an official Helm Chart (experimental)
• Implement support for AWS S3 storage backend
• Add CLI --version option

Changed

• Upgrade fastapi to 0.89.1
• Upgrade httpx to 0.23.3
• Upgrade sentry_sdk to 1.13.0
• Upgrade uvicorn to 0.20.0
• Tray: add the ca_certs path for the ES backend client option (LRS)
• Improve Sentry integration for the LRS
• Update handbook link to https://handbook.openfun.fr
• Upgrade base python version to 3.11 for the development stack and Docker
  image

Fixed

• Restore ES and Mongo backends ability to use client options

Added

• EdX to xAPI converters for video events

Changed

• Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
• Upgrade fastapi to 0.87.0

Added

• Implement edx video browser events pydantic models
• Create a post endpoint for statements implementing the LRS spec
• Implement support for the MongoDB database backend
• Implement support for custom queries when using database backends get method (used in the fetch command)
• Add dotenv configuration file support and python-dotenv dependency
• Add host and port options for the runserver cli command
• Add support for database selection when running the Ralph LRS server
• Implement support for xAPI statement forwarding
• Add database backends status checking
• Add health LRS router
• Tray: add LRS server support

Changed

• Migrate to python-legacy handler for mkdocstrings package
• Upgrade click to 8.1.3
• Upgrade elasticsearch to 8.3.3
• Upgrade fastapi to 0.79.1
• Upgrade ovh to 1.0.0
• Upgrade pydantic to 1.9.2
• Upgrade pymongo to 4.2.0
• Upgrade python-keystoneclient to 5.0.0
• Upgrade python-swiftclient to 4.0.1
• Upgrade requests to 2.28.1
• Upgrade sentry_sdk to 1.9.5
• Upgrade uvicorn to 0.18.2
• Upgrade websockets to 10.3
• Make backends yield results instead of writing to standard streams (BC)
• Use pydantic settings management instead of global variables in defaults.py
• Rename backend and parser parameter environment variables (BC)
• Make project dependencies management more modular for library usage

Removed

• Remove YAML configuration file support and pyyaml dependency (BC)

Fixed

• Tray: do not create a cronjobs list when no cronjob has been defined
• Restore history mixin logger

Added

• Implement edx problem interaction events pydantic models
• Implement edx textbook interaction events pydantic models
• ws websocket stream backend (compatible with the fetch command)
• bundle jq, curl and wget in the fundocker/ralph Docker image
• Tray: enable ralph app deployment command configuration
• Add a runserver command with basic auth and a whoami route
• Create a get endpoint for statements implementing the LRS spec

Changed

• Upgrade uvicorn to 0.17.4
• Upgrade elasticsearch to 7.17.0
• Upgrade sentry_sdk to 1.5.5
• Upgrade fastapi to 0.73.0
• Upgrade pyparsing to 3.0.7
• Upgrade pydantic to 1.9.0
• Upgrade python-keystoneclient to 4.4.0
• Upgrade python-swiftclient to 3.13.0
• Upgrade pyyaml to 6.0
• Upgrade requests to 2.27.1
• Upgrade websockets to 10.1

Changed

• Upgrade elasticsearch to 7.13.3

Fixed

• Restore elasticsearch backend datastream compatibility for bulk operations

Added

• xAPI video interacted pydantic models
• xAPI video terminated pydantic models
• xAPI video completed pydantic models
• xAPI video seeked pydantic models
• xAPI video initialized pydantic models
• xAPI video paused pydantic models
• convert command to transform edx events to xAPI format
• EdX to xAPI converters for page viewed andpage_close events
• Implement core event format converter
• xAPI video played pydantic models
• xAPI page viewed and page terminated pydantic models
• Implement edx navigational events pydantic models
• Implement edx enrollment events pydantic models
• Install security updates in project Docker images
• Model selector to retrieve associated pydantic model of a given event
• validate command to lint edx events using pydantic models
• Support all available bulk operation types for the elasticsearch backend
  (create, index, update, delete) using the --es-op-type option

Changed

• Upgrade elasticsearch to 7.13.2
• Upgrade python-swiftclient to 3.12.0
• Upgrade click to 8.0.1
• Upgrade click-option-group to 0.5.3
• Upgrade pydantic to 1.8.2
• Upgrade sentry_sdk to 1.1.0
• Rename edx models
• Migrate model tests from factories to hypothesis strategies
• Tray: switch from openshift to k8s (BC)
• Tray: remove useless deployment probes

Fixed

• Tray: remove version immutable field in DC selector

Added

• EdX server event pydantic model and factory
• EdX page_close browser event pydantic model and factory
• Tray: allow to specify a self-generated elasticsearch cluster CA certificate

Fixed

• Tray: add missing Swift variables in the secret
• Tray: fix pods anti-affinity selector

Removed

• pandas is no longer required

Added

• Support for Swift storage backend
• Use the push command --ignore-errors option to ignore ES bulk import errors
• The elasticsearch backend now accepts passing all supported client options

Changed

• Remove click_log package dependency
• Upgrade pyyaml to 5.4.1
• Upgrade pandas to 1.2.1

Added

• Implement base CLI commands (list, extract, fetch & push) for supported backends
• Support for ElasticSearch database backend
• Support for LDP storage backend
• Support for FS storage backend
• Parse (gzipped) tracking logs in GELF format
• Support for application's configuration file
• Add optional sentry integration
• Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs