udata

Breaking changes

• Full metrics refactoring 2459:
  • Metric collection is now useless and will not be filled anymore, you can remove it or keep it for archival sake. It will not be automatically removed.
  • udata-piwik now uses InfluxDB as a buffer for trafic data before injecting them into udata's models.
  • Most of celery's tasks related to metrics are removed, this should help performance-wise on a big instance.
  • Charts related to metrics are removed from admin and dashboard panel until we have accurate data to populate them.
  • Site's metrics computation are not triggered by signals anymore.
  • A specific celery job needs to be run periodically to compute site's metrics.

• Fix export-csv command (py3 compat) #2472

• ⚠️ Security fix: fix XSS in markdown w/ length JS filter #2471

• ⚠️ Breaking change / security fix: disallow html tags in markdown-it (JS markdown rendering) #2465

• Allow images to be displayed in markdown by default 2462
• Fix deleted user's authentication on backend side 2460

Breaking changes

• Migration to Python 3.7 #1766
• The new migration system (#1956) uses a new python based format. Pre-2.0 migrations are not compatible so you might need to upgrade to the latest udata version <2.0.0, execute migrations and then upgrade to udata 2+.
• The targeted mongo version is now Mongo 3.6. Backward support is not guaranteed
• Deprecated celery tasks have been removed, please ensure all old-style tasks (pre 1.6.20) have been consumed before migrating #2452

New features

• New migration system #1956:
  • Use python based migrations instead of relying on mongo internal and deprecated js_exec
  • Handle rollback (optionnal)
  • Detailled history
• Template hooks generalization: allows to dynamically extend template with widgets and snippets from extensions. See the dedicated documentation section #2323
• Markdown now supports Github Flavored Markdown (GFM) specs (ie. the already supported CommonMark specs plus tables, strikethrough, autolinks support and predefined disallowed raw HTML) #2341

• New Crowdin translations #2360
• Fix territory routing for @latest #2447
• Refactor Celery: py2/py3 compatibility, use ids as payload #2305
• Automatically archive dangling harvested datasets ⚠️ this is enabled by default #2368
• Refactor celery tasks to avoid models/documents in the transport layer #2305

• rel=nofollow on remote source links #2364
• Fix admin messages and fix user roles selector default value #2365
• Fix new harvester's form tooltip showup #2371
• Fix responsive design of search results #2372
• Fix non-unique ids in datasets' comments #2374
• Case insensitive license matching #2378

• Remove embedded API doc #2343 ⚠️ Breaking change, please customize API_DOC_EXTERNAL_LINK for your needs.
• Removed published date from community ressources #2350
• Added new size for avatars in user's model (udata images render must be run in order to update the size of existing images) #2353
• Fixed user's avatar change #2351
• Removed dead code #2355
• Resolved conflict between id and slug #2356
• Fix next link in posts pagination #2358
• Fix organization's members roles translation #2359

• Disallow URLs in first and last names #2345

• Prevent Google ranking spam attacks on reuse pages (rel=nofollow on reuse link) #2320
• Display admin resources list actions only if user has permissions to edit #2326
• Fix non-admin user not being able to change their profile picture #2327

• Style links in admin modals #2292
• Add activity.key filter to activity.atom feed #2293
• Allow Authorization as CORS header and OAuth minor fixes #2298
• Set dataset.private to False by default (and fix stock) #2307
• Fixes some inconsistencies between admin display (buttons, actions...) and real permissions #2308

• Cleanup permitted_reuses data (migration) #2244
• Proper form errors handling on nested fields #2246
• JS models load/save/update consistency (loading always true on query, always handle error, no more silent errors) #2247
• Ensures that date ranges are always positive (ie. start < end) #2253
• Enable completion on the "MIME type" resource form field (needs reindexing) #2238
• Ensure oembed rendering errors are not hidden by default error handlers and have cors headers #2254
• Handle dates before 1900 during indexing #2256
• spatial load command is more resilient: make use of a temporary collection when --drop option is provided (avoid downtime during the load), in case of exception or keybord interrupt, temporary files and collections are cleaned up #2261
• Configurable Elasticsearch timeouts. Introduce ELASTICSEARCH_TIMEOUT as default/read timeout and ELASTICSEARCH_INDEX_TIMEOUT as indexing/write timeout #2265
• OEmbed support for organizations #2273
• Extract search parameters as settings allowing fine tuning search without repackaging udata (see the Search configuration documentation) #2275
• Prevent DoesNotExist error in activity API: silence the error for the consumer but log it (ie. visible in Sentry) #2268
• Optimize CSV export generation memory wise #2277

• Rename og:image target ⚠️ this will break your custom theme, please rename your logo image file to logo-social.png instead of logo-600x600.png #2217
• Don't automatically overwrite last_update field if manually set #2020
• Spatial completion: only index last version of each zone and prevent completion cluttering #2140
• Init: prompt to loads countries #2140
• Handle UTF-8 filenames in spatial load_logos command #2223
• Display the datasets, reuses and harvesters deleted state on listing when possible #2228
• Fix queryless (no q text parameter) search results scoring (or lack of scoring) #2231
• Miscellaneous fixes on completers #2215
• Ensure filetype='remote' is set when using the manual ressource form #2236
• Improve harvest sources listing (limit last_job fetched and serialized fields, reduce payload) #2214
• Ensure HarvestItems are cleaned up on dataset deletion #2214
• Added config.HARVEST_JOBS_RETENTION_DAYS and a harvest-purge-jobs job to apply it #2214 (migration). Warning, the migration will enforce config.HARVEST_JOBS_RETENTION_DAYS and can take some time on a big HarvestJob collection
• Drop no_dereference on indexing to avoid the "dictionary changed size during iteration" error until another solution is found. Warning: this might result in more resources consumption while indexing #2237

• Archive dataset feature #2172
• Refactor breadcrum includes #2173
• Better dependencies management #2182 and #2172/install.pip
• Reduce following to staring #2192
• Simplify display of spatial coverage in search results #2192
• Add cache for organization and topic display pages #2194
• Dataset of datasets: id as ref instead of slug #2195 ⚠️ this introduces some settings changes, cf documentation for EXPORT_CSV.
• Add meta og:type: make twitter cards work #2196
• Fix UI responsiveness #2199
• Remove social media sharing feature #2200
• Quick fix for activity.atom #2203
• Remove diff from js dependencies to fix CVE #2204
• Replace default sort label for better readability #2206
• Add a condition to up-to-dateness of a dataset #2208
• Prevent deleted harvesters from running until purged. Harvest jobs history is deleted too on purge. #2209
• Better quality.frequency management #2211

• Center incomplete rows of cards #2162
• Allow .dxf upload #2164
• Always use remote_url as harvesting source #2165
• Update jquery to ~3.4.1 #2161
• Fix various issues with search result page #2166
• Restore non broken facet includes #2169

• Remove <br> in badge display #2156
• Display user avatar and fix its sizing #2157
• Redirect unfiltered csv exports to dataset of datasets #2158
• Show organization id in a modal and add hyperlinks to ids in detail modal #2159

• Add user slug to dataset cache key #2146
• Change display of cards of reuses on topic pages #2148
• Display remote source of harvested dataset #2150
• Prefill community resource type on upload form #2151
• Fix user profile UI #2152
• Remove concept of permitted reuse #2153

• Configurable search autocomplete #2138

• Refactor DCAT harvesting to store only one graph (and prevent MongoDB document size overflow) #2096
• Expose sane defaults for TRACKING_BLACKLIST #2098
• Bubble up uploader errors #2102
• Ensure udata worker status --munin always outputs zero values so munin won't see it has a "no data" response #2103
• Metrics tuning: breaks circular dependencies, drop exec_js/eval usage, proper logging... #2113
• Change reuse icon from "retweet" to "recycle" #2122
• Admins can delete a single comment in a discussion thread #2087
• Add cache directives to dataset display blocks #2129
• Export multiple models objects to CSV (dataset of datasets) #2124