aquameta

Here's v0.4.1, which has the following changes:

• Refactor bundle's use of NULL in blobs
• Fix issues with leaky search_path in bundle checkout(), commit() etc
• Start of a new more functional approach to bundle internals, WIP
• Various bundle optimizations
• Much progress with tracking schema in bundles
• Rewrite of several meta views for performance, feature coverage and clarity
• Add write triggers for meta.type
• A new experimental "extension_to_bundle()" function that tries to convert an extension to a bundle
• Drop casts from meta-identifiers to URL paths, move that logic into endpoint
• Some IDE bug fixes, still in a state of somewhat disrepair as deep refactors continue

Install instructions are here.

Ho! Now comes version 0.4 of Aquameta.

Changes include:

• flattening of meta-identifiers, deep refactor to no longer use nested types
• resurrection of pgfs, which represents the database as a filesystem
• switch to git submodules for the meta and meta_triggers extensions
• resource_functions, arbitrary URL patterns that pass their args to a pg function and return the results
• events refactor
• much code cleanup, splitting up the Go daemon into multiple files

This release breaks the import and export of old bundles, due to the refactor of meta-identifiers. Old bundles can be converted to the new .csv format, as described here.

Full Changelog: https://github.com/aquametalabs/aquameta/compare/v0.3.1...v0.4

Last commit before type flattening merge, deep breaking change, meta issue #1.

Notes:

• Flattening meta-types breaks all old bundles, they need to be exported and re-imported through a magical process that is incomprehensible.
• The .tar.gz source release does not recurse into git sub-modules (references to external repositories in git). Don't us it, clone master and check out this tag insteed, recursing submodules.

Here's Aquameta 0.3! Major changes include:

• New server written in Go handles the HTTP server and connection to the database. This eliminates dependencies on plv8, uwsgi, messy install scripts, etc. Now Aquameta is a) a PostgreSQL database and b) a Go binary.
• Switch to using git submodules for extensions, and break out the meta extension into it's own submodule/git repo, with plans of doing it for other reusable extensions as well. This project is too damn big and diverse to be one repo and there are lots of parts that are useful just by themselves.
• Endpoint can serve a new type of resource, endpoint.resource_function maps URLs to database functions
• Events work again! Now you can pass { events: true} to a database instantiation in datum.js and then get change events over WebSocket. Still in alpha but the pattern is there.
• Bundle merge. Take two commits that share a common ancestor, and merge them together. Mostly feature complete, fields change in both databases (similar to git in a merge state) still need some work.
• Bundle is 74x faster, did some cursory performance optimizations and found some very low hanging fruit
• Start of a "bootloader", an embedded PostgreSQL instance that runs before Aquameta is installed, and whose purpose is to present a web-based GUI where people can configure and manage their configuration files. Eyes on GUI install via .dmg, AppImage, .exe etc.
• Tons of general cleanup and clarification of the architecture.

Without fanfare I'm releasing v0.2.0 of Aquameta. The code base is in a relatively stable state, and I'm about to dramatically destabilize it. The UI has really come a long way and the architecture has evolved and been refined a lot. I LOVE building projects in this environment, even though is still so many edge cases that have to be done from the psql promt. No looking back, development belongs in the database and the wins to be found by moving it there keep compounding.

There is a long list of architectural changes that need to be made though, and now is the time to make them. Also finding Go has been a game changer, opening up the door for a simple distribution and deployment pattern of a single self-contained binary with no dependencies (I think?).

The foundations of the project are about to shake. So, here's 0.2.

This release adds many new features including ability to do an "online" install from a central bundle repository called the "hub".

Here's rc5 of Aquameta 0.2! Change include:

• many improvements to the IDE
• improved bundle code search functionality
• unified installer for Debian and Ubuntu
• widget loading optimizations
• remote commit push/pull support
• remote mount(), is_mounted(), unmount() functions
• add ability to install from remote hub
• start of new documentation system
• bug fixes and enhancements

Aquameta 0.2.0-rc4 released

What new in rc4?

• major strides towards version controlling meta-entities
• clarification of meta.table vs meta.relation and their columns
• meta is no longer version-controlled by default, rather a bundle.trackable_nontable_relation table has been created, for tracking non-table relations like views, foreign tables, etc. opt in.
• starting to lean on the meta.*_definition pattern, def views are views that contain meta entities like views, tables, casts, operators, etc., but with only two columns their, say, view_id, and another column called definition, which contains the SQL stmt which creates it. Much better for version controlling meta entities.
• much exploration around server-side rendering of widgets resulted in the addition of endpoint.template and endpoint.template_route which are rendered server-side with plv8 and doT.js, so we can have dynamic results on a base resource (or anything else)
• baby steps on documentation
• refinement of extensions so that pg_dump will export their data (which must be explicitly declared per-relation)
• all extensions' make command now overwrites previously generated files
• big steps forward on a better UI, including file uploads, untracked rows manager, template creation ui, a much better stage, pulling in the semantics ui css library for great beautification, bundle search, filesystem import and export, remote bundle clone and diff, the start of some p2p webrtc features
• two new security schemes, one for total lockdown (no register, access to nothing except auth pages), and one for open registration, which allows public to register for an account.
• complete refactor of installer to support both ubuntu 18 and debian 9, prompt for superuser setup, prompt for security scheme, and generally works much better
• bin/ directory containing scripts to restart the three aquameta services (pg, nginx, uwsgi), and to tail all logs of said services
• include thepg_cron extension from Citus, which enables scheduled execution of arbitrary sql statements
• include plv8 extension (binaries, since apt version is older) which does our template rendering and also allows javascript/ecma procedures to be executed server-side
• numerous bug fixes and enhancements

We're still a ways away from a complete user experience, but things seem to be in a relatively sane state at the moment and it's been a while since I did a release, so here it is.

Ho! Now comes version 0.2.0-rc3 of Aquameta.

Numerous enhancements including:

• All Aquameta core modules are now PostgreSQL EXTENSIONs
• Support for server-side templates that are dynamically rendered using plv8
• Ground-up security audit
• Installer now supports two security schemes, private and open-registration. Private databases do not accept new users, but open-registration databases allow anonymous users to sign up
• Refactor of all auth pages, including /register and /login.
• No more hard-coded installation directory, instead user chooses where to install
• Remove Docker support (until, or if ever, we convert it to docker-compose)
• Bundles can now be diff-ed with another remote copy of the bundle, showing commits that are unique to either the local or remote bundle
• Installation has been vastly simplified, prompting user for things like installation directory, security scheme, and superuser credentials
• Events have been resurrected to be much closer to working
• Complete refactor of code and extension filesystem layout, moving all postgresql extensions and python packages into the src/ directory
• Switch to using PostgreSQL version 10
• Numerous bug fixes and misc. enhancements

Ho! Now comes Aquameta v0.2.0 release candidate 2. Lots of new features to talk about:

• bundle remotes have been completely refactored to use postgres_fdw instead of http. This makes the process of cloning, pushing and pulling between instances of Aquameta much simpler. A new user interface is taking shape called the "bundle manager", which enables pushing and cloning between connected databases.
• bundles can now be exported to csv file with the bundle.bundle_export_csv() function. All parts of a bundle are saved to file, and then can be imported via bundle.bundle_import_csv().
• major strides in making Aquameta secure out of the box, most noteworthy of which is that anonymous is no longer a superuser. Instead a limited set of row-level permissions are granted to the anonymous user, allowing them to only do what is needed to register, confirm and login, or at least that's the idea.
• There is an alternate uwsgi config file that enables SSL encryption via letsencrypt. Users should be able to just follow the letsencrypt instructions, run certbot and create an SSL certificate.
• The endpoint.resource* tables now have an active flag instead of a unique constraint on resource.path, so that multiple resources that use the same path can be activated and deactivated instead of just breaking their checkout.
• Documentation! There's now a quickstart and cheatsheet in the docs/ folder.
• The very simple email facilities in endpoint for sending registration emails etc., have been moved into an email extension. It has a configurable smtp_server table that can contain multiple SMTP configurations. It works very nicely with an Amazon SES instance, or any other SMTP server that supports DKIM, so emails don't get sent to the spam folder. It also has the start of a simple email template system.
• Now each core modules has it's own associated bundle, which contains rows specific to that module:
  • org.aquameta.core.bundle
  • org.aquameta.core.docs
  • org.aquameta.core.event
  • org.aquameta.core.filesystem
  • org.aquameta.core.http_client
  • org.aquameta.core.ide
  • org.aquameta.core.meta
  • org.aquameta.core.p2p
  • org.aquameta.core.semantics
  • org.aquameta.core.widget
  • org.aquameta.core.www
• Lots of user interface bundles in early stages:
  • org.aquameta.ui.admin - Generic data editing admin, with mixin widgets for building apps
  • org.aquameta.ui.auth - Register, login and auth status widgets for general use or extension
  • org.aquameta.ui.dev - Main development UI, still very primitive
  • org.aquameta.ui.fsm - UI state fsm
  • org.aquameta.ui.layout - Tabs, modals, select panes, etc.
  • org.aquameta.ui.tags - HTML tag wrappers for sync'ing datums into tags
• Lots of bug fixes
• Instructions for installing on Amazon EC2

Hi all,

I'm happy to announce the release of Aquameta, a peer-to-peer network for exchanging "bundles", a new kind of complex data object that can be almost anything. They can include code, web apps, data, images and more, and together create a kind of "smart object" that is very versatile and open-ended.

Today, users can exchange emails, publish web pages, exchange binaries via p2p apps, and much more. We'd like to see Aquameta become another kind of communication protocol and network, one where users exchange these bundled complex data objects that hopefully do fun, interesting and useful stuff. Because bundles are made up of data, we claim that it's a very wide pipe for creativity and interactivity, one that opens up a world of new possibilities.

Under the hood, Aquameta is a bunch of PostgreSQL schemas that handle things like version control, web hosting, user interface, events, peer to peer communication, etc. A bundle is made up of rows in these tables and any others that users might wish to create.

Aquameta has a web interface that lets users build and exchange these bundles. They can be sent either server-to-server via http, or browser-to-browser via WebRTC. The networked-communication aspect of Aquameta is still under development and experimentation.

Aquameta can be installed on a desktop, a server, in the cloud or on a small device such as a Raspberry Pi.

Under the hood are a lot of PostgreSQL modules which hopefully can be useful to the PostgreSQL community. They include:

1. meta - a writable system catalog, that makes common DDL operations like schema changes accessible via INSERT, UPDATE and DELETE

insert into meta.role(name, superuser, password) values ('joe', true, 'mypass');
insert into meta.schema(name) values ('my_schema');

2. bundle - a version control system for data, similar to git, but for rows in the database instead of files

select bundle.stage_row_add(
    meta.row_id('public','countries','id','UK')
);
select bundle.commit('org.aquameta.countries','add uk.');
select bundle.checkout('org.aquameta.games.snake');

3. event - a pub/sub event system for subscribing to data changes on a particular column, row or table

select session_create();
select subscribe_table(meta.relation_id('widget','machine'));

4. filesystem - a foreign data wrapper that lets you SELECT files and directories from the filesystem, and a daemon called postgrefs, which lets you navigate the database from the filesystem as if it were files and directories

aquameta=# select name, path, permissions, content, owner from filesystem.file where path='/etc/issue';
 name  |    path    | permissions |         content          | owner 
-------+------------+-------------+--------------------------+-------
 issue | /etc/issue | 0644        | Ubuntu 14.04.5 LTS \n \l+| root
       |            |             |                         +| 
       |            |             |                          | 
(1 row)
$ ls /mnt/aquameta/public/countries

5. www - a web server that exposes the database over http, including a REST interface for data manip and function calls, hosting of arbitrary resources, and an authentication layer that uses PostgreSQL ROLEs and PRIVILEGEs

aquameta=# select path, m.mimetype, substring(content from 1 for 15) 
from endpoint.resource r 
join endpoint.mimetype m on r.mimetype_id = m.id limit 2;
 path  | mimetype  |    substring    
-------+-----------+-----------------
 /docs | text/html | <!DOCTYPE html>
 /dev  | text/html | <!DOCTYPE html>
select endpoint.request('0.1','GET','/docs')

6. widget - a user interface component framework, where web pages are composed of "widgets", UI components made of html, css and javascript, that accept arguments and support javascript dependencies

select * from widget.widget;
select * from widget.widget_dependency_js;

7. semantics - a metadata layer for the database schema, where columns, relations, foreign keys and more can be "decorated" with additional semantic meaning

insert into semantics.relation (relation_id, purpose_id, widget_id) values
(
   meta.relation_id('ecomm','customer'),
   (select purpose_id from semantics.purpose where purpose='row_detail'),
   (select id from widget.widget where name='customer_list_view')
);

Aquameta is in very early stages, despite being under development for years. It is finally at the point of being a marginally functional prototype that hopefully is pointed in the right direction, but there is much work still to be done. The user interface is just beginning to take shape, the system is not secure, test coverage is sparse, and there are surely many bugs and missing features. We are releasing it at this early stage to get feedback and invite contributors to help out.

To give Aquameta a spin, head over to github and consult the README which contains install instructions:

http://github.com/aquametalabs/aquameta

To jump into development, check out the project's roadmap, and join our IRC channel and mailing list:

http://aquameta.org/roadmap
#aquameta on irc.freenode.net
https://groups.google.com/forum/#!aboutgroup/aquameta-discuss