Transforms parsed JSON objects into a uniform data structure
APACHE-2.0 License
When parsing JSON on client-side, the structure of it attracts most of our attention. If the structure evolves over time, it leads to recurring changes in the code that depends on it.
Use the following command to install the library using npm:
npm install data-restructor
Alternatively, the sources can be found inside the source folder:
The development artifacts (not minified) can be found inside the devdist folder:
Here are some code examples on how these modules can be imported:
var template_resolver = template_resolver || require("data-restructor/devdist/templateResolver"); // supports vanilla js
var described_field = described_field || require("data-restructor/devdist/describedfield"); // supports vanilla js
var datarestructor = datarestructor || require("data-restructor/devdist/datarestructor"); // supports vanilla js
The built (minified) versions can be found inside the distribution folder:
The code documentation is generated using JSDoc and is published using GitHub Pages at https://joht.github.io/data-restructor-js.
Use the following commands to build and package the module. A list of all commands can be found in COMMANDS.md.
npm install merger-js -g
npm install
npm run package
Note: merger.js prompts to select a source file. Please select "ALL" using the arrow keys and press enter to continue.
As a starting point you may have a look at the following example. A running, comprehensive example can be found here: DataRestructorUseCaseTest.js
{
"responses": [
{
"hits": {
"total": {
"value": 1
},
"hits": [
{
"_source": {
"iban": "AT424321012345678901",
"accountnumber": "12345678901",
"customernumber": "00001234567",
"currency": "USD",
"tags": [
"active",
"online"
]
}
}
]
}
}
]
}
function restructureJson(jsonData) {
var allDescriptions = [];
allDescriptions.push(summariesDescription());
allDescriptions.push(detailsDescription());
return new datarestructor.Transform(allDescriptions).processJson(jsonData);
}
function summariesDescription() {
return new datarestructor.PropertyStructureDescriptionBuilder()
.type("summary")
.category("account")
.propertyPatternEqualMode()
.propertyPattern("responses.hits.hits._source.accountnumber")
.groupName("summaries")
.groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
.build();
}
function detailsDescription() {
return new datarestructor.PropertyStructureDescriptionBuilder()
.type("detail")
.category("account")
.propertyPatternTemplateMode()
.propertyPattern("responses.hits.hits._source.{{fieldName}}")
.groupName("details")
.groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
.groupDestinationPattern("account--summary--{{index[0]}}--{{index[1]}}")
.build();
}
An Javascript object with mainly this structure (see DescribedEntry for more details) and content is returned, when the function restructureJson
from above is called:
category: "account"
displayName: "Accountnumber"
fieldName: "accountnumber"
type: "summary"
value: "12345678901"
details:
- category: "account"
type: "detail"
displayName: "Iban"
fieldName: "iban"
value: "AT424321012345678901"
- category: "account"
type: "detail"
displayName: "Accountnumber"
fieldName: "accountnumber"
value: "12345678901"
- category: "Konto"
type: "detail"
displayName: "Customernumber"
fieldName: "customernumber"
value: "00001234567"
- category: "Konto"
type: "detail"
displayName: "Currency"
fieldName: "currency"
value: "USD"
- category: "Konto"
type: "detail"
displayName: "Tags"
fieldName: "tags"
value: "active"
- category: "Konto"
type: "detail"
displayName: "Tags"
fieldName: "tags"
value: "online"
- category: "Konto"
type: "detail"
displayName: "Tags"
fieldName: "tags_comma_separated_values"
value: "active, online"
The input data object, e.g. parsed from JSON, is converted to an array of point separated property names and their values. For example this structure...
{
"responses": [
{
"hits": {
"total": {
"value": 1
},
"hits": [
{
"_source": {
"accountnumber": "123"
}
}
]
}
}
]
}
...is flattened to...
responses[0].hits.total.value=1
responses[0].hits.hits[0]._source.accountnumber=123
To make it easier to e.g. display array values like tags, an additional property is added that combines the array values to a single property, that contains the values in a comma separated way. This newly created property gets the name of the array property followed by "_comma_separated_values" and is inserted right after the single array values.
For example these lines...
responses[0].hits.total.value=1
responses[0].hits.hits[0]._source.tags[0]=active
responses[0].hits.hits[0]._source.tags[1]=online
...will lead to an additional property that looks like this...
responses[0].hits.hits[0]._source.tags_comma_separated_values=active, online
For every given description, all properties are searched for matches. If a description matches a property, the description gets attached to it. This can be used to categorize and filter properties. The description builder accepts these ways to configure property matching:
Equal Mode (default):
The property name needs to match the described pattern exactly. It is not needed to set equal mode.
The field name will be (by default) taken from the right most (after the last separator .
) element of the property name.
In the example below the field name will be "accountnumber".
Example:
new datarestructor.PropertyStructureDescriptionBuilder()
.propertyPatternEqualMode()
.propertyPattern("responses.hits.hits._source.accountnumber")
...
Pattern Mode:
The property name needs to start with the described pattern.
The pattern may contain variables inside double curly brackets.
The variable {{fieldName}}
is a special case which describes from where the field name should be taken.
If {{fieldName}}
is not specified, the field name will be taken from the right most (after the last separator .
) element of the property name, which is the same behavior as in "Equal Mode".
This mode needs to set using propertyPatternTemplateMode
, since the default mode is propertyPatternEqualMode
.
Example:
new datarestructor.PropertyStructureDescriptionBuilder()
.propertyPatternTemplateMode()
.propertyPattern("responses.hits.hits._source.{{fieldName}}")
...
Index Matching (Optional):
If the source data is structured in an top level array and all property names look pretty much the same
it may be needed to describe data based on the array index.
The index of an property is taken out of its array qualifiers.
For example, the property name responses[0].hits.hits[1]._source.tags[2]
has the index 0.1.2
.
Index Matching can be combined with property name matching.
This example restricts the description to the first top level array:
new datarestructor.PropertyStructureDescriptionBuilder()
.indexStartsWith("0.")
...
To remove duplicate properties or to override properties with other ones when they exist,
a deduplicationPattern
can be defined.
Variables (listed below) are put into double curly brackets and will be replaced with the contents of the description and the matching property.
If there are two entries with the same resolved deduplicationPattern
(=_identifier.deduplicationId
),
the second one will override the first (the first one will be removed).
Example:
new datarestructor.PropertyStructureDescriptionBuilder()
.deduplicationPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}--{{fieldName}}")
...
Since data had been flattened in the step 1., it is structured as a list of property names and their values.
This non-hierarchical structure is ideal to add further properties, attach descriptions and remove duplicates.
After all, a fully flat structure might not be suitable to display overviews/details or to collect options.
The groupName
defines the name of the group attribute (defaults to "group" if not set).
The groupPattern
describes, which properties belong to the same group.
Variables (listed below) are put into double curly brackets and will be replaced with the contents
of the description and the matching property.
The groupPattern
will be resolved to the _identifier.groupId
. Every property, that leads to a
new groupId gets a new attribute named by the groupName
, where this entry and all others of the
same group will be put into. Example:
new datarestructor.PropertyStructureDescriptionBuilder()
.groupName("details")
.groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
...
After grouping in step 5., every property containing a group and the remaining non-grouped properties
are listed one after another. To organize them further, a group can be moved beneath another (destination) group.
The groupDestinationPattern
contains the pattern of the group to where the own group should be moved.
Variables (listed below) are put into double curly brackets and will be replaced with the contents
of the description and the matching property.
Optionally, the groupDestinationName
can be specified to rename the group when it is moved. Default is the value of groupName
.
Example, where the details group is moved to the summary, because the group destination pattern
of the details resolves to the same id as the resolved group pattern of the summary:
var summaryDescription = new datarestructor.PropertyStructureDescriptionBuilder()
.category("account")
.type("summary")
.groupName("summaries")
.groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
...
var detailsDescription = new datarestructor.PropertyStructureDescriptionBuilder()
.groupDestinationPattern("account--summary--{{index[0]}}--{{index[1]}}")
.groupDestinationName("details")
...
The result is finally converted into an array of DescribedDataFields.
This section lists the types and their fields in detail (mostly taken from jsdoc).
Every field can be used as variable in double curly brackets inside pattern properties.
Additionally, single elements of the index can be used by specifying the index position e.g. {{index[0]}}
(first), {{index[1]}}
(second),...
{{fieldname}}
inside the pattern.This is the data structure of a single output element representing a field. Beside the properties described below, the described data field can also contain custom properties containing groups (arrays) of sub fields of type DescribedDataField.
Before version 3.0.0 this structure was named DescribedEntry and also contained internal fields. Since 3.0.0 and above, DescribedEntry is only used internally and is not public any more.
"responses[2].hits.hits[4]._source.name"
will have an index of [2,4].Since version 3.0.0 and above, there are no functions any more.
This helper was added with version 3.0.0. It adds groups to DescribedDataFields. These groups are dynamically added properties that contain an array of sub fields also of type DescribedDataField.
Since 3.0.0 and above, DescribedEntry is only used internally and is not public any more. It is documented here for sake of completeness and for maintenance purposes. See JSDoc for a more comprehensive reference.
An simple template resolver is included and provided as separate module. Here is an example on how to use it:
var template_resolver = require("templateResolver");
var sourceDataObject = {type: "MyType", category: "MyCategory"};
var resolver = new template_resolver.Resolver(sourceDataObject);
var template = "{{type}}-{{category}}";
var resolvedString = resolver.resolveTemplate(template);
//resolvedString will contain "MyType-MyCategory"
"{{fieldName}}"
, "{{displayName}}"
, "{{value}}"
."{{summaries[0].value}}"
."{{index[1]}}"
."{{customernumber}}"
will be replaced by 123
, if the structure contains fieldname="customernumber", value="123"
. This also applies to sub groups, e.g. "{{details.customernumber}}"
will be replaced by 321
, if the structure contains details[4].fieldname="customernumber", details[4].value="321"
.An comprehensive and up to date reference can be found here: TransformConfig JSDoc.
The restructured data is by nature hierarchical and may contain cyclic data references. Fields may contain groups of fields that may contain groups of fields.... Since JSON can't be generated out of objects with cyclic references, sub-structures are expressed by copies. That leads to recursion and duplication, that need to be limited. This can be configured here.
Although this project doesn't use any runtime dependencies, it is created using these great tools: