cobrix

What's Changed

• #697 Improve metadata merging method in Spark Utils in https://github.com/AbsaOSS/cobrix/pull/698

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.7.3...v2.7.4

What's Changed

• #678 Add the ability to generate Spark schema based on strict integral precision in https://github.com/AbsaOSS/cobrix/pull/691
• #689 Add support for '_' for hierarchical key generation at leaf level in https://github.com/AbsaOSS/cobrix/pull/692

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.7.2...v2.7.3

What's Changed

• #684 Fix incorrect reader used when segment id generation is turned on in https://github.com/AbsaOSS/cobrix/pull/686
• #685 Add methods for unstructing schemas and dataframes in https://github.com/AbsaOSS/cobrix/pull/687

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.7.1...v2.7.2

What's Changed

• #678 Add a method in SparkUtils to convert integral types to decimal types according to PIC of the fields in https://github.com/AbsaOSS/cobrix/pull/681
• #680 Shade ANTLR for cobol-parser so that it won't clash with Spark's ANTLR runtime in https://github.com/AbsaOSS/cobrix/pull/682

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.7.0...v2.7.1

What's Changed

• #669 Allow 'V' to be at the end of scaled PICs in https://github.com/AbsaOSS/cobrix/pull/671
• #672 Implement parsing of copybooks with spark-cobol options in https://github.com/AbsaOSS/cobrix/pull/673
• #666 Add support for record length value mapping in https://github.com/AbsaOSS/cobrix/pull/674
• #674 Extend the usage of indexes for variable record length files with a record length field in https://github.com/AbsaOSS/cobrix/pull/674

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.11...v2.7.0

What's Changed

• #659 Fix record length option when record id generation is turned on in https://github.com/AbsaOSS/cobrix/pull/660

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.10...v2.6.11

What's Changed

• #653 dd support for new EBCDIC code pages: 273, 500, 1140, 1141, 1148 in https://github.com/AbsaOSS/cobrix/pull/654

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.9...v2.6.10

What's Changed

• #634 Retain metadata on schema flattening. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/635
• #644 Add support for Spark 3.5.0 by @yruslan in https://github.com/AbsaOSS/cobrix/pull/645

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.8...v2.6.9

What's Changed

• #624 Add support for binary fields by @yruslan in https://github.com/AbsaOSS/cobrix/pull/626

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.7...v2.6.8

What's Changed

• #620 Fixed backward compatibility of custom record parsers. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/621

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.6...v2.6.7

What's Changed

• #601 Fixed file offset options when used with a file having VB record format. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/605
• #614 Fixed index build stability record extractors by @yruslan in https://github.com/AbsaOSS/cobrix/pull/616
• #602 Added GitHub action for code-coverage check by @miroslavpojer in https://github.com/AbsaOSS/cobrix/pull/603
• #607 Added minimum and maximum record length option. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/609
• #613 Added the header stream to record extractors by @yruslan in https://github.com/AbsaOSS/cobrix/pull/617

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.5...v2.6.6

What's Changed

• #590 Allow more options controlling metadata generation by @yruslan in https://github.com/AbsaOSS/cobrix/pull/592
• #593 Add the ability to generate 'Record_Bytes' by @yruslan in https://github.com/AbsaOSS/cobrix/pull/594
• #539 Fix the mapping from EBCDIC CP00300 to Unicode by @BenceBenedek and @yruslan in https://github.com/AbsaOSS/cobrix/pull/596
• #539 Add support for cp1364 and cp1388. by @BenceBenedek and @yruslan in https://github.com/AbsaOSS/cobrix/pull/597

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.4...v2.6.5

What's Changed

• #576 Add support for reading data from RDDs by @yruslan in https://github.com/AbsaOSS/cobrix/pull/579
• #574 Add support for reading data with fields having multiple code pages. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/581
• #580 Extend metadata fields with additional fields by @yruslan in https://github.com/AbsaOSS/cobrix/pull/582
• #539 Two byte code pages support by @yruslan in https://github.com/AbsaOSS/cobrix/pull/586

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.3...v2.6.4

What's Changed

• #561 Upgrade ANTLR version to 4.8 by @dz-1 in https://github.com/AbsaOSS/cobrix/pull/562
• #561 Recompile parsers using ANTLR 4.8. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/563
• #567 Add code pages 838, 870, 1025 contributed by @sree018 in https://github.com/AbsaOSS/cobrix/pull/568
• #569 Add an arithmetic expression parser to use for record length expressions by @yruslan in https://github.com/AbsaOSS/cobrix/pull/571
• #572 Improve performance of processing ASCII file with custom charset by @yruslan in https://github.com/AbsaOSS/cobrix/pull/573
• #560 Fix 'segment_filter' for ASCII text files parallelized by Spark. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/575
• #550 Added the option to add additional metadata to the generated Spark schema by @yruslan in https://github.com/AbsaOSS/cobrix/pull/577

New Contributors

• @dz-1 made their first contribution in https://github.com/AbsaOSS/cobrix/pull/562

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.2...v2.6.3

What's Changed

• #537 Found unit test with empty name by @miroslavpojer in https://github.com/AbsaOSS/cobrix/pull/538
• #516 Add support for packed unsigned by @yruslan in https://github.com/AbsaOSS/cobrix/pull/546
• #543 Improve performance of basic ASCII processing. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/548
• #545 Add support for 'string' debug columns for ASCII file format by @yruslan in https://github.com/AbsaOSS/cobrix/pull/549
• #542 Implement filler naming strategies and previous fields based strategy. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/554
• #553 Fix a regression bug introduced in #543 concerning variable OCCURS. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/555
• #544 Added an extension option 'data_paths' that replaces 'paths' for multi path load. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/557
• #556 Fix the option 'file_end_offset' truncates records of each parti… by @yruslan in https://github.com/AbsaOSS/cobrix/pull/558
• Update Scala 2.13 to 2.13.10, Spark 3.2.3 and 3.3.1 for CI and publish builds by @yruslan in https://github.com/AbsaOSS/cobrix/pull/551
• Update Scalatest to 3.2.14. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/552

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.1...v2.6.2

What's Changed

• #531 Add support for CP1047 by @yruslan in https://github.com/AbsaOSS/cobrix/pull/535
• #532 - Add code coverage support - Jacoco by @miroslavpojer in https://github.com/AbsaOSS/cobrix/pull/533
• #529 Fix tests failing on EOL issues on Windows. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/530
• Update sbt version used to build the project. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/536

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.6.0...v2.6.1

What's Changed

• Add support for Scala 2.13 by @yruslan in https://github.com/AbsaOSS/cobrix/pull/515
• #192 Decouple AST transformation logic by @yruslan in https://github.com/AbsaOSS/cobrix/pull/519
• Feature/517 string length metadata by @yruslan in https://github.com/AbsaOSS/cobrix/pull/520
• #521 Fixed index generation for variable block (VB) files. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/523

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.5.1...v2.6.0

What's Changed

• Fix FILLERs being dropped for a FILLER with OCCURS GROUP. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/511

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.5.0...v2.5.1

What's Changed

• #491 Overpunching Strictness made configurable, strict by default by @yruslan in https://github.com/AbsaOSS/cobrix/pull/493
• #423 Add byte length record when generating record ids. by @yruslan in https://github.com/AbsaOSS/cobrix/pull/494
• #396 Add support for copybooks without a root record by @yruslan in https://github.com/AbsaOSS/cobrix/pull/496
• Feature/improve null detection on by @yruslan in https://github.com/AbsaOSS/cobrix/pull/503
• Feature/502 fix signed decimal scaling down pic by @yruslan in https://github.com/AbsaOSS/cobrix/pull/504

Full Changelog: https://github.com/AbsaOSS/cobrix/compare/v2.4.10...v2.5.0

• #481 ASCII control characters are now ignored instead of being replaced with spaces.
  A new string trimming policy (keep_all) allows keeping all control characters in strings (including 0x00).
• #484 Fix parsing of ASCII files so that only full records are parsed. The old behavior
  can be restored with .option("allow_partial_records", "true").