arrow-julia

Arrow v2.7.3

Diff since v2.7.2

Merged pull requests:

• release: Fix "Verify release" job on macOS GitHub Actions runner (#514) (@kou)
• ci: Enable Dependabot for GitHub Actions (#516) (@kou)
• Bump actions/checkout from 3 to 4 (#518) (@dependabot[bot])
• Bump actions/cache from 1 to 4 (#519) (@dependabot[bot])
• Bump julia-actions/setup-julia from 1 to 2 (#520) (@dependabot[bot])
• Bump julia-actions/julia-buildpkg from 1.2 to 1.6 (#521) (@dependabot[bot])
• Bump actions/setup-python from 2 to 5 (#522) (@dependabot[bot])
• Bump version to 2.7.3 (#524) (@quinnj)

Closed issues:

• interoperability with round-tripping through data format broken (x-issue) (#508)
• release: "Verify release" job fails on macOS GitHub Actions runner (#513)
• ci: Enable Dependabot for GitHub Actions (#515)

Arrow v2.7.2

Diff since v2.7.1

Merged pull requests:

• Compatibility of schemas with nested types (#504) (@poncito)

Closed issues:

• Arrow-over-HTTP client and server examples in Julia (#502)
• snappy_jll v1.2.0 lead to Arrow_jll failed to build (#505)

Arrow v2.7.1

Diff since v2.7.0

Merged pull requests:

• Bugfix type instability in length(::Arrow.ToList) (#497) (@JoaoAparicio)

Arrow v2.7.0

Diff since v2.6.2

Merged pull requests:

• cleanup testing (#462) (@baumgold)
• Formatting (#464) (@baumgold)
• Ensure that ArrowTypes.default is defined for Vararg tuples (#466) (@quinnj)
• Fix typo in passing largelists keyword arg (#467) (@quinnj)
• Try to fix docs by publishing to apache infra (#468) (@quinnj)
• Use wkspawn from ConcurrentUtilities instead of Threads.spawn (#469) (@quinnj)
• Fix documentation link (#478) (@guilhermebodin)
• Fix documentation example for Arrow.Writer (#479) (@guilhermebodin)
• Fix example in Arrow.Writer (#480) (@guilhermebodin)
• Docs: add a note about using copy() to get a DataFrame where the columns are regular Vectors (#487) (@DilumAluthge)
• bump (#488) (@JoaoAparicio)
• Use https://arrow.apache.org/julia/ as the official document URL (#490) (@kou)
• Fix docs errors (#491) (@ericphanson)
• enable field-order-agnostic overloads of fromarrow for struct types (#493) (@jrevels)

Closed issues:

• Loss of parametric type information for custom types (#134)
• Look into potential over-copying when copying arrow vector (#190)
• Arrow.write with partition/partitioner results batch unique value error (#192)
• Roundtrippability of special strings (#196)
• Support InlineStrings (#304)
• Invalid argument error (#336)
• ERROR: MethodError: no method matching zero(::Type{Union{Nothing, String}}) when trying to serialize Union{Nothing,String} fields (#368)
• github releases not in sync (#370)
• Typo in kwargs of arrowvector for ListKind (#392)
• Introduce automatic code formatting with JuliaFormatter.toml (#398)
• Issue with Union{Missing, VersionNumber} (#461)
• GitHub Pages build error (#463)
• Use https://arrow.apache.org/julia/ as the official Website URL (#470)
• Arrow.write performance on large DataFrame (#473)
• getindex broken with SVector{3, UInt} in the presence of missing data (#486)
• Removing .arrow files without closing Julia seems impossible in Windows (#492)

Arrow v2.6.2

Diff since v2.6.1

Closed issues:

• Does this pkg provide the plasma In-Memory Object Store? (#37)
• Allow for file appends (#105)
• Invalid arrow file produced when writing partitioned with a dictionary-encoded column? (#129)
• New package release process (#279)
• Need to improve code review/release process and reduce developer productivity friction (#342)
• [Feature Request] Support Tensors ? (#433)
• Error with v2.6.0 (#458)

Merged pull requests:

• implement default for SubArray (#456) (@baumgold)
• Fix case of recursively nested lists (#459) (@quinnj)
• Bump Arrow & ArrowTypes versions (#460) (@quinnj)

Arrow v2.6.1

Diff since v2.6.0

Closed issues:

• Release document misses how to register ArrowTypes to the Julia General Registry (#450)
• Arrow.jl 2.6 breaks Legolas.jl's tests (#452)
• Incorrect syntax in ArrowTypes code (#453)

Merged pull requests:

• release: Add how to register ArrowTypes to the Julia General Registry (#451) (@kou)
• #453 fix typeo introduced in #389 (#454) (@baumgold)
• bump version for new release (#455) (@baumgold)

Arrow v2.6.0

Diff since v2.5.2

Closed issues:

• Support Tables.partitions when reading "arrow file" format in addition to "stream" format (#293)
• Make recursive iteration to get dictionaries more defensive for interop (#375)
• Error/Segfault when writing many partitions (#396)
• Vector{UInt8} mis-represented when writing to disk (#411)
• CI doesn't test with multiple threads (#426)
• Malformed file by Arrow.write on a IOStream created with open(filename, "w") (#432)
• Unhandled sentinel value for len in compression causes invalid Array dimensions (#435)

Merged pull requests:

• Get dictionaries of children only when field.children not nothing (#382) (@okartal)
• fix Base.eltype methods and functions that take Type parameters (#404) (@baumgold)
• enable incremental reads of arrow-formatted files (#408) (@baumgold)
• Base.isdone for Stream (#428) (@baumgold)
• Run with 1 and 2 threads during tests (#431) (@quinnj)
• Add handling of len = -1 in uncompress (#436) (@DrChainsaw)
• Don't treat Vector{UInt8} as Arrow Binary type (#439) (@quinnj)
• Bump BitIntegers compat (#441) (@quinnj)
• Handle len of -1 in "compressed" buffers from other languages (#442) (@quinnj)
• Add Tables.partitions definition for Arrow.Table (#443) (@quinnj)
• Remove scopedenum for EnumX (#444) (@quinnj)
• Refactor compressors/decompressors for laziness + safety (#445) (@quinnj)
• Return SubArrays when possible for arrow list types (#446) (@quinnj)
• bump version of Arrow and ArrowTypes to prepare for new release (#447) (@baumgold)

Arrow v2.5.2

Diff since v2.5.1

Merged pull requests:

• Bump Arrow + ArrowTypes versions (#407) (@baumgold)

Arrow v2.5.1

Diff since v2.5.0

Closed issues:

• Performance regression caused by Arrow type piracy (#266)
• Precompilation broken on Julia 1.9-rc1 (#391)
• Does Arrow.write have an upper limit for the number of columns? (#414)
• [Discussion] Need for early-returning friendly iteration interface (#417)
• Test failure on 1.8.5 Rosetta (#425)
• Test Failure (#429)

Merged pull requests:

• Fix deprecation of getproperty(::Pairs, s) (#409) (@baumgold)
• Use getproperty instead of type parameter to get names (#415) (@simsurace)
• Add @testsets for misc tests (#421) (@JoaoAparicio)
• Pre-allocate buffer (#422) (@JoaoAparicio)
• Bump patch version (#423) (@simsurace)
• add dev CI for monorepo (#424) (@Moelf)
• #429 fix alignment for compressed data (#430) (@baumgold)

Arrow v2.5.0

Diff since v2.4.3

Closed issues:

• How to represent customize type field in schema (#383)
• Making Custom types documentation example work (#386)
• Tag release (#387)

Merged pull requests:

• Revert "Tag new version dev/release/release.sh (#381)" (#388) (@kou)
• fix arrowname and arrowmetadata for nullable types (#389) (@baumgold)
• add IPv4/IPv6 support to ArrowTypes (#390) (@baumgold)
• define eltype for Stream (#394) (@baumgold)
• Implement Tables.columnnames and Tables.schema for Arrow.Stream (#395) (@baumgold)
• Bump Arrow version to 2.5.0 (#406) (@baumgold)

Arrow v2.4.3

Diff since v2.2.1

Closed issues:

• Re-use PyArrow memory via PyCall (#92)
• Can't roundtrip NamedTuples of Nanoseconds twice (#214)
• Serializing Dict{String,Real} result in garbage values (#232)
• VersionNumber issues when you also have missing (#243)
• Order of record batches from "arrow file" format files (i.e. Arrow.Table) not preserved (#295)
• dev/release/release.sh misses a feature that removes old releases and RCs (#307)
• Release process misses "add a new release to ASF's report database" (#311)
• RC verification script name is wrong (#313)
• Verify release CI jobs are failed for a RC1 commit (#314)
• Versions in footer and message do not agree, this causes issues reading Arrow files with other libraries (such as arrow-rs). (#320)
• filtering DataFrame loaded from feather file triggers deleteat! error (#324)
• DST ambiguities in ZonedDateTime not supported (#327)
• error earlier when number of entries don't match across all fields (#344)
• Tests fail on Apple silicon on Julia 1.8 (#345)
• Install Registrator.jl github app (#348)
• Allow appending record batches to an existing Arrow file (#352)
• Arrow.append to non-existent file (#354)
• PooledArray are incorrectly saved (#364)
• inappropriately applied Arrow.NullVector optimization to Union{ZonedDateTime,Missing} column (#367)
• Release [email protected] (#376)
• RC verification script doesn't test ArrowTypes (#378)
• dev/release/release.sh doesn't tag (#380)

Merged pull requests:

• refactor Arrow.write to support incremental writes (#277) (@baumgold)
• disallow non-concrete map-like types to prevent incorrect serialization (#305) (@jrevels)
• add missing arrowtype(b, ::Type{<:Period}) method to enable roundtripping of Period types (#306) (@jrevels)
• Bump version to 2.3.0 (#312) (@quinnj)
• Remove old releases and RCs on a new release (#315) (@kou)
• Use a large RC number for CI (#316) (@kou)
• Fix wrong RC verification path (#317) (@kou)
• Add a missing release process to update ASF's report database (#318) (@kou)
• fix version mismatch by changing footer to V5 (#321) (@pcjentsch)
• Fix repo name in the Readme (#322) (@CarloLucibello)
• allow ntasks to be 0 and determine whetheher to use threads based on nthreads rather than ntasks (#325) (@baumgold)
• Store ZDT with a UTC, not local, timestamp (#329) (@tpgillam)
• support reading empty files that are in the process of being written (#338) (@baumgold)
• Use OrderedSynchronizer instead of OrderedChannel (#339) (@quinnj)
• Add ArgumentError for deleteat on ArrowVectors; fixes #324 (#341) (@quinnj)
• Remove requirement to have 1 approval on PRs (#343) (@quinnj)
• Bump version to 2.4.0 (#346) (@baumgold)
• Tar is also needed to verify release candidates (#347) (@bkmgit)
• fix Julia 1.6 (#349) (@SimonDanisch)
• replace custom debug handling with LoggingExtras (#355) (@baumgold)
• fix BatchIterator iterate method to handle partial messages (#356) (@baumgold)
• Ensure Julia types have alignment respected (#357) (@quinnj)
• allow append to act on non-existent/blank file/io (#358) (@tanmaykm)
• don't run tests that fail on Windows (#361) (@visr)
• Bump compat for LoggingExtras for 1.0 (#362) (@quinnj)
• Ensure elements are converted when indexed from ArrowTypes.ToArrow (#365) (@quinnj)
• MINOR: Update issue notifications to issues mailing list to match Arrow repo (#366) (@raulcd)
• Create CompatHelper.yml (#369) (@CarloLucibello)
• Fix issue with missing and non-concrete Arrow types (#371) (@omus)
• Define defaults for Missing/Nothing (#372) (@omus)
• bugfix & test for columns with VersionNumber & missing (#374) (@ericphanson)
• Bump Arrow version to 2.4.2 (#377) (@quinnj)
• Test ArrowTypes package in RC verification script (#379) (@kou)
• Tag new version dev/release/release.sh (#381) (@kou)
• Fix ToArrow when using an abstract eltype with all missing elements (#385) (@omus)

Arrow v2.2.1

Diff since v2.2.0

Closed issues:

• memory not freed reading python exported compressed feather format (#93)
• error when metadata is present but empty? (#253)
• pyarrow unable to load Arrow data written by Arrow.jl (#261)
• Support for Decimal (#264)
• Transfer repository to apache organization (#265)
• Error with Julia 1.7.0 (#269)
• Add GitHub Actions used in this repository to allow list (#273)
• Future of the Julia arrow implementation (#284)
• Check license automatically (#286)
• Implement release script (#287)
• Implement release verification script (#288)
• Release script publishes the artifacts to wrong URL (#301)

Merged pull requests:

• Remove use of symlinks in CI matrix (#256) (@omus)
• Proposal: change @scopedenum to make modules to avoid type piracy (#267) (@NHDaly)
• Configure repository metadata (#272) (@kou)
• Send issue comments notification to [email protected] (#274) (@kou)
• Fix case where metadata is provided but empty (#276) (@quinnj)
• Introduce Release audit tool (Rat) (#289) (@kou)
• Add release scripts (#290) (@kou)
• Add verification script (#292) (@kou)
• Bump version for release (#299) (@quinnj)
• Fix wrong release artifacts URL (#302) (@kou)

Arrow v2.2.0

Diff since v2.1.0

Closed issues:

• Error when displaying value of Arrow.NullVector{ZonedDateTime} (#248)
• Stable docs point towards an old version (#249)
• Avro.jl exists (#251)

Merged pull requests:

• add DOCUMENTER_KEY secret to TagBot (#250) (@KristofferC)
• Add Avro reference, add links (#252) (@KronosTheLate)
• Replace Base.n_waiters with isempty (#254) (@barucden)
• Support AbstractPath where file paths are used (#255) (@omus)

Arrow v2.1.0

Diff since v2.0.0

Merged pull requests:

• Remove unused line (#245) (@nickrobinson251)
• Add ability to pass directory of inputs to Arrow.Table/Arrow.Stream (#246) (@quinnj)

Arrow v2.0.0

Diff since v1.6.2

Closed issues:

• remove global metadata cache (OBJ_METADATA) (#90)
• Ambiguity error with fromarrow with missings and NamedTupless (#199)
• use of setmetadata! on generic tables basically requires calling Tables.columns on the input (#211)
• How to read parquet files, partitioned datasets? (#227)
• bump request (#242)

Merged pull requests:

• fix ambiguity error (#219) (@ericphanson)
• bugfix reading arrays (#234) (@jonalm)
• remove global metadata cache, refactor custom_metadata API (#238) (@jrevels)
• delete deprecations in preparation for v2.0 release (#241) (@jrevels)

Arrow v1.6.2

Diff since v1.6.1

Merged pull requests:

• Support Julia 1.0 for ArrowTypes package (#223) (@omus)
• custom struct getindex should deserialize based on the shape of the actual ArrowType, not the target JuliaType (#229) (@jrevels)

Arrow v1.6.1

Diff since v1.6.0

Merged pull requests:

• Add maxlog=1 to not spam logs (#224) (@ericphanson)
• use _id in warn logging (#225) (@ericphanson)

Arrow v1.6.0

Diff since v1.5.0

Closed issues:

• A vector with elementtype Any stackoverflows Arrow.write: (#191)
• Writing an empty DataFrame errors (#194)
• ArrowTypes as standalone package doesn't interoperate with Arrow.ArrowTypes (#209)
• ERROR: type Nothing has no field fields (#210)
• Remove master branch (#216)

Merged pull requests:

• Rename LICENSE.md to LICENSE (#195) (@simeonschaub)
• Test expected log records (#204) (@omus)
• Use standalone ArrowTypes package (#212) (@omus)
• Set ArrowTypes to version 1.1.0 (#213) (@omus)
• Remove old arrowtypes.jl file (#215) (@omus)
• add metadata to show method (#217) (@ericphanson)
• fix writing an empty table (#221) (@KristofferC)
• Set project version to 1.6.0 (#222) (@omus)

Arrow v1.5.0

Diff since v1.4.1

Closed issues:

• Better fallback when registered types don't match (#133)
• Issue with Tables.rowtable when entries contain a vector of strings (#167)
• VersionNumbers don't serialize (#200)
• Error in "arrow json integration tests" (#202)
• Automated conversion fails for Date{MILLISECOND, Int64} (#206)

Merged pull requests:

• Ensure requested List type is requested on List getindex (#182) (@quinnj)
• Add global metadata lock to ensure thread safety of global metadata (#183) (@quinnj)
• Handle empty tuple (#201) (@omus)
• Support VersionNumber (#205) (@omus)
• support Date with type parameter ms (#207) (@piever)
• bump Project.toml from v1.4.1 to v1.5.0 (#208) (@jrevels)

Arrow v1.4.1

Diff since v1.4.0

Closed issues:

• maxdepth argument not fully forwarded to inner functions from Arrow.write? (#180)

Merged pull requests:

• fix propagation of maxdepth kwarg (#181) (@jrevels)