unstructured

Enhancements

• Remove page_number metadata fields for HTML partition until we have a better strategy to decide page counting.
• Extract OCRAgent.get_agent(). Generalize access to the configured OCRAgent instance beyond its use for PDFs.
• Add calculation of table related metrics which take into account colspans and rowspans

Features

• add ability to get ratio of cid characters in embedded text extracted by pdfminer.

Fixes

• partition_docx() handles short table rows. The DOCX format allows a table row to start late and/or end early, meaning cells at the beginning or end of a row can be omitted. While there are legitimate uses for this capability, using it in practice is relatively rare. However, it can happen unintentionally when adjusting cell borders with the mouse. Accommodate this case and generate accurate .text and .metadata.text_as_html for these tables.
• Remedy macOS test failure not triggered by CI. Generalize temp-file detection beyond hard-coded Linux-specific prefix.
• Remove unnecessary warning log for using default layout model.
• Add chunking to partition_tsv Even though partition_tsv() produces a single Table element, chunking is made available because the Table element is often larger than the desired chunk size and must be divided into smaller chunks.

0.13.6

Enhancements

Features

Fixes

• ValueError: Invalid file (FileType.UNK) when parsing Content-Type header with charset directive URL response Content-Type headers are now parsed according to RFC 9110.

0.13.5

Enhancements

Features

Fixes

• KeyError raised when updating parent_id In the past, combining ListItem elements could result in reusing the same memory location which then led to unexpected side effects when updating element IDs.
• Bump unstructured-inference==0.7.29: table transformer predictions are now removed if confidence is below threshold

Enhancements

• Unique and deterministic hash IDs for elements Element IDs produced by any partitioning
  function are now deterministic and unique at the document level by default. Before, hashes were
  based only on text; however, they now also take into account the element's sequence number on a
  page, the page's number in the document, and the document's file name.
• Enable remote chunking via unstructured-ingest Chunking using unstructured-ingest was
  previously limited to local chunking using the strategies basic and by_title. Remote chunking
  options via the API are now accessible.
• Save table in cells format. UnstructuredTableTransformerModel is able to return predicted table in cells format

Features

• Add a PDF_ANNOTATION_THRESHOLD environment variable to control the capture of embedded links in partition_pdf() for fast strategy.
• Add integration with the Google Cloud Vision API. Adds a third OCR provider, alongside Tesseract and Paddle: the Google Cloud Vision API.

Fixes

• Remove ElementMetadata.section field.. This field was unused, not populated by any partitioners.

Enhancements

• Remove duplicate image elements. Remove image elements identified by PDFMiner that have similar bounding boxes and the same text.
• Add support for start_index in html links extraction
• Add strategy arg value to _PptxPartitionerOptions. This makes this paritioning option available for sub-partitioners to come that may optionally use inference or other expensive operations to improve the partitioning.
• Support pluggable sub-partitioner for PPTX Picture shapes. Use a distinct sub-partitioner for partitioning PPTX Picture (image) shapes and allow the default picture sub-partitioner to be replaced at run-time by one of the user's choosing.
• Introduce starting_page_number parameter to partitioning functions It applies to those partitioners which support page_number in element's metadata: PDF, TIFF, XLSX, DOC, DOCX, PPT, PPTX.
• Redesign the internal mechanism of assigning element IDs This allows for further enhancements related to element IDs such as deterministic and document-unique hashes. The way partitioning functions operate hasn't changed, which means unique_element_ids continues to be False by default, utilizing text hashes.

Features

Fixes

• Add support for extracting text from tag tails in HTML. This fix adds ability to generate separate elements using tag tails.
• Add support for extracting text from <b> tags in HTML Now partition_html() can extract text from <b> tags inside container tags (like <div>, <pre>).
• Fix pip-compile make target Missing base.in dependency missing from requirments make file added

0.13.2

Enhancements

Features

Fixes

• Brings back missing word list files that caused partition failures in 0.13.1.

0.13.1

Enhancements

• Drop constraint on pydantic, supporting later versions All dependencies has pydantic pinned at an old version. This explicit pin was removed, allowing the latest version to be pulled in when requirements are compiled.

Features

• Add a set of new ElementTypes to extend future element types

Fixes

• Fix partition_html() swallowing some paragraphs. The partition_html() only considers elements with limited depth to avoid becoming the text representation of a giant div. This fix increases the limit value.
• Fix SFTP Adds flag options to SFTP connector on whether to use ssh keys / agent, with flag values defaulting to False. This is to prevent looking for ssh files when using username and password. Currently, username and password are required, making that always the case.

0.13.0

Enhancements

• Add .metadata.is_continuation to text-split chunks. .metadata.is_continuation=True is added to second-and-later chunks formed by text-splitting an oversized Table element but not to their counterpart Text element splits. Add this indicator for CompositeElement to allow text-split continuation chunks to be identified for downstream processes that may wish to skip intentionally redundant metadata values in continuation chunks.
• Add compound_structure_acc metric to table eval. Add a new property to unstructured.metrics.table_eval.TableEvaluation: composite_structure_acc, which is computed from the element level row and column index and content accuracy scores
• Add .metadata.orig_elements to chunks. .metadata.orig_elements: list[Element] is added to chunks during the chunking process (when requested) to allow access to information from the elements each chunk was formed from. This is useful for example to recover metadata fields that cannot be consolidated to a single value for a chunk, like page_number, coordinates, and image_base64.
• Add --include_orig_elements option to Ingest CLI. By default, when chunking, the original elements used to form each chunk are added to chunk.metadata.orig_elements for each chunk. * The include_orig_elements parameter allows the user to turn off this behavior to produce a smaller payload when they don't need this metadata.
• Add Google VertexAI embedder Adds VertexAI embeddings to support embedding via Google Vertex AI.

Features

• Chunking populates .metadata.orig_elements for each chunk. This behavior allows the text and metadata of the elements combined to make each chunk to be accessed. This can be important for example to recover metadata such as .coordinates that cannot be consolidated across elements and so is dropped from chunks. This option is controlled by the include_orig_elements parameter to partition_*() or to the chunking functions. This option defaults to True so original-elements are preserved by default. This behavior is not yet supported via the REST APIs or SDKs but will be in a closely subsequent PR to other unstructured repositories. The original elements will also not serialize or deserialize yet; this will also be added in a closely subsequent PR.
• Add Clarifai destination connector Adds support for writing partitioned and chunked documents into Clarifai.

Fixes

• Fix clean_pdfminer_inner_elements() to remove only pdfminer (embedded) elements merged with inferred elements. Previously, some embedded elements were removed even if they were not merged with inferred elements. Now, only embedded elements that are already merged with inferred elements are removed.
• Clarify IAM Role Requirement for GCS Platform Connectors. The GCS Source Connector requires Storage Object Viewer and GCS Destination Connector requires Storage Object Creator IAM roles.
• Change table extraction defaults Change table extraction defaults in favor of using skip_infer_table_types parameter and reflect these changes in documentation.
• Fix OneDrive dates with inconsistent formatting Adds logic to conditionally support dates returned by office365 that may vary in date formatting or may be a datetime rather than a string. See previous fix for SharePoint
• Adds tracking for AstraDB Adds tracking info so AstraDB can see what source called their api.
• Support AWS Bedrock Embeddings in ingest CLI The configs required to instantiate the bedrock embedding class are now exposed in the api and the version of boto being used meets the minimum requirement to introduce the bedrock runtime required to hit the service.
• Change MongoDB redacting Original redact secrets solution is causing issues in platform. This fix uses our standard logging redact solution.

0.12.6

Enhancements

• Improve ability to capture embedded links in partition_pdf() for fast strategy Previously, a threshold value that affects the capture of embedded links was set to a fixed value by default. This allows users to specify the threshold value for better capturing.
• Refactor add_chunking_strategy decorator to dispatch by name. Add chunk() function to be used by the add_chunking_strategy decorator to dispatch chunking call based on a chunking-strategy name (that can be dynamic at runtime). This decouples chunking dispatch from only those chunkers known at "compile" time and enables runtime registration of custom chunkers.

Features

• Added Unstructured Platform Documentation The Unstructured Platform is currently in beta. The documentation provides how-to guides for setting up workflow automation, job scheduling, and configuring source and destination connectors.

Fixes

• Partitioning raises on file-like object with .name not a local file path. When partitioning a file using the file= argument, and file is a file-like object (e.g. io.BytesIO) having a .name attribute, and the value of file.name is not a valid path to a file present on the local filesystem, FileNotFoundError is raised. This prevents use of the file.name attribute for downstream purposes to, for example, describe the source of a document retrieved from a network location via HTTP.
• Fix SharePoint dates with inconsistent formatting Adds logic to conditionally support dates returned by office365 that may vary in date formatting or may be a datetime rather than a string.
• Include warnings about the potential risk of installing a version of pandoc which does not support RTF files + instructions that will help resolve that issue.
• Incorporate the install-pandoc Makefile recipe into relevant stages of CI workflow, ensuring it is a version that supports RTF input files.
• Fix Google Drive source key Allow passing string for source connector key.
• Fix table structure evaluations calculations Replaced special value -1.0 with np.nan and corrected rows filtering of files metrics basing on that.
• Fix Sharepoint-with-permissions test Ignore permissions metadata, update test.
• Fix table structure evaluations for edge case Fixes the issue when the prediction does not contain any table - no longer errors in such case.

0.12.5

Features

• Header and footer detection for fast strategy partition_pdf with fast strategy now
  detects elements that are in the top or bottom 5 percent of the page as headers and footers.
• Add parent_element to overlapping case output Adds parent_element to the output for identify_overlapping_or_nesting_case and catch_overlapping_and_nested_bboxes functions.
• Add table structure evaluation Adds a new function to evaluate the structure of a table and return a metric that represents the quality of the table structure. This function is used to evaluate the quality of the table structure and the table contents.
• Add AstraDB destination connector Adds support for writing embedded documents into an AstraDB vector database.

Fixes

• Fix passing list type parameters when calling unstructured API via partition_via_api() Update partition_via_api() to convert all list type parameters to JSON formatted strings before calling the unstructured client SDK. This will support image block extraction via partition_via_api().
• Add OctoAI embedder Adds support for embeddings via OctoAI.
• Fix check_connection in opensearch, databricks, postgres, azure connectors
• **Fix don't treat plain text files with double quotes as JSON ** If a file can be deserialized as JSON but it deserializes as a string, treat it as plain text even though it's valid JSON.
• **Fix check_connection in opensearch, databricks, postgres, azure connectors **
• Fix cluster of bugs in partition_xlsx() that dropped content. Algorithm for detecting "subtables" within a worksheet dropped table elements for certain patterns of populated cells such as when a trailing single-cell row appeared in a contiguous block of populated cells.
• Improved documentation. Fixed broken links and improved readability on Key Concepts page.
• **Rename OpenAiEmbeddingConfig to OpenAIEmbeddingConfig.
• Fix partition_json() doesn't chunk. The @add_chunking_strategy decorator was missing from partition_json() such that pre-partitioned documents serialized to JSON did not chunk when a chunking-strategy was specified.

0.12.4

Enhancements

• Apply New Version of black formatting The black library recently introduced a new major version that introduces new formatting conventions. This change brings code in the unstructured repo into compliance with the new conventions.
• Move ingest imports to local scopes Moved ingest dependencies into local scopes to be able to import ingest connector classes without the need of installing imported external dependencies. This allows lightweight use of the classes (not the instances. to use the instances as intended you'll still need the dependencies).
• Add support for .p7s files partition_email can now process .p7s files. The signature for the signed message is extracted and added to metadata.
• Fallback to valid content types for emails If the user selected content type does not exist on the email message, partition_email now falls back to anoter valid content type if it's available.

Features

• Add .heic file partitioning .heic image files were previously unsupported and are now supported though partition_image()
• Add the ability to specify an alternate OCR implementation by implementing an OCRAgent interface and specify it using OCR_AGENT environment variable.
• Add Vectara destination connector Adds support for writing partitioned documents into a Vectara index.

Fixes

• Fix partition_pdf() not working when using chipper model with file
• Handle common incorrect arguments for languages and ocr_languages Users are regularly receiving errors on the API because they are defining ocr_languages or languages with additional quotationmarks, brackets, and similar mistakes. This update handles common incorrect arguments and raises an appropriate warning.
• Default hi_res_model_name now relies on unstructured-inference When no explicit hi_res_model_name is passed into partition or partition_pdf_or_image the default model is picked by unstructured-inference's settings or os env variable UNSTRUCTURED_HI_RES_MODEL_NAME; it now returns the same model name regardless of infer_table_structure's value; this function will be deprecated in the future and the default model name will simply rely on unstructured-inference and will not consider os env in a future release.
• Fix remove Vectara requirements from setup.py - there are no dependencies
• Add missing dependency files to package manifest. Updates the file path for the ingest
  dependencies and adds missing extra dependencies.
• Fix remove Vectara requirements from setup.py - there are no dependencies
• Add title to Vectara upload - was not separated out from initial connector
• Fix change OpenSearch port to fix potential conflict with Elasticsearch in ingest test

Enhancements

• Driver for MongoDB connector. Adds a driver with unstructured version information to the
  MongoDB connector.

Features

• Add Databricks Volumes destination connector Databricks Volumes connector added to ingest CLI. Users may now use unstructured-ingest to write partitioned data to a Databricks Volumes storage service.

Fixes

• Fix support for different Chipper versions and prevent running PDFMiner with Chipper
• Treat YAML files as text. Adds YAML MIME types to the file detection code and treats those
  files as text.
• Fix FSSpec destination connectors check_connection. FSSpec destination connectors did not use check_connection. There was an error when trying to ls destination directory - it may not exist at the moment of connector creation. Now check_connection calls ls on bucket root and this method is called on initialize of destination connector.
• Fix databricks-volumes extra location. setup.py is currently pointing to the wrong location for the databricks-volumes extra requirements. This results in errors when trying to build the wheel for unstructured. This change updates to point to the correct path.
• Fix uploading None values to Chroma and Pinecone. Removes keys with None values with Pinecone and Chroma destinations. Pins Pinecone dependency
• Update documentation. (i) best practice for table extration by using 'skip_infer_table_types' param, instead of 'pdf_infer_table_structure', and (ii) fixed CSS, RST issues and typo in the documentation.
• Fix postgres storage of link_texts. Formatting of link_texts was breaking metadata storage.

0.12.2

Enhancements

Features

Fixes

• Fix index error in table processing. Bumps the unstructured-inference version to address and
  index error that occurs on some tables in the table transformer object.

0.12.1

Enhancements

• Allow setting image block crop padding parameter In certain circumstances, adjusting the image block crop padding can improve image block extraction by preventing extracted image blocks from being clipped.
• Add suport for bitmap images in partition_image Adds support for .bmp files in
  partition, partition_image, and detect_filetype.
• Keep all image elements when using "hi_res" strategy Previously, Image elements with small chunks of text were ignored unless the image block extraction parameters (extract_images_in_pdf or extract_image_block_types) were specified. Now, all image elements are kept regardless of whether the image block extraction parameters are specified.
• Add filetype detection for .wav files. Add filetpye detection for .wav files.
• Add "basic" chunking strategy. Add baseline chunking strategy that includes all shared chunking behaviors without breaking chunks on section or page boundaries.
• Add overlap option for chunking. Add option to overlap chunks. Intra-chunk and inter-chunk overlap are requested separately. Intra-chunk overlap is applied only to the second and later chunks formed by text-splitting an oversized chunk. Inter-chunk overlap may also be specified; this applies overlap between "normal" (not-oversized) chunks.
• Salesforce connector accepts private key path or value. Salesforce parameter private-key-file has been renamed to private-key. Private key can be provided as path to file or file contents.
• Update documentation: (i) added verbiage about the free API cap limit, (ii) added deprecation warning on Staging bricks in favor of Destination Connectors, (iii) added warning and code examples to use the SaaS API Endpoints using CLI-vs-SDKs, (iv) fixed example pages formatting, (v) added deprecation on model_name in favor of hi_res_model_name, (vi) added extract_images_in_pdf usage in partition_pdf section, (vii) reorganize and improve the documentation introduction section, and (viii) added PDF table extraction best practices.
• Add "basic" chunking to ingest CLI. Add options to ingest CLI allowing access to the new "basic" chunking strategy and overlap options.
• Make Elasticsearch Destination connector arguments optional. Elasticsearch Destination connector write settings are made optional and will rely on default values when not specified.
• Normalize Salesforce artifact names. Introduced file naming pattern present in other connectors to Salesforce connector.
• Install Kapa AI chatbot. Added Kapa.ai website widget on the documentation.

Features

• MongoDB Source Connector. New source connector added to all CLI ingest commands to support downloading/partitioning files from MongoDB.
• Add OpenSearch source and destination connectors. OpenSearch, a fork of Elasticsearch, is a popular storage solution for various functionality such as search, or providing intermediary caches within data pipelines. Feature: Added OpenSearch source connector to support downloading/partitioning files. Added OpenSearch destination connector to be able to ingest documents from any supported source, embed them and write the embeddings / documents into OpenSearch.

Fixes

• Fix GCS connector converting JSON to string with single quotes. FSSpec serialization caused conversion of JSON token to string with single quotes. GCS requires token in form of dict so this format is now assured.
• Pin version of unstructured-client Set minimum version of unstructured-client to avoid raising a TypeError when passing api_key_auth to UnstructuredClient
• Fix the serialization of the Pinecone destination connector. Presence of the PineconeIndex object breaks serialization due to TypeError: cannot pickle '_thread.lock' object. This removes that object before serialization.
• Fix the serialization of the Elasticsearch destination connector. Presence of the _client object breaks serialization due to TypeError: cannot pickle '_thread.lock' object. This removes that object before serialization.
• Fix the serialization of the Postgres destination connector. Presence of the _client object breaks serialization due to TypeError: cannot pickle '_thread.lock' object. This removes that object before serialization.
• Fix documentation and sample code for Chroma. Was pointing to wrong examples..
• Fix flatten_dict to be able to flatten tuples inside dicts Update flatten_dict function to support flattening tuples inside dicts. This is necessary for objects like Coordinates, when the object is not written to the disk, therefore not being converted to a list before getting flattened (still being a tuple).
• Fix the serialization of the Chroma destination connector. Presence of the ChromaCollection object breaks serialization due to TypeError: cannot pickle 'module' object. This removes that object before serialization.
• Fix fsspec connectors returning version as integer. Connector data source versions should always be string values, however we were using the integer checksum value for the version for fsspec connectors. This casts that value to a string.

• Drop support for python3.8 All dependencies are now built off of the minimum version of python being 3.10

0.11.8

Enhancements

• Add SaaS API User Guide. This documentation serves as a guide for Unstructured SaaS API users to register, receive an API key and URL, and manage your account and billing information.

Enhancements

• Add intra-chunk overlap capability. Implement overlap for split-chunks where text-splitting is used to divide an oversized chunk into two or more chunks that fit in the chunking window. Note this capability is not yet available from the API but will shortly be made accessible using a new overlap kwarg on partition functions.
• Update encoders to leverage dataclasses All encoders now follow a class approach which get annotated with the dataclass decorator. Similar to the connectors, it uses a nested dataclass for the configs required to configure a client as well as a field/property approach to cache the client. This makes sure any variable associated with the class exists as a dataclass field.

Features

• Add Qdrant destination connector. Adds support for writing documents and embeddings into a Qdrant collection.
• Store base64 encoded image data in metadata fields. Rather than saving to file, stores base64 encoded data of the image bytes and the mimetype for the image in metadata fields: image_base64 and image_mime_type (if that is what the user specifies by some other param like pdf_extract_to_payload). This would allow the API to have parity with the library.

Fixes

• Fix table structure metric script Update the call to table agent to now provide OCR tokens as required
• Fix element extraction not working when using "auto" strategy for pdf and image If element extraction is specified, the "auto" strategy falls back to the "hi_res" strategy.
• Fix a bug passing a custom url to partition_via_api Users that self host the api were not able to pass their custom url to partition_via_api.

0.11.6

Enhancements

• Update the layout analysis script. The previous script only supported annotating final elements. The updated script also supports annotating inferred and extracted elements.
• AWS Marketplace API documentation: Added the user guide, including setting up VPC and CloudFormation, to deploy Unstructured API on AWS platform.
• Azure Marketplace API documentation: Improved the user guide to deploy Azure Marketplace API by adding references to Azure documentation.
• Integration documentation: Updated URLs for the staging_for bricks

Features

• Partition emails with base64-encoded text. Automatically handles and decodes base64 encoded text in emails with content type text/plain and text/html.
• Add Chroma destination connector Chroma database connector added to ingest CLI. Users may now use unstructured-ingest to write partitioned/embedded data to a Chroma vector database.
• Add Elasticsearch destination connector. Problem: After ingesting data from a source, users might want to move their data into a destination. Elasticsearch is a popular storage solution for various functionality such as search, or providing intermediary caches within data pipelines. Feature: Added Elasticsearch destination connector to be able to ingest documents from any supported source, embed them and write the embeddings / documents into Elasticsearch.

Fixes

• Enable --fields argument omission for elasticsearch connector Solves two bugs where removing the optional parameter --fields broke the connector due to an integer processing error and using an elasticsearch config for a destination connector resulted in a serialization issue when optional parameter --fields was not provided.

0.11.5

Enhancements

Features

Fixes

• Fix partition_pdf() and partition_image() importation issue. Reorganize pdf.py and image.py modules to be consistent with other types of document import code.

0.11.4

Enhancements

• Refactor image extraction code. The image extraction code is moved from unstructured-inference to unstructured.
• Refactor pdfminer code. The pdfminer code is moved from unstructured-inference to unstructured.
• Improve handling of auth data for fsspec connectors. Leverage an extension of the dataclass paradigm to support a sensitive annotation for fields related to auth (i.e. passwords, tokens). Refactor all fsspec connectors to use explicit access configs rather than a generic dictionary.
• Add glob support for fsspec connectors Similar to the glob support in the ingest local source connector, similar filters are now enabled on all fsspec based source connectors to limit files being partitioned.
• Define a constant for the splitter "+" used in tesseract ocr languages.

Features

• Save tables in PDF's separately as images. The "table" elements are saved as table-<pageN>-<tableN>.jpg. This filename is presented in the image_path metadata field for the Table element. The default would be to not do this.
• Add Weaviate destination connector Weaviate connector added to ingest CLI. Users may now use unstructured-ingest to write partitioned data from over 20 data sources (so far) to a Weaviate object collection.
• Sftp Source Connector. New source connector added to support downloading/partitioning files from Sftp.

Fixes

• Fix pdf hi_res partitioning failure when pdfminer fails. Implemented logic to fall back to the "inferred_layout + OCR" if pdfminer fails in the hi_res strategy.
• Fix a bug where image can be scaled too large for tesseract Adds a limit to prevent auto-scaling an image beyond the maximum size tesseract can handle for ocr layout detection
• Update partition_csv to handle different delimiters CSV files containing both non-comma delimiters and commas in the data were throwing an error in Pandas. partition_csv now identifies the correct delimiter before the file is processed.
• partition returning cid code in hi_res occasionally pdfminer can fail to decode the text in an pdf file and return cid code as text. Now when this happens the text from OCR is used.