ranking

This is the 0.5.3 release of TensorFlow Ranking. We release a ranking distillation benchmark called RD-Suite that will accompany a research paper.

We also include a new ranking loss, YetiLogisticLoss: Adapted to neural network models from the Yeti loss implementation for GBDT.

This is the 0.5.2 release of TensorFlow Ranking. We fix bugs and make a few improvements to the library. We also update the API reference on www.tensorflow.org/ranking and on Github docs.

This is the 0.5.1 release of TensorFlow Ranking. We provide new ranking losses, metrics, layers, and pipeline based on the latest research progresses in Learning to Rank and Unbiased Ranking. We also update the API reference on www.tensorflow.org/ranking and on Github docs. The new changes include:

• Ranking losses added in tfr.keras.losses:
  • PairwiseMSELoss: Implement a pairwise mean squared error loss.
  • OrdinalLoss: Implement a pointwise multi-head ordinal regression on ordered multilabel.
  • MixtureEMLoss: Implement a listwise Expectation-Maximization algorithm on a mixture model, introduced in Revisiting two tower models for unbiased learning to rank.
• Lambda weights for Lambda losses added in tfr.keras.losses:
  • NDCGLambdaWeightV2: Implement an NDCG-based lambda weight for lambda losses, introduced in On Optimizing Top-K Metrics for Neural Ranking Models.
  • LabelDiffLambdaWeight: Implement a lambda weight based on the absolute difference of two labels.
• Ranking metric added in tfr.keras.metrics:
  • HitsMetric: Implement Hits@k metric.
• Ranking layer added in tfr.keras.layers:
  • Bilinear: A layer to implement a bilinear interaction of two vectors, used in Revisiting two tower models for unbiased learning to rank.
• Ranking pipeline added in tfr.keras.pipeline:
  • MultiObjectivePipeline: A pipeline to apply multi-objective losses, used in Scale Calibration of Deep Ranking Models.
• API reference updated on www.tensorflow.org/ranking and consistently on Github docs.

Dependencies: The following packages will be installed as required when installing tensorflow-ranking.
tensorflow-serving-api>= 2.0.0, < 3.0.0
tensorflow>=2.7.0.

This is the 0.5.0 release of TensorFlow Ranking. We provide a detailed overview, tutorial notebooks and API reference on www.tensorflow.org/ranking. The new changes are:

• Move task.py and premade tfrbert_task.py to extension.
• Remove RankingNetwork based tfr-bert example. The latest tfr-bert example using native Keras is available at tfrbert_antique_train.py.
• Remove dependency on tf-models-official package to reduce install time. Users of tfr.ext.task or modules that depend on the above package will need to manually install it.
• Updated all docstrings to be more detailed. Made several docstrings to be testable.
• Add colab notebooks for quickstart tutorial and distributed ranking tutorial, also available on www.tensorflow.org/ranking.
• Update strategy_utils to support parameter server strategy.
• Add symmetric log1p to tfr.utils.
• Remove references to Estimator/Feature Column related APIs in API reference.

This is the 0.4.2 release of TensorFlow Ranking. The main changes are the TFR-BERT module based on the Orbit framework in tf-models, which facilitates users to write customized training loops. The new components are:

TFR-BERT in Orbit

• tfr.keras.task: This module contains the general boilerplate code to train TF-Ranking models in the Orbit framework. Particularly, there are:
  • RankingDataLoader, which parses an ELWC formatted data record into tensors
  • RankingTask, which specifies the behaviors of each training and evaluation step, as well as the training losses and evaluation metrics.
  • In addition, there are config data classes like RankingDataConfig and RankingTaskConfig to store configurations for above classes.
• tfr.keras.premade.tfrbert_task: This module contains the TFR-BERT specification of the TF-Ranking Orbit task.
  • TFRBertDataLoader, which subclasses the RankingDataLoader and further specifies the feature specs of a TFR-BERT model.
  • TFRBertScorer and TFRBertModelBuilder, which defines a model builder that can create a TFR-BERT ranking model as a Keras model, based on tf-models’ implementation of BERT encoder.
  • TFRBertTask, which is a subclass of RankingTask. It defines the build_model behavior. It also defines the initialization method which would load an pretrained BERT checkpoint to initialize the encoder. It also provides the function to output the prediction results along with query ids and document ids.
  • In addition, there are config data classes like TFRBertDataConfig, TFRBertModelConfig and TFRBertConfig which stores configurations for above classes.
• examples/keras/tfrbert_antique_train.py: This file provides an example of training a TFR-BERT model on the Antique data set. There is also an .yaml file where users can specify parameter configurations.

Dependencies: The following packages will be installed as required when installing tensorflow-ranking.

• tf-models-official >= 2.5.0
• tensorflow-serving-api>= 2.0.0, < 3.0.0
• tensorflow==2.5.0.

This release is one of the major releases for TF-Ranking. It provides full support to build and train a native Keras model for ranking problems. It includes necessary Keras layers for a ranking model, a module to construct a model in a flexible manner, and a pipeline to train a model with minimal boilerplate. To get started, please follow the example here. In addition, the new release adds RaggedTensor support in losses and metrics and we provide a handy example to show how to use it in a ranking model.

The new components are listed below:

• Keras Layers:

  • Use input packing for layer signatures for SavedModel compatibility.
  • create_tower function to create a feedforward neural network with batch normalization and dropout.
  • GAMLayer, a Keras layer which implements the neural generalized additive ranking model.
  • Update build method of DocumentInteractionAttention layer to ensure SavedModel is restored correctly.

• ModelBuilder to build tf.keras.Model using Functional API:

  • AbstractModelBuilder class for users to inherit.
  • ModelBuilder class that wraps the boilerplate code to build tf.keras.Model for a ranking model.
  • InputCreator abstract class to implement create_inputs in ModelBuilder.
    • FeatureSpecInputCreator class to create inputs from feature_specs.
    • TypeSpecInputCreator class to create inputs from type_specs.
  • Preprocessor abstract class to implement preprocess in ModelBuilder.
    • PreprocessorWithSpec class to do Keras preprocessing or feature transformations with functions specified in Specs.
  • Scorer abstract class to implement score in ModelBuilder.
    • UnivariateScorer class to implement univariate scoring functions.
      • DNNScorer class to implement fully connected DNN univariate scoring.
      • GAMScorer class to implement feature based GAM univariate scoring.

• Pipeline to wrap the boilerplate codes for training:

  • AbstractDatasetBuilder abstract class to build and serve the dataset for training.
  • BaseDatasetBuilder class to build training and validation datasets and signatures for SavedModel from feature_specs.
    • SimpleDatasetBuilder class to build datasets with a single label feature spec.
    • MultiLabelDatasetBuilder class to build datasets for multi-task learning.
  • DatasetHparams dataclass to specify all hyper-parameters used in BaseDatasetBuilder class.
  • AbstractPipeline abstract class to train and validate the ranking tf.keras.Model.
  • ModelFitPipeline class to train the ranking models using model.fit() compatible with distribution strategies.
    • SimplePipeline class for single-task training.
    • MultiTaskPipeline class for multi-task training.
    • An example client to showcase training a deep neural network model with a distribution strategy using SimplePipeline.
  • PipelineHparams dataclass to specify all hyper-parameters used in ModelFitPipeline class.
  • strategy_utils helper module to support tf.distribute strategies.

• RaggedTensor support in losses and metrics:

  • Losses in tfr.keras.losses and metrics in tfr.keras.metrics support to act on tf.RaggedTensor inputs. To do so, set argument ragged=True when defining the loss and metric objects:
    • E.g.: loss = tf.keras.losses.SoftmaxLoss(name=’softmax_loss’, ragged=True)
    • Add this argument in get to get the losses and metrics support ragged tensors: loss = tf.keras.losses.get(‘softmax_loss’, ragged=True)
    • An example client to showcase training a deep neural network model using model.fit() with ragged inputs and outputs.

Dependencies: The following packages will be installed as required when installing tensorflow-ranking.
tf-models-official >= 2.5.0
tensorflow-serving-api>= 2.0.0, < 3.0.0
tensorflow==2.5.0.

This is the 0.3.3 release of TensorFlow Ranking. It depends on tf-models-official >= 2.4.0 and tensorflow-serving-api>= 2.0.0, < 3.0.0. It is compatible with tensorflow==2.4.1. All of these packages will be installed as required packages when installing tensorflow-ranking.

The main changes in this release contain the Document Interaction Network (DIN) layer and layers for training Keras models using Functional API. The new components are listed below:

• Document Interaction Network: See paper.

  • Building Keras ranking models for DIN using Keras Preprocessing Layers.
    • Native Keras training: An example client to showcase such a model using model.fit().
    • Estimator based training: Another example client to showcase training a DIN model as an Estimator.
  • tfr.keras.layers.DocumentInteractionAttention: A keras layer to model cross-document interactions. Applies cross-document attention across valid examples identified using a mask.

• Keras Layers: for easy transformation of context and example features and related utilities.

  • tfr.keras.layers.FlattenList: Flattens the batch_size dimension and the list_size dimension for the example_features and expands list_size times for the context_features.
  • tfr.keras.layers.ConcatFeatures: Concatenates context features and example features in a listwise manner.
  • tfr.keras.layers.RestoreList: Output layer to restore batch_size dimension and list_size dimension for the output shape of logits.

• Others

  • tfr.keras.metrics.get(metric_key): Add a get metric factory for keras metrics.
  • Masking support in tfr.data: Add support for parsing a boolean mask tensor which indicates number of valid examples via mask_feature_name argument in tfr.data._RankingDataParser and all associated input data parsing and serving_input_fn builders.

In the latest release of TensorFlow Ranking v0.3.2, we introduce TFR-BERT extension to better support ranking models for text data based on BERT. BERT is a pre-trained language representation model which has achieved substantial improvement over numerous NLP tasks. We find that fine-tuning BERT with ranking losses further improve the ranking performance (arXiv). You can read detailed information about what is included in TFR-BERT extension here. There is also an example showing how to use TFR-BERT here.

This is the 0.3.1 release of TensorFlow Ranking. It depends on tensorflow-serving-api==2.1.0 and is fully compatible with tensorflow==2.2.0. Both will be installed as required packages when installing tensorflow-ranking.

The main changes in this release are canned Neural RankGAM estimator, canned DNN estimators, canned Neural RankGAM keras models and their examples. The new components are:

• Neural RankGAM

  • make_gam_ranking_estimator, which makes a canned estimator of a neural ranking generalized additive model. An example client to showcase the usage of make_gam_ranking_estimator.

  • make_dnn_ranking_estimator, which makes a canned estimator of a feed-forward neural network for ranking. An example client to showcase the usage of make_dnn_ranking_estimator.

  • GAMRankingNetwork, which encapsulates Neural RankGAM models in a keras ranking network. An example client to showcase the usage of GAMRankingNetwork.

• Keras

  • Add serialization and deserialization support for feature columns via tfr.keras.feature.serialize_feature_columns and tfr.keras.feature.deserialize_feature_columns. RankingNetwork’s get_config and from_config methods rely on this.

This is the 0.3.0 release of TensorFlow Ranking. It depends on tensorflow-serving-api==2.1.0 and is fully compatible with tensorflow==2.1.0. Both will be installed as required packages when installing tensorflow-ranking.

The main changes in this release are related to the DNN Estimator Builder and Keras APIs.

A DNN Estimator Builder is available at tfr.estimator.make_dnn_ranking_estimator().

For Keras, we provide an example to showcase the use of Keras APIs to build ranking models , and a documentation providing step-by-step user instructions outlining the Keras user journey.

The new Keras components are:

• Losses: Ranking losses in Keras object oriented loss format, along with a base class and a factory method. The APIs are:

  • tfr.keras.losses.get(loss_key)
  • Base class: tfr.keras.losses._RankingLoss
  • Losses under tfr.keras.losses.*

• Metrics: Ranking metrics in Keras object oriented metric format, along with a base class and a default metrics getter method. The APIs are:

  • tfr.keras.metrics.get_default_metrics()
  • Base class: [tfr.keras.metrics._RankingMetric`
  • Metrics under tfr.keras.metrics.*

• Feature Transformations: tfr.keras.feature.EncodeListwiseFeatures, to convert sparse ranking features to dense. The APIs are:

  • tfr.keras.feature.EncodeListwiseFeatures

• Ranking Network: Base classes for building Ranking Networks, which define scoring logic. The APIs are:

  • tfr.keras.network.RankingNetwork
  • tfr.keras.network.UnivariateRankingNetwork

• Premade Networks: We support premade architectures users can access out-of-the-box. The APIs are:

  • tfr.keras.canned.DNNRankingNetwork

• Keras Model : Ranking models can be built using Keras Functional Model API. The APIs are:

  • tfr.keras.model.create_keras_model()

• Integration with Estimators and RankingPipeline: Keras model can be converted to Estimator to use Estimator’s training utilities and is compatible with RankingPipeline. The APIs for conversion are:

  • tfr.keras.estimator.model_to_estimator()

This is the 0.2.3 release of TensorFlow Ranking. It depends on tensorflow-serving-api==2.1.0 and is fully compatible with tensorflow==2.1.0. Both will be installed as required packages when installing tensorflow-ranking.

The main changes in this release are:

• Added an EstimatorBuilder Class to encapsulate boilerplate codes when constructing a TF-ranking model Estimator. Clients can access it via tfr.estimator.EstimatorBuilder.
• Added a RankingPipeline Class to hide the boilerplate codes regarding the train and eval data reading, train and eval specs definition, dataset building, exporting strategies. With this, clients can construct a RankingPipeline object using tfr.ext.pipeline.RankingPipeline and then call train_and_eval() to run the pipeline.
• Provided an example to demo the use of tfr.ext.pipeline.RankingPipeline.

This is the 0.2.2 release of TensorFlow Ranking. It depends on tensorflow-serving-api==2.1.0 and is fully compatible with tensorflow==2.1.0. Both will be installed as required packages when installing tensorflow-ranking.
The main changes in this release are:

• Fixed metric computation to include lists without any relevant examples.
• Updated demo code to be TF 2.1.0 compatible.
• Replaced deprecated dataset.output_dtypes with tf.compat.v1.get_output_dtypes(dataset).

This is the 0.2.1 release of TensorFlow Ranking. It depends on tensorflow-serving-api==2.0.0 and is fully compatible with tensorflow==2.0.0. Both will be installed as required packages when installing tensorflow-ranking.

The main changes in this release are:

• Updated demo code to use Antique data in ELWC format.
• Updated tutorial script to demonstrate using weights in metrics and losses.
• Removed LIBSVM generator from tfr.data and updated the docs.
• Make gain and discount parameters in the definition of NDCG configurable.
• Added MAP as a ranking metric.
• Added a topn parameter to MRR metric.

This is the 0.2.0 release of TensorFlow Ranking. It depends on tensorflow-serving-api>=2.0.0 and is fully compatible with tensorflow==2.0.0. Both will be installed as required packages when installing tensorflow-ranking.

There is no new functionality added compared with v0.1.6. This release marks a milestone that our future development will be based on TensorFlow 2.0.

This is the 0.1.6 release of TensorFlow Ranking. We add the dependency to tensorflow-serving-api to use tensorflow.serving.ExampleListWithContext as our input data format. It is tested and stable against TensorFlow 1.15.0 and TensorFlow 2.0.0. The main changes in this release are:

• Support tensorflow.serving.ExampleListWithContext as our input data format (commit). This is a more user-friendly format than the ExampleInExample one.
• Add a demo script for data stored in TFRecord. The stored format can be ExampleListhWithContext or other format defined in data.py.

This is the 0.1.5 release of TensorFlow Ranking. It is tested and stable against TensorFlow version 1.14.0 and TensorFlow version 2.0 RC0. The main changes in this release are:

• Support for Multi-Task Learning and Multi-Objective Learning (Issue #85).
• Deprecate the input_size argument for tfr.feature. encode_listwise_features and infer it automatically in the function.
• Fix the weighted mrr computation for doc-level weights.

This is the 0.1.4 release of TensorFlow Ranking. It is tested and stable against TensorFlow version 1.14.0 and TensorFlow version 2.0 RC0. The main changes in this release are:

• Documentation for APIs. List of symbols/operations are available here.
• Demo for using sparse and embedded features on ANTIQUE dataset.
• Example for prediction using ranking estimator in demo code.
• Code and test cases are fully TF2.0 RC0 compatible.
• Updated tfr.utils.sort_by_scores to break ties.
• Added ApproxMRR loss function.

Announcement:
A hands-on tutorial for TF-Ranking, with relevant theoretical background will be presented on Oct 2 at ICTIR 2019, hosted in Santa Clara, CA. Please consider attending!

This is the 0.1.3 release of TensorFlow Ranking. It is tested and stable against TensorFlow version 1.14.0. The main changes in this release are:

• Introduced an ExampleInExample data format.
• Introduced a factory method to build tf.dataset in different data formats.
• Introduced a factory method to build serving receiving input functions for different data formats.
• Refactored the main modules to be object-oriented to increase the code extensibility.