Iterator that fetches values in batch
OTHER License
Small library providing an Iterator
that batches requests for additional values.
This is useful as a foundation for iterators over data that is in an expensive to
access location, such as a database or a web API.
Core interface:
BatchingIterator
- iterator that batches requests via a BatchingFetcher
BatchingFetcher
- interface with fetchNext
method. You will likely need to create an implementationUtilities:
MultipleBatchingFetcher
- combines a number of BatchingFetcher
instances into oneInMemoryBatchingFetcher
- adapts an array
to the BatchingFetcher
interfaceIteratorBasedBatchingFetcher
- adapts an Iterator
to the BatchingFetcher
interfaceCreate a service that uses an Iterator
.
class TweetImporter {
public function importTweets( Iterator $tweets ) {
foreach ( $tweets as $tweet ) {
$this->tweetStore->saveTweet( $tweet );
}
}
}
Note how this service only depends on Iterator. It is not aware of how the Iterator provides its
results. You thus decoupled the service from who retrieves the results, and from when this happens.
They could be coming from values already in memory, wrapped in an ArrayIterator
, or be pulled from
a web service as iteration happens. Using an ArrayIterator
is very helpful for testing.
Implement the BatchingFetcher
interface. If you already have a service to retrieve the data, this
can be a simple wrapper.
class BatchingTweetFetcher implements BatchingFetcher {
public function fetchNext( $maxFetchCount ) {
// Make a call to some external service to fetch $tweets
return $tweets;
}
public function rewind() {
// Go back to the first tweet
}
}
Now you can easily instantiate the service, have the batching optimization, and have all responsibilities nicely decoupled.
class TweetImportCli {
public function importTweets() {
$tweetIterator = new BatchingIterator( new BatchingTweetFetcher() );
$tweetIterator->setMaxBatchSize( 42 );
$this->tweetImporter->importTweets( $tweetIterator );
}
}
You can use Composer to download and install this package as well as its dependencies.
To add this package as a local, per-project dependency to your project, simply add a
dependency on jeroen/batching-iterator
to your project's composer.json
file.
Here is a minimal example of a composer.json
file that just defines a dependency on
BatchingIterator 3.x:
{
"require": {
"jeroen/batching-iterator": "~3.0"
}
}
BatchingIterator
now defaults to 10, avoiding usage of thejeroen/batching-iterator
instead of jeroen-de-dauw/batching-iterator
.MultipleBatchingFetcher
now accepts an array of BatchingFetcher
in its constructorBreaking changes:
rewind
method to the BatchingFetcher
interfaceBatchingIterator\InMemoryBatchingFetcher
to BatchingIterator\Fetchers\InMemoryBatchingFetcher
New features and enhancements:
BatchingIterator
can now be iterated over multiple timesMultipleBatchingFetcher
IteratorBasedBatchingFetcher
Initial release with
BatchingIterator
classBatchingFetcher
interfaceInMemoryBatchingFetcher
trivial implementation of BatchingFetcher