Build and manage real-life ML, AI, and data science projects with ease!
APACHE-2.0 License
Published by savingoyal over 3 years ago
The Metaflow 2.2.12 release is a minor patch release.
Prior to this release, the State Machines created by Metaflow while deploying flows to AWS Step Functions had the same name as that of the flow. With this release, Metaflow users can now override the name of the State Machine created by passing in a --name
argument : python flow.py step-functions --name foo create
or python flow.py step-functions --name foo trigger
.
Metaflow now registers heartbeats at the run level and the task level for all flow executions (with the exception of flows running on AWS Step Functions where only task-level heartbeats are captured). This provides the necessary metadata to ascertain if a run/task has been lost. Subsequent releases of Metaflow will expose this information through the client.
Click >=8.0.x
The latest release of Click (8.0.0) broke certain idempotency assumptions in Metaflow which PR #526 addresses.
Published by romain-intel over 3 years ago
The Metaflow 2.2.11 release is a minor patch release.
shlex.quote
, introduced in #493, is not compatible with Python 2.7. pipes.quote
is now used for Python 2.7.
Some plugins may need to escape shell variables when using them in command lines. This patch allows this to work.
In some cases, a hard crash in a step would cause the status of the step to not be properly reported.
The Conda environment now delegates get_environment_info
to the DEFAULT_ENVIRONMENT
as opposed to the MetaflowEnvironment
. This does not change the current default behavior.
Published by savingoyal over 3 years ago
The Metaflow 2.2.10 release is a minor patch release.
For tasks that execute on AWS Batch, Metaflow now records the location where the AWS Batch instance writes the container logs in AWS Logs. This can be handy in locating the logs through the client API -
Step('Flow/42/a').task.metadata_dict['aws-batch-awslogs-group']
Step('Flow/42/a').task.metadata_dict['aws-batch-awslogs-region']
Step('Flow/42/a').task.metadata_dict['aws-batch-awslogs-stream']
PR: #478
All Metaflow runtime/task logs are now published via a sidecar process to the datastore. The user-visible logs on the console are streamed directly from the datastore. For Metaflow's integrations with the cloud (AWS at the moment), the compute tasks logs (AWS Batch) are directly written by Metaflow into the datastore (Amazon S3) independent of where the flow is launched from (User's laptop or AWS Step Functions). This has multiple benefits
python flow.py logs 42/start
or Step('Flow/42/start').task.stdout
. PR: #449ping/
endpoint for Metadata service
Fix a regression introduced in v2.2.9
where the endpoint responsible for ascertaining the version of the deployed Metadata service was erroneously moved to ping/
from ping
PR: #484
--namespace=
CLI args when executing a flow
python flow.py run --namespace=
now correctly makes the global namespace visible within the flow execution. PR: #461
Published by romain-intel over 3 years ago
The Metaflow 2.2.9 release is a minor patch release.
/
in image parameter for batchPylint dependency was unpinned and made floating. See PR #462.
/
in image parameter for batch
You are now able to specify docker images of the form foo/bar/baz:tag
in the batch decorator. See PR #466.
The order in which parameters are specified by the user in the FlowSpec is now preserved when displaying them with --help
. See PR #456.
Published by savingoyal over 3 years ago
The Metaflow 2.2.8 release is a minor patch release.
@environment
behavior for conflicting attribute values
Metaflow was incorrectly handling environment variables passed through the @environment
decorator in some specific instances. When @environment
decorator is specified over multiple steps, the actual environment that's available to any step is the union of attributes of all the @environment
decorators; which is incorrect behavior. For example, in the following workflow -
from metaflow import FlowSpec, step, batch, environment
import os
class LinearFlow(FlowSpec):
@environment(vars={'var':os.getenv('var_1')})
@step
def start(self):
print(os.getenv('var'))
self.next(self.a)
@environment(vars={'var':os.getenv('var_2')})
@step
def a(self):
print(os.getenv('var'))
self.next(self.end)
@step
def end(self):
pass
if __name__ == '__main__':
LinearFlow()
var_1=foo var_2=bar python flow.py run
will result in
Metaflow 2.2.7.post10+gitb7d4c48 executing LinearFlow for user:savin
Validating your flow...
The graph looks good!
Running pylint...
Pylint is happy!
2021-03-12 20:46:04.161 Workflow starting (run-id 6810):
2021-03-12 20:46:04.614 [6810/start/86638 (pid 10997)] Task is starting.
2021-03-12 20:46:06.783 [6810/start/86638 (pid 10997)] foo
2021-03-12 20:46:07.815 [6810/start/86638 (pid 10997)] Task finished successfully.
2021-03-12 20:46:08.390 [6810/a/86639 (pid 11003)] Task is starting.
2021-03-12 20:46:10.649 [6810/a/86639 (pid 11003)] foo
2021-03-12 20:46:11.550 [6810/a/86639 (pid 11003)] Task finished successfully.
2021-03-12 20:46:12.145 [6810/end/86640 (pid 11009)] Task is starting.
2021-03-12 20:46:15.382 [6810/end/86640 (pid 11009)] Task finished successfully.
2021-03-12 20:46:15.563 Done!
Note the output for the step a
which should have been bar
. PR #452 fixes the issue.
environment is not callable
error when using @environment
Using @environment
would often result in an error from pylint
- E1102: environment is not callable (not-callable)
. Users were getting around this issue by launching their flows with --no-pylint
. PR #451 fixes this issue.
Published by savingoyal over 3 years ago
The Metaflow 2.2.7 release is a minor patch release.
Workflows orchestrated by AWS Step Functions were failing to properly execute for-each
steps on AWS Fargate. The culprit was lack of access to instance metadata for ECS. Metaflow instantiates a connection to Amazon DynamoDB to keep track of for-each
cardinality. This connection requires knowledge of the region that the job executes in and is made available via instance metadata on EC2; which unfortunately is not available on ECS (for AWS Fargate). This fix introduces the necessary checks for inferring the region correctly for tasks executing on AWS Fargate. Note that after the recent changes to Amazon S3's consistency model, the Amazon DynamoDB dependency is no longer needed and will be done away in a subsequent release. PR: #436
Published by savingoyal almost 4 years ago
The Metaflow 2.2.6 release is a minor patch release.
At AWS re:invent 2020, AWS announced support for AWS Fargate as a compute backend (in addition to EC2) for AWS Batch. With this feature, Metaflow users can now submit their Metaflow jobs to AWS Batch Job Queues which are connected to AWS Fargate Compute Environments as well. By setting the environment variable - METAFLOW_ECS_FARGATE_EXECUTION_ROLE
, users can configure the ecsTaskExecutionRole for the AWS Batch container and AWS Fargate agent. PR: #402
shared_memory
, max_swap
, swappiness
attributes for Metaflow tasks launched on AWS Batch
The @batch
decorator now supports shared_memory
, max_swap
, swappiness
attributes for Metaflow tasks launched on AWS Batch to provide a greater degree of control for memory management. PR: #408
The tag metaflow_version:
and runtime:
is now available for all packaged executions and remote executions as well. This ensures that every run logged by Metaflow will have metaflow_version
and runtime
system tags available. PR: #403
Run
objects generated through AWS Step Functions executions
Run
objects generated by flows executed on top of AWS Step Functions were missing the tags assigned to the flow; even though the tags were correctly persisted to tasks. This release fixes and brings inline the tagging behavior as observed with local flow executions. PR: #386
stderr
Execution set-up logs for @conda
and IncludeFile
were being piped to stdout
which made manipulating the output of commands like python flow.py step-functions create --only-json
a bit difficult. This release moves the workflow set-up logs to stderr
. PR: #379
IncludeFile
properly
A workflow executed without a required IncludeFile
parameter would fail when the parameter was referenced inside the flow. This release fixes the issue by assigning a null value to the parameter in such cases. PR: #421
Published by savingoyal almost 4 years ago
The Metaflow 2.2.5 release is a minor patch release.
metaflow_version:
and runtime:
tag for all executionsmetaflow_version:
and runtime:
tag for all executionsThe tag metaflow_version:
and runtime:
is now available for all packaged executions and remote executions as well. This ensures that every run logged by Metaflow will have metaflow_version
and runtime
system tags available. PR: #376, #375
Conda fails to correctly set up environments for linux-64 packages on macOS at times due to inconsistently cased filesystems. Environment creation is needed to collect the necessary metadata for correctly setting up the conda environment on AWS Batch. This fix simply ignores the error-checks that conda throws while setting up the environments on macOS when the intended destination is AWS Batch. PR: #377
Published by savingoyal almost 4 years ago
The Metaflow 2.2.4 release is a minor patch release.
curl
to requests
AWS GovCloud & AWS CN users can now enjoy all the features of Metaflow within their region partition with no change on their end. PR: #364
Metaflow v2.1.0 introduced a bug in IncludeFile functionality which prevented users from overriding the default value specified. PR: #346
curl
to requests
Metaflow's AWS Step Functions' integration relies on AWS DynamoDb to manage foreach constructs. Metaflow was leveraging curl
at runtime to detect the region for AWS DynamoDb. Some docker images don't have curl
installed by default; moving to requests
(a metaflow dependency) fixes the issue. PR: #343
Published by jasonge27 about 4 years ago
The Metaflow 2.2.3 release is a minor patch release.
Fix the issue where default help
for parameters was not handled properly. #305 Flow fails because IncludeFile
's default value for the help
argument is None. PR: #318
The previously pinned library version does not work with python 3.8. Now we have two sets of different version combinations which should work for python 2.7, 3.5, 3.6, 3.7, and 3.8. PR: #308
Previously the executable installed in conda environment was not visible inside metaflow steps. Fixing this issue by appending conda bin path to the PATH environment variable PR: #307
A typo fix. PR: #304
Published by romain-intel about 4 years ago
The Metaflow 2.2.2 release is a minor patch release.
Metaflow 2.2.1 included a commit which was merged too early and broke the use of Conda. This release reverses this patch.
Recent versions of Pandas are not backward compatible with the one used in the tutorial; a small comment was added to warn of this fact.
In some cases, the metadata service would not properly create runs or tasks.
PRs #296, #297, #298
Published by romain-intel about 4 years ago
The Metaflow 2.2.1 release is a minor patch release.
include
parameter to merge_artifacts
.include
parameter for merge_artifacts
You can now specify the artifacts to be merged explicitly by the merge_artifacts
method as opposed to just specifying the ones that should not be merged.
Fixes the regression described in #285.
In some cases, Conda is installed system wide and the user cannot write to its installation directory. This was causing issues when trying to use the Conda environment. Fixes #179.
Retries were not properly handled when uploading artifacts to the S3 datastore. This fix addresses this issue.
PRs #282, #286, #287, #288, #289, #290, #291
Published by savingoyal about 4 years ago
The Metaflow 2.2.0 release is a minor release and introduces Metaflow's support for R lang.
This release provides an idiomatic API to access Metaflow in R lang. It piggybacks on the Pythonic implementation as the backend providing most of the functionality previously accessible to the Python community. With this release, R users can structure their code as a metaflow flow. Metaflow will snapshot the code, data, and dependencies automatically in a content-addressed datastore allowing for resuming of workflows, reproducing past results, and inspecting anything about the workflow e.g. in a notebook or RStudio IDE. Additionally, without any changes to their workflows, users can now execute code on AWS Batch and interact with Amazon S3 seamlessly.
PR #263 and PR #214 .
Published by savingoyal about 4 years ago
The Metaflow 2.1.1 release is a minor patch release.
/step
endpoint of metadata service./step
endpoint of metadata service.The foreach
step in AWS Step Functions launches multiple AWS Batch tasks, each of which tries to register the step metadata, if it already doesn't exist. This can result in a race condition and cause the task to fail. This patch properly handles the 409 response from the service.
PR #258 & PR #260
Published by savingoyal about 4 years ago
The Metaflow 2.1.0 release is a minor release and introduces Metaflow's integration with AWS Step Functions.
Netflix uses an internal DAG scheduler to orchestrate most machine learning and ETL pipelines in production. Metaflow users at Netflix can seamlessly deploy and schedule their flows to this scheduler. Now, with this release, we are introducing a similar integration with AWS Step Functions where Metaflow users can easily deploy & schedule their flows by simply executing
python myflow.py step-functions create
which will create an AWS Step Functions state machine for them. With this feature, Metaflow users can now enjoy all the features of Metaflow along with a highly available, scalable, maintenance-free production scheduler without any changes in their existing code.
We are also introducing a new decorator - @schedule
, which allows Metaflow users to instrument time-based triggers via Amazon EventBridge for their flows deployed on AWS Step Functions.
With this integration, Metaflow users can inspect their flows deployed on AWS Step Functions as before and debug and reproduce results from AWS Step Functions on their local laptop or within a notebook.
Documentation
Launch Blog Post
PR #211 addresses Issue #2 .
Metaflow was inadvertently removing leading whitespace from user-visible logs on the console. Now Metaflow presents user-visible logs with the correct formatting.
PR #244 fixed issue #223.
Due to malformed permissions, AWS Batch might not be able to fetch the code package from Amazon S3 for user code execution. In such scenarios, it wasn't apparent to the user, where the code package was being pulled from, making triaging any permission issue a bit difficult. Now, the Amazon S3 file location is part of the exception stack trace.
PR #243 fixed issue #232.
Metaflow uses time
to store the created_at
and finished_at
information for the Run
object returned by Metaflow client. time
unfortunately does not support the %f
directive, making it difficult to parse these fields by datetime
or time
. Since Metaflow doesn't expose timings at millisecond grain, this PR drops the %f
directive.
PR #227 fixed issue #224.
When launching jobs on AWS Batch, the CloudWatchLogStream might not be immediately created (and may never be created if say we fail to pull the docker image for any reason whatsoever). Metaflow will now simply retry again next time.
PR #209.
Published by savingoyal over 4 years ago
datatools.S3._read_many_files
.pylint
version to < 2.5.0
for compatibility issues.The Metaflow 2.0.5 release is a minor patch release.
Avoid a cryptic error message when datatools.S3._read_many_files
is unsuccessful by converting prefixes
from a generator to a list.
Modify the retry behavior for log fetching on AWS Batch by adding jitters to exponential backoffs as well as reset the retry counter for every successful request.
Additionally, fail the metaflow task when we fail to stream the task logs back to the user's terminal even if AWS Batch task succeeds.
pylint
version 2.5.0
would mark Metaflow's self.next()
syntax as an error. As a result, python helloworld.py run
would fail at the pylint check step unless we run with --no-pylint
. This version upper-bound is supposed to automatically downgrade pylint
during metaflow
installation if pylint==2.5.0
has been installed.
Published by savingoyal over 4 years ago
retry_count
in Current
ThrottleExceptions
in AWS Batch job logsDescribeJobs
API for AWS BatchPYTHONNOUSERSITE
for conda
environmentsRUNNING
stateThe Metaflow 2.0.4 release is a minor patch release.
retry_count
in Current
You can now use the current
singleton to access the retry_count
of your task. The first attempt of the task will have retry_count
as 0 and subsequent retries will increment the retry_count
. As an example:
@retry
@step
def my_step(self):
from metaflow import current
print("retry_count: %s" % current.retry_count)
self.next(self.a)
ThrottleExceptions
in AWS Batch job logsThe AWS Logs API for get_log_events
has a global hard limit on 10 requests per sec. While we have retry logic in place to respect this limit, some of the ThrottleExceptions
usually end up in the job logs causing confusion to the end-user. This release addresses this issue (also documented in #184).
DescribeJobs
API for AWS BatchThe AWS Batch API for describe_jobs
throws ThrottleExceptions
when managing a flow with a very wide for-each
step. This release adds retry behavior with backoffs to add proper resiliency (addresses #138).
PYTHONNOUSERSITE
for conda
environmentsIn certain user environments, to properly isolate conda
environments, we have to explicitly override PYTHONNOUSERSITE
rather than simply relying on python -s
(addresses #178).
RUNNING
stateFixes a bug where if the AWS Batch job crashes before entering the RUNNING
state (often due to incorrect IAM perms), the previous log collection behavior would fail to print the correct error message making it harder to debug the issue (addresses #185).
Published by romain-intel over 4 years ago
The Metaflow 2.0.3 release is a minor patch release.
You can now use the current
singleton (documented here) to access the names of the parameters passed into your flow. As an example:
for var in current.parameter_names:
print("Parameter %s has value %s" % (var, getattr(self, var))
This addresses #137.
A few issues were addressed to improve the usability of Metaflow. In particular, show
now properly respects indentation making the description of steps and flows more readable. This addresses #92. Superfluous print messages were also suppressed when executing on AWS batch with the local metadata provider (#152).
A smaller, newer and standalone Conda installer is now used resulting in faster and more reliable Conda bootstrapping (#123).
We now check for the command line --datastore-root
prior to using the environment variable METAFLOW_DATASTORE_SYSROOT_S3
when determining the S3 root (#134). This release also fixes an issue where using the local Metadata provider with AWS batch resulted in incorrect directory structure in the .metaflow
directory (#141).
Published by romain-intel over 4 years ago
Published by savingoyal almost 5 years ago
Enhancements
metaflow configure [import|export]
for importing/exporting Metaflow configurations.metaflow configure aws
command to address usability concerns.Bug Fixes