AWS Machine Learning Blog

APRIL 21, 2023

If the model performs acceptably according to the evaluation criteria, the pipeline continues with a step to baseline the data using a built-in SageMaker Pipelines step. For the data drift Model Monitor type, the baselining step uses a SageMaker managed container image to generate statistics and constraints based on your training data.

Data Drift 98

Data Drift Metadata Data Quality ML 98

The MLOps Blog

JUNE 27, 2023

Databricks Databricks is a cloud-native platform for big data processing, machine learning, and analytics built using the Data Lakehouse architecture. Delta Lake Delta Lake is an open-source storage layer that provides reliability, ACID transactions, and data versioning for big data processing frameworks such as Apache Spark.

Metadata 134

Metadata Machine Learning Data Scientist Data Quality 134

AWS Machine Learning Blog

NOVEMBER 29, 2023

For instance, a notebook that monitors for model data drift should have a pre-step that allows extract, transform, and load (ETL) and processing of new data and a post-step of model refresh and training in case a significant drift is noticed. Run the notebooks The sample code for this solution is available on GitHub.

Data Drift 110

Data Drift BERT Data Scientist Python 110

AWS Machine Learning Blog

AUGUST 29, 2023

The batch inference pipeline includes steps for checking data quality against a baseline created by the training pipeline, as well as model quality (model performance) if ground truth labels are available. If the batch inference pipeline discovers data quality issues, it will notify the responsible data scientist via Amazon SNS.

Data Scientist 85

Data Scientist Data Quality Python ML 85

Snorkel AI

JULY 5, 2023

We thought we’d structure this more as a conversation where we walk you through some of our thinking around some of the most common themes in data centricity in applied AI. Is more data always better? One of them is that it is really hard to maintain high data quality with rigorous validation.

Large Language Models 52

Large Language Models Metadata Machine Learning AI 52

Snorkel AI

JULY 5, 2023

We thought we’d structure this more as a conversation where we walk you through some of our thinking around some of the most common themes in data centricity in applied AI. Is more data always better? One of them is that it is really hard to maintain high data quality with rigorous validation.

Large Language Models 52

Large Language Models Metadata AI AI 52

Snorkel AI

JULY 5, 2023

We thought we’d structure this more as a conversation where we walk you through some of our thinking around some of the most common themes in data centricity in applied AI. Is more data always better? One of them is that it is really hard to maintain high data quality with rigorous validation.

Large Language Models 52

Large Language Models Metadata AI AI 52