article thumbnail

Super charge your LLMs with RAG at scale using AWS Glue for Apache Spark

AWS Machine Learning Blog

This enables you to preprocess your external data in the phases including cleaning, sanitization, chunking documents, generating vector embeddings for each chunk, and loading into a vector store. About the Authors Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team.

LLM 107
article thumbnail

Introducing generative AI troubleshooting for Apache Spark in AWS Glue (preview)

Flipboard

About the Authors Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike. Vishal Kajjam is a Software Development Engineer on the AWS Glue team.