2025 FOSS4G NA | ReMaking Cirrus - Jarrett Keifer
In this session from FOSS4G NA 2025, Jarrett Keifer of Element 84 presents ReMaking Cirrus: Reflecting on Five Years of Open Source Data Orchestration. Jarrett takes the audience through the evolution of Cirrus, a specialized framework designed to handle massive geospatial data processing pipelines—from ingest and metadata generation to indexing into searchable STAC catalogs.
Jarrett shares the project's journey from its roots in NASA's Cumulus project to its current state as a lightweight, STAC-native orchestrator used to manage over 30 petabytes of data. He provides a candid look at the technical "pendulum swings" the project has experienced, including the bold decision to remove all infrastructure-as-code dependencies and serverless framework logic in response to licensing changes, while distilling five years of development into three key lessons on funding, project vision, and managing technical debt.
Highlights:
🛰️ What is Cirrus? An introduction to the data orchestration component of Element 84's Filmpop suite for building geospatial data lakes
🔄 The Evolution of STAC Workflows: How Cirrus uses STAC items as the primary message format to track state across distributed processing tasks
🧩 Scaling to Petabytes: Managing massive archives like Sentinel-2 with a system that handles 30,000+ items a day and nearly 80 million total items
⚙️ The Serverless Redesign: Why Element 84 deleted 4,400 lines of code to remove closed-source build dependencies and move toward a "bring-your-own-IaC" model
🌐 Lessons in Open Source: Reflections on how funding sources, project vision, and inevitable failures shape the long-term success of a software project
🏢 From Cirrus to Swoop: A look at a cloud-agnostic "meta-orchestrator" experiment designed for Kubernetes and Argo Workflows
🏗️ Operational Reliability: Why making failures cheap, visible, and easy to resolve is a core design principle for high-volume data pipelines
For more content like this check out www.projectgeospatial.com
#Geospatial #FOSS4G #DataOrchestration #STAC #Cirrus #OpenSource #AWS #CloudNative #DataEngineering #Python #Serverless #Element84 #ProjectGeospatial #DataLake #SoftwareDevelopment
