Scaling Zero Copy from 1 Trillion to 120 Trillion Rows with File Federation
Summary
Zero Copy technology at Salesforce evolved from Query Federation to File Federation to handle exploding data volumes from under 1 trillion to 120 trillion rows monthly. By shifting from dual-query execution to directly accessing shared storage through Apache Iceberg, it removes bandwidth and compute bottlenecks, enabling AI workloads at petabyte scale without forcing data centralization. This architecture maintains customer data governance while significantly improving scalability and performance of distributed enterprise data access. Salesforce teams can leverage this pattern to build more efficient, scalable AI-integrated data solutions across complex, multi-platform environments.
Takeaways
- Leverage File Federation to scale Zero Copy by unifying storage access via Apache Iceberg.
- Avoid costly data movement by operating compute closer to distributed data storage layers.
- Coordinate with ecosystem partners to align on common storage formats for interoperability.
- Implement temporary credentials and catalog layers to safeguard data governance.
- Use comprehensive observability and testing to handle scaling challenges across distributed environments.
In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we spotlight Srini Krishnamoorthy, Vice President of Engineering for Data 360. Srini leads the evolution of Zero Copy across Data 360 and Agentforce. Zero Copy was originally designed to eliminate data movement. Then customers began using it to power AI workloads across increasingly large volumes of distributed enterprise data. What started as a data access problem quickly became a distributed systems problem, forcing the team to rethink the architecture behind Query Federation as adoption accelerated from less than 1 trillion rows per month to approximately 120 trillion rows monthly. Explore how the team evolved Zero Copy from a Query Federation architecture into a File Federation architecture capable of supporting AI at petabyte scale without requiring customers to centralize their data.