Imagine waiting 20 minutes just to preview your code changes. This was the reality for engineers before optimizing the deployment preview process. This blog post dives into how a staggering 98% reduction in data import times was achieved, making ephemeral environments truly ephemeral.
The Pain Point: Slow Data Imports
In December 2023, user interviews (think: developer experience check-in) revealed a major pain point: slow deployment previews. While the concept itself was a hit, the time to import de-identified data into ephemeral environments deterred users. These delays hindered developer productivity and overall workflow.
The Culprit: Seeding Data on Demand
The previous approach involved seeding data on demand when creating a preview environment. This meant relying on pg_restore for Postgres and similar tools for other datastores, leading to significant waiting times as data size increased.
The Solution: Velero Backups and PVC Snapshots
Several options were explored to optimize data imports, eventually landing on Velero, a powerful Kubernetes backup tool. Velero’s ability to back up any resource and restore it to a different namespace made it perfect for these needs.
Here’s the breakdown:
- AWS EBS CSI Driver: Manages EBS volumes used as storage for Kubernetes PVs.
- VolumeSnapshotClass: Creates snapshots of volumes at specific points in time.
- Velero Plugins: Integrate with AWS for backups, snapshots, and volume restoration.
- Snapshot Controller: Captures VolumeSnapshot events for automation.
Keeping Data Fresh
To ensure the restored data is up-to-date, the existing data anonymization process was adjusted. Now, data exports are regularly scheduled to a dedicated namespace. This namespace serves as the source for Velero backups, guaranteeing fresh data for ephemeral environments.
Resource Hooks for Orderly Execution
ArgoCD ApplicationSet Resource Hooks were implemented to ensure the correct order of operations:
- PreSync Hook: Triggers data import Jobs (no longer a bottleneck).
- PostSync Hook: Runs a Velero backup Job to capture a snapshot.
The Results: Lightning Fast Data Imports
The impact is undeniable. Data import times have plummeted from a staggering 19 minutes to a jaw-dropping 25 seconds. This translates to a massive boost in developer productivity and a smoother overall experience.
Beyond the Performance Gain
This optimization goes beyond speed. It also ensures:
- Data Integrity: Backups guarantee consistent data across ephemeral environments.
- Resource Cleanup: Automatic deletion of resources after an environment’s TTL expires.
Empowering Engineers with Faster Previews
Fast and reliable deployment previews are crucial for the Software Development Lifecycle (SDL). By overcoming data import bottlenecks, engineers are empowered to test changes quickly and confidently, ultimately leading to better software.
This is just one example of how the tools and processes are continuously improved. Stay tuned for future blog posts as we delve deeper into specific aspects of the development environment!
Reference to the Article- BlueGround
