How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark
Google Cloud Blog · 软件开发
As a global media and technology company connecting hundreds of millions of users to finance, sports, and entertainment platforms, Yahoo operates a massive data infrastructure where analytics workloads must run continuously at high speed. In deadline-driven data environments, relying on fixed virtual machine (VM) configurations creates a brittle system; if a specific machine shape faces a regional capacity constraint, cluster provisioning in
Managed Service for Apache Spark
(formerly Dataproc) can experience delays and stall critical data pipelines.
Yahoo utilizes
flexible VMs
in
Managed Service for Apache Spark
clusters to automatically absorb these resource fluctuations by defining a ranked list of acceptable VM shapes. This allows the system to dynamically search regional zones and maintain pipeline execution without manual intervention. To search for capacity across a region, teams must also enable
Auto-Zone placement
.
This optimization builds on Yahoo's broader data modernization journey, which involved
migrating on-premises Hadoop and big data estates
directly to Google Cloud. By transitioning those legacy workloads, the team established a cloud foundation capable of running high-scale batch and streaming analytics with dynamic resource flexibility.
Hadoop pioneer to cloud innovator: Yahoo’s data lake modernization journey
This post provides a technical blueprint for configuring flexible VM instance rankings in
Managed Service for Apache Spark
to automatically manage capacity constraints and maintain pipeline execution.
Operational trade-offs of static configurations
Configuring clusters with a single, fixed machine type in a specific zone introduces constraints when regional zonal capacity fluctuations occur, potentially impacting cluster provisioning. Rather than manage these capacity variations through custom retry logic or manual intervention, using flexible configurations allows your infrastructure to automatically adapt. By accepting multiple VM shapes and searching across zones in the selected region, flexible configurations help streamline provisioning to better support high-scale analytics workloads.
Rules for configuring flexible clusters
Deploying flexible configurations requires aligning several connected design choices:
Enable auto-zone placement:
You must pass a region(
--region=${REGION}
) or an empty zone string (
--zone=""
) so Managed Spark can search for available capacity across the entire region.
Maintain core and memory symmetry:
If your Managed Spark cluster uses
autoscaling
, all machine types in your flexible list must share a similar core count and memory size, even if they come from different VM families. A uniform CPU-to-memory ratio across primary and secondary workers prevents performance degradation, as the smallest ratio determines your effective container sizing.
Align component properties:
Managed Spark calculates system properties based on VM cores and memory. When mixing machine shapes, you may need explicit property overrides to keep YARN and Spark resource allocations aligned with your expected worker behavior.
Two ways flexible VMs support massive workloads
For large-scale data environments, flexible configurations support operations in two ways:
Higher cluster creation success:
Instead of failing when a preferred VM type is out of stock, Managed Spark selects from a ranked list to keep provisioning moving.
Better regional resource use:
Auto-zone placement searches the entire region to find capacity, which reduces provisioning friction during high-demand periods.
gcloud example
code_block
<ListValue: [StructValue([('code', 'gcloud dataproc clusters create analytics-cluster \\\r\n --region=us-central1 \\\r\n --zone="" \\\r\n --num-workers=10 \\\r\n --master-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --master-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f52906f29d0>)])]>
API example
You can also build this capacity policy into your automated pipelines or
Managed Service for Apache Airflow
DAGS using the
instanceFlexibilityPolicy
field in the ‘Dataproc’ API:
code_block
<ListValue: [StructValue([('code', '{\r\n "projectId": "PROJECT_ID",\r\n "clusterName": "analytics-cluster",\r\n "config": {\r\n "gceClusterConfig": {\r\n "zoneUri": ""\r\n },\r\n "secondaryWorkerConfig": {\r\n "numInstances": 8,\r\n "instanceFlexibilityPolicy": {\r\n "instanceSelectionList": [\r\n {\r\n "machineTypes": ["n2-standard-8"],\r\n "rank": 0\r\n },\r\n {\r\n "machineTypes": ["e2-standard-8", "t2d-standard-8"],\r\n "rank": 1\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f52906f2190>)])]>
This API policy achieves the same goal: it establishes your preferred shape, documents valid fallbacks, and lets Managed Spark resolve resource constraints without breaking your automation scripts.
Establishing an infrastructure policy
Managing data at this scale requires standardizing a clear resource policy rather than relying on a single rigid machine type. Your configuration standards should outline:
Preferred and fallback VM families for secondary workers.
Default auto-zone placement to enable flexible provisioning.
Identical core and memory configurations when using autoscaling.
Uniform CPU-to-memory ratios across all worker groups to maintain predictable container sizing.
Explicit YARN or Spark property overrides to guarantee consistent runtime behavior across different machine lines.
Shuffle-safe patterns for Spark workloads running on Spot or highly elastic capacity.
By adopting flexible configurations, you turn infrastructure scarcity into a predictable fallback plan, keeping your critical data pipelines up and running.