[anchapin]

Summary

Closes #892

Introduces comprehensive Head Node and Headless operational topologies for OpenStudio Server external batch runs on AWS Batch, including an optional AWS Step Functions orchestration layer for fire-and-forget client workflows, automated serverless post-processing fan-in, and ephemeral control-plane execution.

---

What Changes

1. AWS Step Functions Orchestration Layer

• ASL State Machine (external_batch/aws/orchestration/state_machine.asl.json):
  • Manages input validation, chunk count branching (NumChunks >= 2 vs NumChunks == 1), and AWS Batch array job dispatch using arn:aws:states:::batch:submitJob.sync.
  • Built-in retry policies for transient Batch server exceptions and Spot node reclaims with exponential backoff.
  • Error budget evaluation (AllowPartialFailure).
  • Automated Cloud Finalizer task running directly in S3 upon array completion.
  • Optional Amazon SNS completion notification.
• Terraform Module (external_batch/aws/infra/step_functions.tf):
  • Controlled by variable "enable_step_functions" { default = false }.
  • IAM execution role with least-privilege permissions (batch:SubmitJob, batch:DescribeJobs, S3, CloudWatch Logs, and EventBridge rules).
  • Supports enterprise pre-provisioned roles via existing_step_functions_role_arn.

2. Cloud Head Node Deployment (Option B & B+)

• Terraform Module (external_batch/aws/infra/head_node.tf):
  • Provisions an optional EC2 instance (t3.large/c6i.xlarge) with OpenStudio Server, security groups (ports 8080, 3000, 22), and IAM instance profile.
• Ephemeral Head Node Architecture (Option B+):
  • Slashes monthly EC2 baseline from $65–$128/mo down to ~$0.93–$1.85 / month (assuming 3 runs/month) by running only during active simulation windows.
  • Bypasses the 28.5-hour serial MongoDB ingestion bottleneck for 100k+ runs by using EventBridge webhook callbacks (POST /analyses/:id/complete_external_batch) to register summary S3 dataset links.

3. Headless Deployment & Diagnostic Tools (Option A & A+)

• Lightweight Results Viewer (external_batch/aws/viewer/index.html):
  • Single-file, zero-dependency HTML/JS dashboard with real-time status bars, filtering, search, and modal diagnostics (status.json, run.log, eplusout.err, measure_attributes.json).
• Headless Aggregator (external_batch/demo/aggregate_csv.rb):
  • Single-pass Ruby script synthesizing all completed datapoint measure_attributes.json files into a unified ML/surrogate-ready CSV.
• Container Finalizer Integration:
  • Enhanced external_batch/aws/task_wrapper.sh and Dockerfile to support BATCH_TASK_MODE=finalize for in-cloud aggregation.

4. CLI & Server Auto-Dispatch

• Enhanced external_batch/aws/submit_batch.rb with --step-functions ARN, --wait, --allow-partial-failure, and --sns-topic ARN.
• Enhanced server/app/lib/analysis_library/external_batch_run.rb with native Step Functions auto-dispatch.
• Created demo test harnesses in external_batch/demo/ (run_100_headless.rb, run_100_head_node.rb, run_100_step_functions.rb).

5. Benchmark & Scaling Documentation for Brian Ball

• Detailed empirical evaluation in external_batch/aws/BRIAN_BALL_DEMO_RESULTS.md and external_batch/aws/HEAD_VS_HEADLESS.md:
  • 100-simulation empirical benchmark on Fargate Spot.
  • Scaled 100,000-simulation projections and cost models for 3 runs/month.

---

Testing

• RSpec Suite: All 9 tests passing in server/spec/models/external_batch_aws_spec.rb.
• Terraform Validation: terraform validate passed with 0 errors.
• Empirical Execution: Successfully executed 100-simulation SEB LHS benchmarks across both Head and Headless Step Functions modes.