Problem Description

You are tasked with building a provider-agnostic Command Line Interface (CLI) tool. This tool will enable users to request and manage individual GPU virtual machines (VMs) and fleets of machines across three different mock cloud providers: Crusoe Cloud, Lambda Cloud, and Nebius AI Cloud.

Background: Cloud Providers

Your company utilizes GPU VMs from the following three providers, each with distinct API characteristics and resource models:

• Crusoe Cloud
  • Uses a REST API.
  • Resources are scoped to specific projects.
  • Operations are asynchronous.
  • Supports VM reservations.
  • Lifecycle operations include reboot, reset, and restart semantics.
• Lambda Cloud
  • Uses a REST API.
  • Features a flat and simple resource model.
  • Operations are mostly synchronous.
  • Supports reserved instances.
• Nebius AI Cloud
  • Uses a gRPC interface.
  • Resources are parent-scoped.
  • Operations are asynchronous.
  • Supports reservation policies.

Layer 1: Unified CLI for Individual VMs

Develop a CLI that offers a unified interface for managing individual VMs across all three providers.

Required commands:

# List all instances across all providers, or filter by provider
vm list [--provider <name>]

# Create new instances
vm create --provider <name> --gpu <type> --count <n> [--name <name>] [--region <region>]

# Get instance details
vm get <instance_id> --provider <name>

# Stop an instance
vm stop <instance_id> --provider <name>

# Start an instance
vm start <instance_id> --provider <name>

# Destroy or terminate an instance
vm destroy <instance_id> --provider <name>

Layer 2: Fleet Manager

Extend the functionality from Layer 1 to support fleet operations. A "fleet" is defined as a logical group of VMs that can be distributed across multiple providers.

Required commands:

# Request N machines of a given GPU type, spread across providers
vm fleet create --gpu <type> --count <n> [--name <fleet_name>]

# List all fleets
vm fleet list

# Show fleet status, including VMs, providers, and states
vm fleet status <fleet_name>

# Destroy an entire fleet
vm fleet destroy <fleet_name>

Reservation Behavior

Each provider implements reservation mechanisms differently:

• Crusoe Cloud
  • VMs are automatically assigned to the cheapest matching reservation if available.
  • Specifying an explicit reservation_id is optional.
  • Stopping a VM releases its reserved capacity; starting it attempts to reclaim capacity.
• Lambda Cloud
  • Instances can be flagged with is_reserved: true.
  • Reserved instances cannot be terminated via the API.
  • Launching an instance with a reservation_id consumes the specified reserved capacity.
• Nebius AI Cloud
  • Reservation behavior is controlled by a ReservationPolicy within the instance specification:
    • AUTO: The system attempts to use a reservation first.
    • FORBID: The system explicitly uses on-demand capacity.
    • STRICT: The system must use a specific reservation, failing if it cannot.

Follow-up Discussion

Be prepared to discuss the following points:

1. What are the major API differences between the three providers, and how would your code handle them?
2. How would you store the final set of machines that belong to a fleet?
3. How would you clean up partially created machines if fleet creation fails midway?
4. What problems could occur during implementation or production use, and how would you mitigate them?