Implement E2E encrypted multi-tenant network architecture (WireGuard/VXLAN/BGP EVPN/L3VPN) - #5
Conversation
- Fix src/network.rs: replace boringtun/getrandom/x25519_dalek (missing deps) with x25519-dalek v2 + rand; add WireguardKeypair, generate_wireguard_keypair(), derive_public_key(), generate_client_config(), and fixed config parser - Add pub mod network to lib.rs and main.rs - Add base64, rand, x25519-dalek deps to Cargo.toml - Extend src/api/vyos.rs with VyOS HTTP API methods: configure_set/delete, configure_wireguard_interface/peer, configure_vxlan, configure_bgp_system/evpn_peer, enable_bgp_evpn, configure_vrf, configure_vrrp, provision_tenant, get_bgp_summary/vxlan_status/vrf_routes/wireguard_status - Add src/models/tenant.rs: Tenant, VrfConfig, VxlanConfig, TenantWireguardConfig, VrrpConfig - Add src/services/tenant.rs: TenantService and TenantStorage - Add Tenants and Wireguard CLI subcommands to main.rs Co-authored-by: danielbodnar <1790726+danielbodnar@users.noreply.github.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
There was a problem hiding this comment.
Pull request overview
Introduces tenant modeling/provisioning primitives and expands the VyOS client + CLI to support multi-tenant networking workflows (WireGuard/VXLAN/VRF/VRRP).
Changes:
- Add
Tenant/TenantServicewith in-memory tenant storage and router provisioning flow. - Refactor
src/network.rsinto WireGuard key/config utilities (keypair generation, pubkey derivation, client config generation, config parsing). - Extend
VyOSClientwith/configurehelpers and higher-level methods for WireGuard/VXLAN/BGP/VRF/VRRP + tenant provisioning.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| src/services/tenant.rs | New in-memory tenant storage + provisioning service integrating with ProviderService/VyOS. |
| src/services/mod.rs | Exposes the new tenant service module. |
| src/network.rs | Replaces tunnel implementation with WireGuard key/config utility functions. |
| src/models/tenant.rs | New tenant domain model including VRF/VXLAN/WireGuard/VRRP configuration fields. |
| src/models/mod.rs | Exposes the new tenant model module. |
| src/main.rs | Adds tenants and wireguard CLI subcommands and exposes the root network module. |
| src/lib.rs | Exposes the root network module for library consumers. |
| src/api/vyos.rs | Adds configure/show helpers and higher-level configuration/provisioning methods. |
| Cargo.toml | Adds crypto/keygen dependencies (base64, rand, x25519-dalek). |
| Cargo.lock | Locks new dependencies and their transitive crates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| .ok_or_else(|| anyhow!("API key is required for HTTP API operations"))?; | ||
| let client = self.http_client.as_ref().unwrap(); | ||
| let url = format!("https://{}:{}/configure", self.config.host, self.config.api_port); | ||
| debug!("VyOS configure op: POST {} {:?}", url, body); |
| self.configure_set( | ||
| &["interfaces", "wireguard", interface, "port"], | ||
| Some(&port.to_string()), |
| self.configure_set( | ||
| &["interfaces", "wireguard", interface, "peer", peer_name, "persistent-keepalive"], | ||
| Some(&ka.to_string()), |
| let iface = format!("vxlan{}", vni); | ||
| self.configure_set( | ||
| &["interfaces", "vxlan", &iface, "vni"], | ||
| Some(&vni.to_string()), | ||
| ).await?; |
| self.configure_set( | ||
| &["interfaces", "vxlan", &iface, "mtu"], | ||
| Some(&mtu.to_string()), | ||
| ).await?; |
| self.configure_set( | ||
| &["high-availability", "vrrp", "group", group_id, "vrid"], | ||
| Some(&vrid.to_string()), | ||
| ).await?; | ||
| self.configure_set( | ||
| &["high-availability", "vrrp", "group", group_id, "priority"], | ||
| Some(&priority.to_string()), | ||
| ).await?; |
| self.configure_wireguard_interface( | ||
| &wg_iface, | ||
| &tenant.wireguard.address, | ||
| &tenant.wireguard.private_key, | ||
| tenant.wireguard.port, | ||
| Some(&format!("Tenant {} WireGuard", tenant.name)), |
| pub fn generate_wireguard_keypair() -> Result<WireguardKeypair> { | ||
| let mut rng = rand::thread_rng(); | ||
| let mut secret_bytes = [0u8; 32]; | ||
| rng.fill_bytes(&mut secret_bytes); | ||
|
|
||
| let secret = StaticSecret::from(secret_bytes); | ||
| let public = PublicKey::from(&secret); |
| let vni = 10000 + tenant_id; | ||
| let table_id = 1000 + tenant_id; | ||
| let rt = format!("{}:{}", bgp_as, table_id); | ||
| let wg_address = format!("100.64.{}.1/24", tenant_id); | ||
| let network_cidr = format!("100.65.{}.0/24", tenant_id); | ||
|
|
| public_key: wg_public_key, | ||
| private_key: wg_private_key, | ||
| // Keep port in the valid range (51820–52819) using modulo 1000 | ||
| port: 51820 + (tenant_id as u16 % 1000), |
|
@copilot resolve the merge conflicts in this pull request |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (2)
src/models/tenant.rs:162
tenant_id as u16 % 1000truncatestenant_idbefore the modulo. For large tenant IDs this will wrap and produce unexpected ports; compute modulo inu32first and only then cast tou16.
// Keep port in the valid range (51820–52819) using modulo 1000
port: 51820 + (tenant_id as u16 % 1000),
src/api/vyos.rs:239
- This debug log prints the full
/configurerequest body. That body can include WireGuard private keys (private-key) and should not be written to logs. Log only op/path (or redactvalue) to avoid leaking secrets when debug logging is enabled.
let url = format!("https://{}:{}/configure", self.config.host, self.config.api_port);
debug!("VyOS configure op: POST {} {:?}", url, body);
| /// Return the next available numeric tenant_id (starting at 1). | ||
| pub fn next_tenant_id(&self) -> u32 { | ||
| (1u32..) | ||
| .find(|id| !self.tenant_id_index.contains_key(id)) | ||
| .unwrap() // infinite range always has a result | ||
| } |
| source_address: Option<&str>, | ||
| vrrp: Option<VrrpConfig>, | ||
| ) -> Result<Uuid> { | ||
| let tenant_id = self.storage.next_tenant_id(); |
| self.configure_set( | ||
| &["protocols", "bgp", "address-family", "l2vpn-evpn", "advertise-all-vni"], | ||
| None, | ||
| ).await?; |
| // 3. Configure WireGuard interface for the tenant | ||
| let wg_iface = format!("wg{}", tenant.wireguard.interface_index); | ||
| self.configure_wireguard_interface( | ||
| &wg_iface, | ||
| &tenant.wireguard.address, | ||
| &tenant.wireguard.private_key, | ||
| tenant.wireguard.port, | ||
| Some(&format!("Tenant {} WireGuard", tenant.name)), | ||
| ).await?; | ||
|
|
| // 5. Commit and save | ||
| self.commit().await?; | ||
| self.save().await?; |
Resolved. I merged |
![foxguard-app[bot]](https://avatars.githubusercontent.com/in/3854821?s=60&v=4){kind=small}
| self.init_http_client()?; | ||
| let api_key = self.config.api_key.clone() | ||
| .ok_or_else(|| anyhow!("API key is required for HTTP API operations"))?; | ||
| let client = self.http_client.as_ref().unwrap(); |
There was a problem hiding this comment.
foxguard · MEDIUM · rs/no-unwrap-in-lib (CWE-248)
.unwrap() can panic at runtime — use proper error handling with ? or match
| self.init_http_client()?; | ||
| let api_key = self.config.api_key.clone() | ||
| .ok_or_else(|| anyhow!("API key is required for HTTP API operations"))?; | ||
| let client = self.http_client.as_ref().unwrap(); |
There was a problem hiding this comment.
foxguard · MEDIUM · rs/no-unwrap-in-lib (CWE-248)
.unwrap() can panic at runtime — use proper error handling with ? or match
|
|
||
| /// Return the next available numeric tenant_id (starting at 1). | ||
| pub fn next_tenant_id(&self) -> u32 { | ||
| (1u32..) |
There was a problem hiding this comment.
foxguard · MEDIUM · rs/no-unwrap-in-lib (CWE-248)
.unwrap() can panic at runtime — use proper error handling with ? or match
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (10)
src/api/vyos.rs:313
- This passes a reference to a temporary
String(&port.to_string()) into anasync fn. That temporary is dropped immediately, which will not compile (borrowed value does not live long enough). Bind the string to a local first.
self.configure_set(
&["interfaces", "wireguard", interface, "port"],
Some(&port.to_string()),
).await?;
src/api/vyos.rs:361
- This passes a reference to a temporary
String(&ka.to_string()) into anasync fn, which will not compile. Bind the string to a local first.
self.configure_set(
&["interfaces", "wireguard", interface, "peer", peer_name, "persistent-keepalive"],
Some(&ka.to_string()),
).await?;
src/api/vyos.rs:390
configure_vxlanpasses references to temporaryStrings (&vni.to_string(),&mtu.to_string()) intoasync fncalls, which will not compile. Bind these to locals before awaiting.
let iface = format!("vxlan{}", vni);
self.configure_set(
&["interfaces", "vxlan", &iface, "vni"],
Some(&vni.to_string()),
).await?;
src/api/vyos.rs:428
- This passes a reference to a temporary
String(&system_as.to_string()) into anasync fn, which will not compile. Bind the string to a local first.
self.configure_set(
&["protocols", "bgp", "system-as"],
Some(&system_as.to_string()),
).await?;
src/api/vyos.rs:447
- This passes a reference to a temporary
String(&remote_as.to_string()) into anasync fn, which will not compile. Bind the string to a local first.
self.configure_set(
&["protocols", "bgp", "neighbor", peer_ip, "remote-as"],
Some(&remote_as.to_string()),
).await?;
src/api/vyos.rs:490
- This passes a reference to a temporary
String(&table_id.to_string()) into anasync fn, which will not compile. Bind the string to a local first.
self.configure_set(
&["vrf", "name", vrf_name, "table"],
Some(&table_id.to_string()),
).await?;
src/api/vyos.rs:539
- These pass references to temporary
Strings (&vrid.to_string(),&priority.to_string()) intoasync fncalls, which will not compile. Bind them to locals before awaiting.
self.configure_set(
&["high-availability", "vrrp", "group", group_id, "vrid"],
Some(&vrid.to_string()),
).await?;
self.configure_set(
&["high-availability", "vrrp", "group", group_id, "priority"],
Some(&priority.to_string()),
).await?;
src/api/vyos.rs:465
enable_bgp_evpnsetsprotocols bgp address-family l2vpn-evpn advertise-all-vni, but the repo's VyOS config template usesset protocols bgp l2vpn-evpn advertise-all-vni(noaddress-familysegment). The current path likely won't map to the intended config node.
self.configure_set(
&["protocols", "bgp", "address-family", "l2vpn-evpn", "advertise-all-vni"],
None,
).await?;
src/api/vyos.rs:603
provision_tenantusesself.commit()/self.save(), but those methods currently call the legacy/api/commitand/api/saveendpoints. The docs in this repo show commit/save done viaPOST /configurewith{ "op": "commit" }/{ "op": "save" }. As-is, provisioning is likely to fail at commit/save even if the set operations succeed.
// 5. Commit and save
self.commit().await?;
self.save().await?;
src/services/tenant.rs:118
- Tenant IDs are used directly as an IPv4 octet in
Tenant::new(e.g.100.64.<tenant_id>.1/24). Oncetenant_id > 255, generated addressing becomes invalid. Add an explicit bound check socreate_tenantfails with a clear error instead of creating an unusable tenant record.
let tenant_id = self.storage.next_tenant_id();
| if line.is_empty() { | ||
| if current_section.as_deref() == Some("Peer") { | ||
| if let Some(peer) = current_peer.take() { | ||
| if !peer.public_key.is_empty() && !peer.endpoint.is_empty() { |
| current_section = Some("Peer".to_string()); | ||
| // Flush peer on new section header | ||
| if let Some(peer) = current_peer.take() { | ||
| if !peer.public_key.is_empty() && !peer.endpoint.is_empty() { |
|
|
||
| // Flush the last peer | ||
| if let Some(peer) = current_peer { | ||
| if !peer.public_key.is_empty() && !peer.endpoint.is_empty() { |
| let mut body = serde_json::json!({ | ||
| "op": "set", | ||
| "path": path | ||
| }); | ||
| if let Some(v) = value { | ||
| body["value"] = serde_json::Value::String(v.to_string()); | ||
| } | ||
| self.configure_op(body).await?; | ||
| Ok(()) |
| println!("Creating tenant '{}'", name); | ||
| println!(" BGP AS : {}", bgp_as); | ||
| println!(" Source addr : {}", source_address.as_deref().unwrap_or("(use provider default)")); | ||
| println!(" WG public key: {}", kp.public_key); | ||
| println!(" WG private key: {} (store securely!)", kp.private_key); |
src/network.rsreferencedboringtun,getrandom, andx25519_dalek— none declared inCargo.toml— making the module uncompilable and excluded from the build. This PR fixes that, integrates the module, and implements the full multi-tenant networking stack against the VyOS HTTP API.Dependencies
Added
base64 = "0.21",rand = "0.8",x25519-dalek = { version = "2", features = ["static_secrets"] }.src/network.rs— rewrittenDropped the native
boringtunWireGuard tunnel (dataplane belongs to VyOS). Replaced with:WireguardKeypair+generate_wireguard_keypair()via X25519/Curve25519derive_public_key()— public from base64 privategenerate_client_config()— emits a valid.conffileparse_wireguard_config()parser (blank-line peer flushing was broken)Module is now declared in
lib.rsandmain.rs.src/api/vyos.rs— VyOS HTTP API extensionsAll config ops go through
POST /configure; monitoring throughGET /show/…:Monitoring:
get_bgp_summary,get_vxlan_status,get_vrf_routes,get_wireguard_status.src/models/tenant.rs— newTenantwith embeddedVrfConfig,VxlanConfig,TenantWireguardConfig,VrrpConfig. Addressing auto-derived from numerictenant_id:100.64.<id>.1/24, port51820 + (id % 1000)10000 + id1000 + id, BGP RT:<as>:<table>src/services/tenant.rs— newTenantServicewraps storage + provider access:create_tenant(auto-generates WireGuard keys),provision_tenant_on_router(pushes config to named VyOS provider, marks tenant active),delete_tenant.CLI
Two new top-level subcommands:
Original prompt
This section details on the original issue you should resolve
<issue_title>Comprehensive E2E Encrypted Multi-Tenant Network Architecture</issue_title>
<issue_description># Comprehensive E2E Encrypted Multi-Tenant Network Architecture
This document synthesizes our complete plan for building a secure, end-to-end encrypted, multi-tenant overlay network using VyOS, WireGuard, VXLAN, OSPF, L3VPN, and other technologies. The architecture implements a Unix philosophy-aligned approach with modular components that can be composed together while maintaining separation of concerns.
Architecture Overview
graph TB subgraph Physical["Physical Infrastructure"] direction TB DC1["Datacenter 1<br>5.254.54.0/26"] DC2["Datacenter 2<br>5.254.43.160/27"] CloudExt["Cloud Extensions<br>Dynamic"] end subgraph Hypervisor["Hypervisor Layer"] direction TB ArchLinux["Arch Linux OS"] OVS["Open vSwitch<br>Hardware Offload"] SRIOV["SR-IOV<br>Virtual Functions"] SystemdVMSpawn["systemd-vmspawn"] end subgraph Router["Virtual Router Layer"] direction TB VyOSVMs["VyOS VMs"] WireGuard["WireGuard Mesh<br>172.27.0.0/20"] VXLAN["VXLAN Tunnels"] OSPF["OSPF Areas"] BGP["BGP EVPN"] L3VPN["L3VPN (VRF)"] end subgraph Tenant["Tenant Layer"] direction TB TenantVMs["Tenant VMs"] ManagedServices["Managed Services"] K8S["Kubernetes Clusters"] Backups["Backup Systems"] end Physical --> Hypervisor Hypervisor --> Router Router --> TenantNetwork Addressing Schema
graph LR subgraph PublicSpace["Public Address Space"] DC1Public["DC1: 5.254.54.0/26"] DC2Public["DC2: 5.254.43.160/27"] DC2Additional["DC2 Additional: 5.254.43.208/29"] end subgraph ManagementSpace["Management Networks"] ControlPlane["Control Plane: 172.27.0.0/20"] BackboneNetwork["Backbone: 172.16.0.0/20"] end subgraph TenantSpace["Tenant Address Space"] CGNATBase["Base: 100.64.0.0/10"] WireGuardOverlay["WireGuard: 100.64.0.0/16"] TenantNetworks["Tenant Networks: 100.65.0.0/16"] TenantServices["Services: 100.80.0.0/16"] MigrationSpace["Migration: 100.96.0.0/16"] endImplementation Plan
1. Physical Infrastructure Setup
The physical infrastructure consists of:
Datacenter 1:
Datacenter 2:
2. Hypervisor Layer Configuration
Each bare metal server runs:
NIC Configuration:
3. VyOS VM Deployment Using mkosi and systemd-vmspawn
Create a base VyOS image using mkosi:
4. WireGuard Control Plane Configuration
The secure manageme...
💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.