GlitchMesh 🧪

GlitchMesh is a lightweight, developer-focused chaos engineering proxy designed for testing microservice resilience. It helps you simulate everything from simple application-level hiccups (like slow responses and random HTTP 500s) to brutal network-level disasters (like packet loss, connection drops, and complete host blackouts).

Build robust, fault-tolerant distributed systems by breaking things on purpose—before production does it for you.

---

⚠️ Note: This project is under active development. Many features are still being implemented, and things may change frequently. Feedback and contributions are highly appreciated!

---

🚀 Why GlitchMesh?

Testing the "happy path" is easy. But what happens when your database suddenly has 100ms of network jitter? What if a downstream service accepts TCP connections but drops half the packets? GlitchMesh lets you test these exact scenarios locally or in a lab environment.

✨ Core Features

1. Application-Layer Faults (L7 Proxy)

Route traffic through GlitchMesh's HTTP proxy (/redirect/{service-name}/{endpoint}) to inject application-aware faults:

• ⏳ Latency & Jitter - Add configurable delays to simulate slow services.
• 💥 HTTP Errors - Return custom HTTP status codes (e.g., 503 Service Unavailable) and payloads.
• 🔌 Connection Timeouts & Drops - Simulate services that take forever to respond or drop connections mid-flight.
• 🎯 Probability & Priority - Control exactly how often faults happen (e.g., "fail 10% of requests") and in what order they are applied.

2. Network-Layer Faults (L3/L4 Sidecar Mode) 🔥 NEW

GlitchMesh isn't just an HTTP proxy anymore. It can now act as a privileged sidecar container to manipulate the actual OS network stack using native Linux tools (tc/netem and iptables), affecting all traffic to and from a container.

• 🌪 Traffic Shaping (tc/netem) - Inject true network-level latency, packet loss, duplication, and reordering on a specific network interface.
• 🧱 Firewall Rules (iptables) - Simulate complete or partial infrastructure outages by dropping packets destined for specific IP addresses or ports.
• 🛡 Crash-Safe - Network rules are journaled to disk. If GlitchMesh crashes mid-experiment, it cleans up orphaned rules upon restart so your lab doesn't stay permanently broken.

3. Dynamic Control Plane

• Live Configuration - Hot-reload your YAML configuration without restarting the proxy.
• REST Admin API - Arm and disarm faults on the fly using the /admin/services and /admin/netchaos endpoints. No YAML editing required to trigger a quick experiment!

---

🛠 Usage & Configuration

Application-Layer Configuration (YAML)

Define your services and fault profiles in a simple YAML file:

service:
  - name: "service-one"
    url: "http://localhost:8080/"
    fault:
      enabled: true
      probability: 0.25 # Apply faults to 25% of requests
      priority: ["error", "latency", "connection_drop"]
      types:
        latency:
          delay: 5000  # 5 second delay
        error:
          statuscode: 500
          message: "something went really wrong!!"
        connection_drop:
          droprate: 0.4

Running the Proxy

# Start the proxy server on port 9000
go run ./cmd/glitchmesh start server

# Make requests through the proxy
curl http://localhost:9000/redirect/service-one/api/users

Running as a Network Sidecar (Docker)

To use kernel-level network faults, GlitchMesh needs to run as a privileged sidecar sharing the network namespace of your target application.

# Example docker-compose snippet
services:
  glitchmesh-sidecar:
    build: 
      context: .
      dockerfile: lab/Dockerfile.sidecar
    cap_add:
      - NET_ADMIN  # Required for tc/netem
      - SYS_ADMIN  # Required for iptables
    network_mode: "service:my-target-app" # Share network namespace
    volumes:
      - glitchmesh_run:/var/run # For crash-safe rule journaling

You can then use the Admin API to inject network faults dynamically:

# Inject 100ms delay and 5% packet loss for 60 seconds on interface eth0
curl -X POST http://localhost:9001/admin/netchaos \
  -H "Content-Type: application/json" \
  -d '{
    "type": "netem",
    "netem": {
      "interface": "eth0",
      "delay_ms": 100,
      "loss_pct": 5,
      "duration_s": 60
    }
  }'

---

🗺 Upcoming Features

We're constantly expanding GlitchMesh's capabilities to make it a more comprehensive chaos engineering tool:

Fault Types & Simulation

• Bandwidth Throttling - Limit throughput to simulate network congestion.
• Response Corruption - Corrupt random bytes or truncate response bodies.
• Circuit Breaker Simulation - Stateful simulation that fails fast after consecutive failures.
• eBPF Fault Injection - Extremely low-overhead kernel-level packet manipulation.

Observability & Architecture

• Prometheus metrics endpoint & structured logging.
• Grafana dashboard out-of-the-box.
• Chaos Scenario Engine - Programmatic, multi-step chaos experiments with steady-state validation.
• Multi-instance support backed by Redis for true horizontal scalability.