Architecture Decision Records (ADR): The Team's Long-Term Technical Memory

The Archaeology Code Problem:

Git blame line 847 of gateway/MessageBroker.java:
// Process messages in batches of exactly 47
private static final int BATCH_SIZE = 47;

Questions:
- Why 47? Not 50, not 100, not a power of 2?
- Was this tuned from performance testing?
- Is this a constraint from a third-party system?
- Is this a coincidence from a developer's keyboard?
- What breaks if we change it to 100?

The original developer left 3 years ago.
The code that made 47 significant was deleted 2 years ago.
Nobody knows. Nobody changes it. The magic number persists forever.

# ADR 023: Use PostgreSQL as the Primary Datastore

**Date:** 2026-04-17
**Status:** Accepted
**Architect:** Sarah Chen (Lead Architect)
**Deciders:** Engineering leadership meeting 2026-04-15

## Context

We are building the new Order Management System (OMS) that will replace the 
legacy Oracle Forms application. We need to select a primary relational database.

**Constraints:**
- Must support ACID transactions (order state changes must be atomic)
- Team has no DBA staff — operations must be manageable by engineers
- Budget: $0/month for database licensing (startup phase)
- Scale requirement: ~50K orders/day, 500 concurrent users initially
- Must run on AWS (company standard cloud provider)

**Alternatives evaluated:**
- MySQL 8.0 (AWS RDS)
- PostgreSQL 16 (AWS RDS)
- Amazon Aurora PostgreSQL
- MongoDB (considered briefly, rejected — document model unsuitable for relational order data)

## Decision

We will use **PostgreSQL 16 on AWS RDS** as the primary datastore for the OMS.

Aurora PostgreSQL was considered but its minimum cost (~$180/month) is not 
justified at our current scale. We will re-evaluate Aurora when monthly active 
orders exceed 500K.

## Consequences

**Positive:**
- Zero licensing cost (open source)
- Strong ACID guarantees for order state machine
- Rich ecosystem: pgvector, PostGIS, full-text search available without extra services
- Team has existing PostgreSQL expertise (3/5 engineers have production experience)
- AWS RDS provides automated backups, failover, and patching

**Negative:**
- No native horizontal write scaling (sharding must be added manually when needed)
- Read scaling requires manual read-replica configuration
- RDS managed service costs ~$50/month (t3.medium, Multi-AZ)

**Neutral:**
- Limits to features available in PostgreSQL; MySQL-specific features unavailable
- Team must learn PostgreSQL-specific tooling (pgAdmin, pg_dump vs mysqldump)

**Migration path:**
- If we need horizontal write scaling: evaluate Citus or add sharding layer
- If we need Aurora features: migration path exists (Aurora is PostgreSQL-compatible)

## Context
The authentication service is hitting 10K token validations/second during peak 
traffic. At this rate, the HMAC-SHA256 verification adds 8ms average latency 
to every API call, contributing 25% of our p99 latency budget.

Constraints:
- GDPR requires tokens expire within 24 hours
- Must be compatible with our existing JWT RS256 tokens (100M issued)
- Cannot require user re-authentication (UX constraint from Product)

Alternatives evaluated:
1. Redis token cache (rejected: adds Redis as critical dependency, ops overhead)
2. Paseto tokens (rejected: requires migrating all existing JWT clients)
3. Ed25519 signature algorithm (selected: 3× faster ECDSA verification vs RSA-2048)

# Use OpenTelemetry Collector for Observability Pipeline

## Status
Accepted

## Context and Problem Statement
How do we collect telemetry (traces, metrics, logs) from 40+ microservices 
without locking ourselves into a single observability vendor?

## Decision Drivers
* Ability to switch observability backends without code changes
* Support for all three signal types (traces/metrics/logs) 
* Production-grade reliability (not experimental)

## Considered Options
* Option A: Vendor-specific agents (Datadog Agent, New Relic Agent)
* Option B: OpenTelemetry Collector with OTLP exporters  
* Option C: Direct SDK integration to each backend

## Decision Outcome
Chosen option: **Option B: OpenTelemetry Collector**

Because:
- Changes backend (Grafana → Datadog) requires only Collector config change
- CNCF project — vendor-neutral, broad community support
- Supports all three signal types in one pipeline

### Consequences
* Good: Can switch observability vendors in hours, not weeks
* Good: Centralized sampling and filtering in one place
* Bad: Adds an OTel Collector fleet to operate (Kubernetes DaemonSet)
* Neutral: Team must learn OTel Collector configuration and pipeline design

Repository structure:
├── docs/
│   └── architecture/
│       └── decisions/
│           ├── README.md           (Index of all ADRs)
│           ├── ADR-001-postgresql.md        (Accepted)
│           ├── ADR-002-kafka-messaging.md   (Accepted)
│           ├── ADR-015-graphql-rejected.md  (Rejected — keep rejections too!)
│           ├── ADR-022-redis-cache.md       (Accepted)
│           └── ADR-023-otel-collector.md    (Proposed — under review)

1. Engineer drafts ADR-024-use-typescript.md with Status: Proposed
2. Opens Pull Request with description: "ADR: Adopt TypeScript for all new services"
3. PR comments become the official decision discussion record
4. Deciders approve → Status changed to Accepted → PR merged
5. Future engineers git blame the ADR → see who approved it and when
6. If decision reversed: new ADR-031-revert-to-javascript.md with Status: Supersedes ADR-024
   ADR-024 updated: Status: Superseded by ADR-031 (history preserved — never deleted)