.avif)
.svg.avif)
In late February, it was business as normal for practices using NexHealth. Patients were booking appointments online, filling out intake forms, and receiving automated reminders, just like usual. Staff were checking in patients, and everything synced seamlessly with their EHRs. Developers who had built on top of the NexHealth API were getting the same responses they expected. Nothing felt different.
That was the goal. Behind the scenes, NexHealth had just finished moving roughly 4 TB of live production data across 19,000+ healthcare institutions to a dedicated database. The migration ran for three weeks while our systems served real traffic, and produced zero customer-facing incidents.
What we do and why the database matters
NexHealth connects patients to care. We do this in two ways: through our own products, which help practices manage appointments, records, and patient communication; and as a platform, powering the health tech companies building on our APIs. Healthcare has historically lagged on technology adoption, largely because the EHR (Electronic Health Record) landscape is so fragmented. There are hundreds of EHR vendors, each with their own data formats and integration models. NexHealth unlocks innovation in healthcare by removing that complexity for developers building on this legacy infrastructure, enabling modern products to be built for practices and patients.
Sitting in the critical path for practices, patients, and the developers building on our APIs makes database reliability a first-order concern. Every query that slows down is a patient waiting longer to book an appointment. Every query that fails is a front desk team staring at an error when they should be checking patients in. And every write that doesn't land is a practice acting on a schedule that no longer matches reality.
The pressure was building
For most of NexHealth's history, our backend had run on a single primary PostgreSQL instance on Amazon RDS with a separate read replica for read-only queries. Storing healthcare data across separate databases is difficult due to its inherent graph-like, highly interconnected structure. Our one database approach worked well until it couldn't.
IOPS (input/output operations per second) is the rate at which a database reads and writes to disk. RDS backs its storage with EBS (Elastic Block Store), a single provisioned volume where IOPS is a fixed resource you pay for upfront. In other words, you commit to a ceiling, and when your workload approaches it, transaction throughput drops and query latency climbs.
The natural response is to scale up to a larger instance type. But vertical scaling has economic and physical limits. You can only go so big. In early 2025, we ran into this same issue when rolling out our payments product. Our solution was to extract the data related to payments into its own database. Heading into 2026 with aggressive growth targets, it was time to do it again. The domain we targeted carries all clinical care data which we call the “treatment domain”.
The new treatment database runs on Aurora PostgreSQL rather than standard RDS PostgreSQL. Aurora uses a distributed storage architecture spread across multiple nodes and availability zones, so I/O is spread across many nodes rather than hitting one ceiling. We've designed it to be shardable in the future, so that we can partition data by institution and utilize the primary DB to lookup which treatment DB shard an institution's data belongs to.
4 TB of patient history
Clinical care data is the record of what happens to a patient once they are in care. At NexHealth, this includes:
- Procedures and procedure cost estimates: the individual services rendered, with associated cost and insurance information
- Treatment plans and plan procedures: proposed care pathways and the procedures attached to them
- Clinical notes and clinical note procedures: provider documentation of each visit, linked to the specific procedures performed
- Fee schedules and fee schedule procedure codes: pricing structures and the industry-standard codes that normalize procedure descriptions across different EHR systems
The procedures table alone was 3.3 TB of the total 4 TB of data in the domain. That's over 3 billion records for services rendered to over 94 million practice patients (cumulative distinct patients per institution).
Size alone is not what makes a domain a good extraction candidate. What mattered equally was the ratio of storage to computation. What made the treatment domain a good extraction candidate was not just its size but its shape. The domain sees a steady flow of writes as practices sync records from their EHR systems into NexHealth. But once that data is in, the computation is simple: look it up, maybe join it to something, and return it. Think of it as a giant lookup table.
The consequence of having 4 TB of lookup data in the primary DB was that reads against it regularly spiked IOPS to 20-25% of capacity on their own. When total utilization exceeded 80%, queries were being dropped and requests were retried leading to higher latencies for patients and practices. By moving this data out to a dedicated database, we could ensure the access patterns for the treatment domain wouldn't impact every other scenario in the primary DB.
Logical separation before physical separation
The decision that allowed the data migration to be safe was made months before a single row moved: we logically separated the treatment domain from the rest of the models in primary DB first.
Like most fast moving startups, our backend was built as a monolith. When all tables live in the same database, cross-domain dependencies accumulate over years of pragmatic development as we add a join here and an association there. The first phase of work was an audit. We found every place where a treatment domain model was coupled to something it should not be, and rewrote those relationships. Cross-domain joins became subqueries. Cross-domain associations became service-layer lookups.
Our monolithic codebase prevented AI from helping us; it was leading to confused output and unreliable plans. Thus this was mostly manual work to create the new domain with clear boundaries. Now, a developer working on a clinical care feature no longer needs to reason across the whole monolith and neither does the AI tool helping them. Untangling this domain was infrastructure work, but it was also an investment in development velocity going forward.
Once the entanglements were resolved, we updated the application to access the treatment domain through a separate database connection. This isolated the models from those on primary DB even though it pointed to the same physical database. Any query that tried to join across the domain boundary would fail immediately in tests and local development. This gave us confidence that no business logic would break once we started physically separating the treatment domain data.
The one model that couldn't be untangled
Not everything could be cleanly extracted. One model gave us particular trouble: procedure codes.
Procedure codes are the industry-standard CPT (Current Procedural Terminology) and CDT (Code on Dental Procedures and Nomenclature) codes used to represent medical and dental procedures respectively. They are referenced by clinical care models (to describe what was done) and by other parts of the system (for billing, analytics, and appointment management). The table is large, joins against it are pervasive on both sides of the domain boundary, and it fundamentally needs to be queryable from both databases.
We evaluated three options, each with real implications for patients and practices:
Cross-database reads at query time. When a clinical care query needs procedure code data, issue a second query to the primary DB and assemble the result in the application. This permanently couples clinical care read latency to the primary DB. Every practice pulling a patient's treatment history is a common, latency-sensitive operation, and each one would require an unavoidable network round-trip to a second database. We would be trading one scalability problem for a different performance one.
Application-layer double writes. Write to both databases whenever a procedure code is created or updated. Simpler to understand, but fragile. Any developer who modifies a procedure code without knowing about the dual-write requirement could silently leave the clinical care copy out of date and a provider could see wrong or missing information on a patient's treatment plan.
PostgreSQL Foreign Data Wrapper (FDW). Postgres allows databases to expose a “foreign table.” This is a proxy backed by a table in a different database. We keep procedure codes as the source of truth in the primary DB and use database-level triggers to propagate every insert, update, and delete to the treatment database automatically through the foreign table, providing a “reference model” in the treatment database.
We chose the FDW approach. Propagation happens at the database layer, independent of application code, and no developer can accidentally bypass it.
The tradeoff is that the FDW requires the two databases to be able to talk directly to each other, rather than having the application mediate all communication. That means managing network routing between them, storing and rotating credentials in both environments, and writing a bootstrapping task that sets up the FDW on every fresh database initialization since the configuration contains secrets and therefore cannot be stored in the schema file. It is meaningfully more infrastructure to own. For a reference model as fundamental as procedure codes, we judged that reliability was worth the operational overhead. For a more loosely coupled model, the calculus might go the other way.
Moving the data
With the domain logically isolated and the reference model handled, the physical migration could begin. We used PostgreSQL logical replication: a pub/sub mechanism where the source database publishes a stream of changes as they happen, and the new database subscribes and applies them. This keeps both sides synchronized during the migration window and enables a clean, zero-downtime cutover once the subscriber has caught up.
We batched migrations by customer size. We moved internal test institutions first, then early access customers, then progressively larger cohorts up through our largest enterprise accounts. The logic: start where the blast radius is smallest. A bug caught on an internal test institution is a learning. The same bug surfacing at enterprise scale is an incident.
What we learned
Postgres does not propagate replica identity to range partitions
On the first day, we hit a problem: deletes were failing on one of the destination tables during replication. The root cause turned out to be a documented gap in PostgreSQL that has been open since 2019.
Some background: for Postgres to replicate an update or delete to a subscriber, it needs to know which row to target. For tables without a suitable unique key, you can set REPLICA IDENTITY FULL, which treats the entire row as the identity. We set this globally on all target tables before starting the migration.
One of our tables utilizes range partitioning, where Postgres automatically creates new child partition tables as data arrives. REPLICA IDENTITY FULL set on the parent does not propagate to range partitions, so the partitions were left without a usable row identity of their own. Core contributors have acknowledged this as a usability gap, and no clean built-in workaround exists.
We needed to make sure all new range partitions also had REPLICA IDENTITY FULL. We built the fix into the migration tooling itself: it now programmatically applies REPLICA IDENTITY FULL to all tables and range partitions that don't already have it set as part of its setup step, covering any partitions created mid-migration. This will also ensure it won't be a miss for future similar migrations.
WAL accumulation and slot invalidation
PostgreSQL logical replication keeps track of progress via a replication slot. The slot holds back the Write-Ahead Log (WAL), which is the append-only record of every database change, until the subscriber confirms it has processed those changes. If the subscriber falls behind, unprocessed WAL accumulates. When retained WAL exceeds the size limit, Postgres invalidates the slot rather than allow storage to spiral. With no slot, replication cannot continue.
We hit this issue when we tried to migrate too many institutions in a single batch. Postgres spins up temporary per-table sync slots during the initial data copy phase of logical replication. The sync slot for our largest table, procedures, fell increasingly behind as the subscriber struggled to keep up with the volume of data. The sync slot had accumulated 2.8 TB of lag. Replication slots are designed to hold WAL until the subscriber catches up, but there is a limit to how long Postgres will do that. Allowing a lagging slot to retain WAL indefinitely risks filling the disk entirely and taking down the primary DB. So when the retained WAL crosses a threshold, Postgres invalidates the slot to protect the machine and marks the slot wal_removed, breaking replication.
Recovery meant killing the replication session, deleting all partially migrated data from the destination, and re-running the migration for those institutions in smaller batches. For future migrations, we estimated the size of the data on the largest table to build appropriately sized batches.
Logical replication creates hidden I/O pressure on the primary DB
The Dedicated Log Volume (DLV), the physical disk on RDS where WAL files live, is shared between two very different workloads during a migration. Production transactions are constantly writing new WAL to it. At the same time, the WAL sender process for logical replication reads large volumes of data from it during the backfill phase to stream to the new database. That sustained read load consumed a significant portion of the disk's total I/O bandwidth, leaving less headroom for production writes. The result was elevated WalWrite wait events and higher query latency for practices using the system during those windows.
Once we identified the pattern, we adjusted our approach. We avoided running multiple backfills concurrently and were more deliberate about batch sizing to keep the backfill volume manageable relative to normal write pressure. The more complete fix for future migrations is to replicate from a read replica rather than primary writer instance. Postgres supports logical replication from a standby, which would move all WAL decode I/O off the primary DB's disk entirely and avoid competing with production traffic.
The results
The migration completed with zero customer-detected issues.
The impact on the primary DB was immediate and measurable:
- IOPS consumption decreased by 30%
- Cache miss rate went from ~4% to ~3%
While IOPS and cache utilization are influenced by many factors across the system, the timing and magnitude of the improvements strongly suggest this migration was the primary driver.
The freed IOPS translates directly into room for new customer locations. IOPS is the primary bottleneck in scaling our infrastructure, so the 30% decrease in consumption directly translates to a 30% increase in location headroom. This gives us meaningful capacity for the next wave of practices onboarding onto NexHealth.
The improvement in cache miss rate is smaller in percentage terms but meaningful at scale. Postgres caches frequently accessed data in shared buffers to avoid hitting disk. With the treatment domain's large tables no longer competing for that space, every query on the primary DB benefits from a warmer cache. Going from a 4% to a 3% miss rate across our query volume represents a real reduction in disk reads happening constantly in the background and faster experiences for our customers.
What comes next
This migration is one step in a broader transformation of NexHealth's infrastructure, and honestly, some of the most interesting work is still ahead. We are rethinking how our entire data ecosystem is organized, moving toward a set of purpose-built systems each designed for the workload it handles. There are harder problems to solve, more interesting architectural decisions to make, and more scale to build for. The healthcare market is vast and underserved. We believe it is on the verge of a technology shift that has been a long time coming, and NexHealth's ambition is to be the platform that enables it. Building infrastructure that can grow into that opportunity is work we are genuinely excited about.
If you want to work on transforming this space, take a look at our careers page.
when you demo NexHealth in October