Research Proposal · 2026-10-03 · Enterprise / Sovereign AI

Private AI Improvement Without Raw Data Export

Publication type

Research Proposal

Evidence status

Proposal / Hypothesis: A proposed direction, system, or hypothesis that remains untested.research survey and proposal; no Ethen implementation or measurement; requires privacy, security and legal review

Research program

Enterprise / Sovereign AI

Published

October 3, 2026

Authors

Ethen Research Lab

Reading time

13 min read

Enterprises want their AI systems to get better from the work they do. They also cannot let that work leave. The techniques that seem to resolve the tension each come with a threat model, and most fail quietly outside it.

Abstract

An organization deploying agents generates exactly the evidence that would improve them: verified outcomes, failures, corrections and recoveries on its own work. Contracts, regulation and common sense usually forbid exporting that evidence to a vendor. Private AI improvement asks how a tenant can benefit from learning, its own and possibly other tenants', without raw content leaving its boundary. This proposal surveys the available mechanisms and states the threat model each one assumes: tenant-local learning and evaluation, bounded aggregate statistics, federated evaluation and learning, secure aggregation, differential privacy, contribution limits, and rights-based withdrawal. For each we describe what it protects against and what it does not, with attention to membership inference, attribute and property inference, reconstruction, malicious contributors, small cohorts and revocation. Three cautions run through the paper: federated is not automatically private, aggregate is not automatically anonymous, and an embedding is not automatically safe. We propose an ordering in which tenant-local improvement comes first, cross-tenant statistics come only under explicit grants and accounting, and cross-tenant model updates remain a research question. The protocol built on these primitives is Toward a Sovereign Improvement Protocol for Enterprise AI. No Ethen system implementing these mechanisms exists; this is a research survey and proposal.

The tension in one paragraph

A vendor that serves many organizations could, in principle, learn from all of them: which configurations work for which kinds of task, which recoveries succeed, which skills transfer. Each organization would benefit from the others' experience. But the experience is made of customer records, internal documents, code and process details. Contracts often prohibit vendors from training on customer data, and regulators treat pseudonymized records as personal data where re-identification is possible. Detailed text records can now be re-identified at scale using language models (Lermen et al.), and agent trajectories carry identifying traces in error strings, resource names, timing and tool sequences. The question is not whether to protect the data but which forms of learning, if any, can be made safe enough to permit.

What exactly would be shared?

"Improvement" covers several different artifacts, and their risks differ by orders of magnitude.

• Evaluation results. A few numbers per tenant per candidate change, such as the change in verified success for a task family. Lowest risk, because the output is small and its structure is known in advance.
• Configuration priors. Statistics about which models, tools or recovery families tend to work for which task classes. Moderate risk; the categories themselves can reveal what a tenant does.
• Skills and procedures. Distilled instructions or code derived from a tenant's work. High risk, because a skill can embed names, rules and process details verbatim. Skills derived from private work should be treated as private unless reviewed and rewritten.
• Model updates. Gradients or weight deltas from training on tenant data. Highest risk, for the reasons given above, and the hardest to withdraw.

A design that begins with evaluation results and configuration priors can deliver much of the practical value of shared learning, such as knowing whether an upgrade tends to help, at a fraction of the exposure of sharing skills or updates. Private-derived synthetic data deserves the same caution as the private data it came from: generating synthetic examples from tenant records does not by itself remove what those records reveal.

Three cautions

Federated is not automatically private. Federated learning keeps raw data on clients and shares model updates (McMahan et al.). Updates leak. Shared gradients can be inverted to reconstruct training inputs (Zhu et al., 2019), and an adversarial participant can infer both membership and properties of subsets of other participants' data from the updates exchanged during collaborative training (Melis et al.). Federated architecture moves the data problem; it does not remove it.

Aggregate is not automatically anonymous. Statistics computed over records can reveal facts about individual records, and models trained on data can reveal whether a given record was in the training set (Shokri et al.) or reproduce it verbatim (Carlini et al.). An aggregate over a small cohort, or a sequence of aggregates released over time, can single out an organization or a person.

An embedding is not automatically safe. Dense text embeddings are sometimes treated as an anonymized representation. Embedding inversion has recovered most short inputs exactly and recovered full names from clinical notes (Morris et al.). Exporting embeddings of customer content should be treated as exporting the content unless shown otherwise.

A threat model before a mechanism

Figure 1 summarizes the adversaries considered.

• Curious vendor. Operates the aggregation service and would like to learn tenant content.
• Curious or malicious peer tenant. Participates in shared learning and tries to infer other tenants' data, or to poison the shared result.
• External analyst. Sees released aggregates or models over time and combines them with outside information.
• Compromised component. A replay runner, aggregator or model endpoint under attacker control.
• Insider at the tenant. Has legitimate access within the boundary; largely outside what these mechanisms can address.

Every mechanism below is described in terms of which of these adversaries it addresses. A mechanism presented without a threat model should not be relied on.

The mechanisms

Figure 2 compares the mechanisms against the adversaries and attacks they address.

Tenant-local learning and evaluation

The simplest private improvement is improvement that never leaves. A tenant's own verified experience can improve its own retrieval, skills, routing and context policies inside its boundary, and its own historical tasks can evaluate changes before they ship, as proposed in Tenant Replay. This addresses the curious vendor and peer tenants entirely, because nothing crosses the boundary. Its limitation is scale: small tenants have little data, and nobody benefits from anyone else's experience.

Bounded aggregate statistics

The next step is to export only aggregates: success rates per task family and configuration, recovery-family outcomes per failure class, counts. Bounding means minimum cohort sizes, coarse categories, limits on how many queries or releases are made, and review before release. These controls reduce risk from external analysts but provide no formal protection, and they are vulnerable to differencing across releases.

Differential privacy

Differential privacy gives a formal bound on how much any single record, or any single tenant, can change a released result (Dwork & Roth). It can be applied to aggregate statistics or to model training, as in differentially private stochastic gradient descent with privacy accounting across steps (Abadi et al.). Where the noise is added matters: if the aggregator adds it centrally, the aggregator still sees exact values, so protection against the operator requires noise or secure aggregation applied before data leaves each tenant. NIST has published guidance on evaluating differential-privacy claims, including the importance of stating the unit of privacy and the accounting over repeated releases (NIST SP 800-226). Three points matter for enterprise use. The unit must match the threat: record-level privacy does not protect a tenant whose many records share a pattern; tenant-level privacy does, at much higher utility cost. The budget is consumed by every release, so a dashboard refreshed daily needs accounting across days. And the utility cost for small cohorts can make results useless. Differential privacy is appropriate only when the unit, budget and accounting are explicit; it is not a label to attach to noisy statistics.

Secure aggregation

Secure aggregation lets a server compute the sum of participants' vectors without learning any individual contribution, and tolerates participants dropping out (Bonawitz et al.). It addresses the curious vendor for the individual updates, but not what the sum itself reveals, and it makes poisoning harder to detect, because the server cannot inspect individual contributions. It is usually combined with differential privacy on the sum.

Federated evaluation

Before any shared learning, a weaker and more useful step is shared evaluation: each tenant runs the same candidate change on its own tasks and reports a bounded, privatized result. A vendor learns whether a new model or skill tends to help across tenants, without seeing any tenant's tasks. The federated-learning literature identifies evaluation on decentralized data as an open problem with its own heterogeneity and privacy issues (Kairouz et al.). Because its output is a few numbers per tenant, federated evaluation is far easier to protect than federated training.

Contribution limits and robustness

Any shared computation is exposed to participants who lie. In federated learning, a single malicious participant can implant a backdoor in the shared model through model replacement, and can evade anomaly detection designed to catch it (Bagdasaryan et al.). Contribution limits, such as clipping each tenant's influence, a cap on how much any tenant can move a shared statistic, and admission rules for participants, reduce but do not eliminate this risk. For shared evaluation, a malicious tenant can misreport its results; robust aggregation and outlier review are the defenses, and they work less well with few tenants.

Small cohorts

Many of the protections above weaken or fail when there are few participants. Minimum cohort sizes become hard to meet within a single industry. Differential privacy noise swamps the signal. A malicious participant is a larger fraction of the cohort. Secure aggregation reveals more when the sum is over three tenants than over three thousand. Enterprise AI will often be in the small-cohort regime, especially in regulated sectors, and the honest response is to restrict cross-tenant learning there to the most coarse and heavily protected statistics, or to forgo it.

Withdrawal, revocation and rights

Consent and rights change. A tenant may withdraw from shared learning; a data subject may request deletion. The mechanism must be able to honor that, and the record of who contributed what must exist to make it possible. The rights model behind this is developed in Rights as Infrastructure, and the procedure for building datasets that respect it in A Rights-Aware Dataset Compiler.

Withdrawal is easy for statistics not yet released and hard for anything already released or trained. Released aggregates cannot be recalled. Removing a contributor's influence from a trained model is the subject of machine unlearning, which can be exact for models designed for it (Bourtoule et al.) but is approximate in general, and approximately unlearned language models can sometimes be made to recall the removed information through benign relearning (Hu et al.). Unlearning mechanisms can also create new risks for the data that remains (Cohen et al.). The practical rule that follows is conservative: do not put data whose rights may be withdrawn into shared model weights at all, and prefer retraining from lineage or retiring a model over promising deletion from it.

A proposed ordering

Figure 3 shows the ordering we propose, from least to most risky.

1. Tenant-local improvement and evaluation. No cross-boundary flow. The default.
2. Federated evaluation with bounded, privatized results. Few numbers per tenant, under explicit grant.
3. Cross-tenant aggregate statistics. Under grant, minimum cohort, tenant-level privacy accounting and contribution limits.
4. Cross-tenant model updates. Research only, in sealed settings, with secure aggregation, differential privacy at the tenant level, robust aggregation, and no data whose rights may be withdrawn.

Moving down the ordering requires a stated threat model, a privacy and security review, and a measured utility gain at the previous level that justifies the added risk. Whether shared learning adds enough value to justify its risk is itself an empirical question, related to the training-utility measurement in What Makes AI Data Defensible?.

Open research questions

1. How much of the value of cross-tenant learning for agents can be obtained from federated evaluation and configuration priors alone, without sharing skills or model updates?
2. At what cohort sizes do tenant-level privacy accounting and robust aggregation yield statistics that are both protected and useful for enterprise task families?
3. Can agent trajectories or their derived records be transformed so that measured re-identification risk is acceptable, or is structure itself too identifying?
4. How should privacy budgets be allocated across the many small releases an operational system makes, such as dashboards, assurance reports and research statistics?

Limitations

This paper surveys mechanisms and proposes an ordering; it reports no Ethen implementation or measurement. The privacy literature evolves quickly, and attacks published after this survey may weaken mechanisms described here. Differential-privacy parameters, cohort minimums and contribution limits are deployment decisions that need privacy expertise and legal review. Regulatory treatment of aggregates, embeddings and model updates differs across jurisdictions and is not settled. Federated evaluation and learning for agents are research-grade, not production-proven.

Conclusion

Private AI improvement is possible in a limited, layered sense: most of the value can come from learning that never leaves the tenant, some from carefully protected shared evaluation, and a little, perhaps, from shared statistics or updates under explicit accounting. Each step outward needs a threat model, a mechanism matched to it and a measured reason to take it. Federated, aggregate and embedded data can all leak, and the design should assume they will unless shown otherwise.

FAQ

Is federated learning private? Not by itself. Updates can leak training data and properties, and malicious participants can poison the shared model. Privacy requires additional mechanisms with stated privacy bounds and threat models.

Are embeddings of customer data safe to export? Not by default. Embedding inversion can recover much of the original text, including names.

What is the safest form of improvement? Tenant-local learning and evaluation, where nothing leaves the boundary.

Related research

• Toward a Sovereign Improvement Protocol for Enterprise AI — the protocol built on these primitives.
• Tenant Replay: Private Evaluation Inside Enterprise Boundaries — tenant replay produces local metrics.
• Rights as Infrastructure: Building AI Datasets That Know How They May Be Used — consent and purpose.
• A Rights-Aware Dataset Compiler for AI Training and Evaluation — rights-aware builds.
• What Makes AI Data Defensible? — data defensibility under privacy.

References

1. Lermen, S. et al. (2026). Large-scale online deanonymization with LLMs. arXiv:2602.16800. https://arxiv.org/abs/2602.16800
2. McMahan, H. B. et al. (2016). Communication-Efficient Learning of Deep Networks from Decentralized Data. arXiv:1602.05629. https://arxiv.org/abs/1602.05629
3. Zhu, L. et al. (2019). Deep Leakage from Gradients. arXiv:1906.08935. https://arxiv.org/abs/1906.08935
4. Melis, L. et al. (2018). Exploiting Unintended Feature Leakage in Collaborative Learning. arXiv:1805.04049. https://arxiv.org/abs/1805.04049
5. Shokri, R. et al. (2016). Membership Inference Attacks against Machine Learning Models. arXiv:1610.05820. https://arxiv.org/abs/1610.05820
6. Carlini, N. et al. (2020). Extracting Training Data from Large Language Models. arXiv:2012.07805. https://arxiv.org/abs/2012.07805
7. Morris, J. X. et al. (2023). Text Embeddings Reveal (Almost) As Much As Text. arXiv:2310.06816. https://arxiv.org/abs/2310.06816
8. Dwork, C., Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science 9(3–4). https://doi.org/10.1561/0400000042
9. Abadi, M. et al. (2016). Deep Learning with Differential Privacy. arXiv:1607.00133. https://arxiv.org/abs/1607.00133
10. NIST (2025). SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees. https://doi.org/10.6028/NIST.SP.800-226
11. Bonawitz, K. et al. (2016). Practical Secure Aggregation for Federated Learning on User-Held Data. arXiv:1611.04482. https://arxiv.org/abs/1611.04482
12. Kairouz, P. et al. (2019). Advances and Open Problems in Federated Learning. arXiv:1912.04977. https://arxiv.org/abs/1912.04977
13. Bagdasaryan, E. et al. (2018). How To Backdoor Federated Learning. arXiv:1807.00459. https://arxiv.org/abs/1807.00459
14. Bourtoule, L. et al. (2019). Machine Unlearning. arXiv:1912.03817. https://arxiv.org/abs/1912.03817
15. Hu, S. et al. (2024). Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. arXiv:2406.13356. https://arxiv.org/abs/2406.13356
16. Cohen, A. et al. (2026). Protecting the Undeleted in Machine Unlearning. arXiv:2602.16697. https://arxiv.org/abs/2602.16697

Ethen Research Lab is Upcube's public research publication program. It publishes papers, protocols, benchmark designs, and system cards, each labelled with its evidence status. It is separate from Ethen Research, the AI research workspace product.