FAFFAbout

Fine-tuned Attrition Forecasting From Archives: know where your protein is likely to die before you order the gene.

FAFFAbout is a decision-support tool for the first week of a structure project. You paste a FASTA, a UniProt accession or a PDB ID and get a breakdown of the protein plus calibrated guidance on where comparable targets historically stalled (cloning, expression, solubility, purification, crystallisation, diffraction, structure, deposition), which construct and host choices moved the needle, and how much confidence the evidence actually supports. It is grounded in the Protein Structure Initiative’s TargetTrack archive: 335,771 targets, 961,548 experimental trials and 3.8 million status events from 41 structural genomics centres, 2000 to 2017.

Why it matters: the PSI archive is the only large corpus that records where protein production attempts stopped, not just which ones succeeded, and nobody has turned it into a forecasting tool. The trap is that a target parked at “cloned” in June 2017 is one of three things (failed on scientific grounds, censored because its centre’s funding ended, or still in flight when the archive froze), and a model that cannot tell them apart learns to predict when US funding programmes ended. FAFFAbout derives censoring empirically per centre, keeps it as a first-class flag through every table, and never lets it become a negative label. It is useful for: triaging a target list before ordering genes, choosing between orthologues, deciding on a host and tag with evidence rather than habit, and setting honest expectations with collaborators.

🧭 What it is not

It is not a structure predictor, a fold recogniser or a replacement for AlphaFold. It predicts experimental attrition, which is a different and largely unmodelled quantity. The archive has a selection bias too: PSI targets already passed a soluble-prokaryotic filter, so predictions for membrane and eukaryotic proteins are extrapolation, and the interface says so.

🧱 Stack

🔧 Installation

cd FAFFAbout
uv venv --python 3.14 .venv
uv pip install -r requirements.txt
brew install mmseqs2

🚀 Usage: Phase 1 pipeline

Flags:

📊 Tables

All tables live in data/parquet/ and are exposed as DuckDB views of the same name. Text columns that hold several values are |-joined.

Canonical ladder

config/status_map.yaml maps all 30 values of the TargetTrack status vocabulary (read from the archive’s own targetTrackEnumeratedDataItems-v1.4.1.xls and cross-checked against the XSD) onto nine gates. NMR and EM milestones map onto gates 5 to 7 of their own ladders and method stays on the target, so the ladder is selectable at inference.

“Expression tested” records that a test was run, not that protein was seen, so it does not advance past cloned. Every value seen in the data that is not in the map lands in status_unmapped.csv with its count rather than becoming a silent NULL.

🧪 Phase 1 acceptance

Run scripts/build_duckdb.py. Gate from the spec: count(*) from targets in the low 300,000s, status_canon has no NULLs outside the documented unmapped set, and the cluster file resolves every protein target.

Result on 2026-09-07:

views built in data/faffabout.duckdb
targets                           335,771   (expect low 300,000s)
duplicate target_id                     0
status_history                  3,783,070
trials                            961,548
protocols                           1,501
outcomes                           11,954
target_sequences                  385,139
status_canon NULL rows                  0   over 0 distinct raw values (must all be in status_unmapped.csv)
documented unmapped values              0
protein targets with sequence     335,575
protein targets w/o cluster             0   (must be 0)
clusters (30% id)                  88,452

targets per centre (top 12):
 centre     n  mean_len  pct_deposited  pct_stopped
   MCSG 69579     260.0            2.3         32.5
   NESG 59953     347.0            1.8         57.9
   JCSG 40881     267.0            3.7          0.0
 NYSGRC 33193     404.0            0.7         43.0
   SGPP 20856     344.0            0.1          2.8
NYSGXRC 15927     432.0            5.6          0.4
  SECSG 15287     375.0            0.3          4.8
NYCOMPS 14930     378.0            0.1         76.3
 SSGCID 14517     369.0            4.9          0.0
    EFI 10567     366.0            1.9         41.9
  CSGID  9161     398.0            7.4         58.9
   CESG  8840     272.0            1.4         92.8

max_stage distribution:
max_stage      n   pct
        0 101189 30.14
        1  97013 28.89
        2  47737 14.22
        3  27163  8.09
        4  41655 12.41
        5   6067  1.81
        6   3950  1.18
        7    396  0.12
        8  10500  3.13
     none    101  0.03

PHASE 1 ACCEPTANCE: PASS

Every one of the 88,452 clusters is a unit for splitting: 50,114 are singletons and the largest holds 447 sequences. One thing the per-centre table already shows is that the work stopped marker is a centre convention rather than a scientific one (JCSG never uses it, CESG closes 93% of its targets with it), which is why Phase 2 derives censoring from activity dates and never from that flag.

Tests: .venv/bin/python -m pytest -q (status-map integrity and the parser on a synthetic target).

🪦 Censoring: the result that decides the project

The specification’s rule (per-centre 99.5th-percentile last-activity date, 180-day window) censors 5.22% of the archive, against its own stated expectation of 15 to 20%. It misses the mass closures entirely, and those are the whole phenomenon: NESG stopped 34,605 targets on 2010-06-30, 99.6% of every stop it ever recorded, while its 99.5th-percentile last-activity date sits at 2015-02-25, five years later. Not one of those targets falls inside a 180-day window.

FAFFAbout therefore applies a second rule alongside it. A bulk closure is a single date on which one centre stopped at least 200 targets amounting to at least 5% of all the stops it ever recorded. That is an administrative act, which is precisely the mechanism the specification describes, observed directly rather than inferred from a quantile.

Thirteen bulk-closure dates are detected across seven centres. The rule is validated by a column that was not used to build it: on a bulk-closure date, 93.4% of stop reasons are “other” or “duplicate target found” and only 6.5% name an experimental failure, whereas every other stop in the archive names a specific scientific cause 68% of the time (expression failed 33.3%, cloning failed 14.3%, purification failed 12.7%). Administrative closure and experimental failure look completely different, and the rule separates them.

The per-centre picture is the reason none of this can be skipped. JCSG censors at 0% (it never used the work stopped marker and stayed active until 2016), NESG at 58.5%, CESG at 47.6%. A model trained without this correction would learn the funding history of US structural genomics, and it would learn it very well.

🎯 The label sets

scripts/06_build_labels.py. A gate is the transition out of a stage, so the nine-rung ladder has eight of them and stage 8 is terminal.

906,573 rows, of which 842,681 enter the loss and 63,892 are held out as censored: exactly one per censored target, since a censored target still passed every gate below its last. The gradient is the one a crystallographer would predict. Getting a purified protein to crystallise is the wall at 36.2%, and once a structure exists depositing it is near-automatic at 97.1%.

L2 is one row per uncensored target, deposited or stalled_at_{g}, for Brier score and expected calibration error: 271,619 rows at a 3.86% positive rate, against the specification’s expected 4%.

L3 is 172,695 within-cluster preference pairs, uncensored, differing by more than one gate. 72,505 are hard (the two proteins also share a 70%-identity cluster) and 37,760 of those are same-centre, which is the most valuable stratum in the corpus: the same lab, a near-identical protein, a different fate, so the difference is the construct or the host rather than the pipeline. The per-cluster cap keeps hard pairs first rather than an arbitrary slice.

🧬 Features

scripts/05_derive_features.py, with the sequence functions isolated in features_seq.py and cross-checked against Biopython (GRAVY matches exactly, isoelectric point within 0.6 units on differing pKa sets).

Two things worth stating. The archive-context features are computed over the training split only, with the target’s own contribution subtracted back out, because without that leave-one-out a target reads its own fate off its cluster’s base rate and every metric flatters itself. And the transmembrane predictor validates against the archive’s own annotation: targets the archive calls membrane proteins average 9.57 predicted helices against 0.78 for the rest, and 89.4% of them are called polytopic against 7.5%.

Host and tag come from mining the 1,501 shared protocol documents that 95.5% of trials point at, which is why a few hundred kilobytes of free text annotates half the archive.

📉 The baseline, and the leak it caught

baseline/gbm_baseline.py. The specification calls this non-negotiable and it earned that immediately: the first run scored a mean AUROC of 0.985 across the gates and exactly 1.0000 on the terminal outcome, on a clean cluster-held-out split. That is the meaningless result the specification warns about, and it arrived through the feature columns rather than through the split.

Three columns were the answer in disguise. A PDB reference is present for 49.5% of targets that cleared crystallisation and 0.1% of those that did not, because a deposition reference means it was deposited. The method field is derived from the status history, so xray carries a mean maximum stage of 6.68 against 1.40 for none. Successful targets average 16.4 trials against 4.1. config/features.yaml now separates what a scientist knows before ordering the gene from what was recorded while the attempt ran, and a test reads the boundary back off the saved boosters.

A subtler one: host, tag and protease are mined from the protocol a trial referenced, so their presence tracks progression. At the first gate, host is missing for 96.2% of failures and 27.3% of successes, because a target that died at selection never had an expression protocol attached. They are excluded from the headline and evaluated separately on rows that all declare a protocol, where only the choice varies.

Cluster-held-out, 37 prediction-time features:

All four configurations:

The last row is kept as a negative control and asserted by a test, because a check that has only ever passed is not a check. The gap between the first row and the last is what the post-hoc block gives away.

Archive-context features deserve one caveat. Under the cluster split they are structurally absent at test time: whole clusters are held out, so 0.0% of test targets have a precedent against 91.9% of training targets. The temporal split is the one where precedent is both available (82.3% of test targets) and honest, since a 2014 target may have pre-2014 precedents in its own cluster.

🧪 Evaluation

eval/eval_calibration.py scores the GBM today and a served adapter with --llm-endpoint, reporting Brier, 10-bin expected calibration error, per-gate AUROC, bottleneck top-1 accuracy and the GBM delta.

The temporal reliability table shows the honest shape of forecasting the future: the model is systematically under-confident in the upper bins (predicting 0.75 where 0.86 is observed) and over-confident in the lower ones. Expected calibration error more than doubles against the cluster split, which is what a real 2014-onwards deployment would have felt.

eval/eval_generative.py has two halves. The automatic half fails the run outright on a hallucinated identifier, a target id or PDB code in a completion that was not in its own prompt, since that means facts leaked into the weights that should have stayed in retrieval. Building the identifier pattern from the archive’s 41 real centre names rather than a general one matters: [A-Z][A-Za-z0-9]+-... also matches ordinary hyphenated English, and reported “Gate-by” out of “Gate-by-gate outlook” as a hallucination in 10 of 40 cases. The manual half writes a 40-case rubric form for a domain expert, which is the acceptance test and is not automatable.

Two structural points the harness had to be built around. Cluster-held-out prompts carry no precedent table at all, because whole clusters are held out, so the citation and censoring checks score zero there for a structural reason rather than a model one. The generative eval therefore grades the temporal held-out set, where 1,433 of 2,000 records carry precedents and 1,265 carry censored context. And sampling from the head of a corpus written in token-length order systematically picks the shortest, least-context records: that drew 2 cases with censored context where the full set has 1,265.

🎓 Roadmap and the science

The full plan is in PROJECT_PLAN.md and the specification in faffabout_build_spec_v1.md. The parts that matter most:

• Censoring (Phase 2). For each centre, the 99.5th percentile of its last activity date marks when it effectively stopped reporting. A target below gate 8 whose last activity falls within 180 days of that date, or of the 2017-06-30 freeze, is censored, not failed. Expect 15 to 20% of the archive.
• Three label families. L1 gate transitions (cleared, failed, censored) for every gate a target actually entered; L2 terminal outcome for calibration; L3 within-cluster preference pairs for an optional DPO stage.
• Four negative-mining rules, in code. Censored is never a negative. Not-attempted is never a negative. Hard negatives (same cluster, over 70% identity, divergent fate) are tagged and oversampled. Easy negatives are capped at 25% of the negative mass.
• Three splits. Cluster-held-out 80/10/10, leave-one-centre-out, and temporal (train before 2014, test from 2014).
• GBM first. The gradient-boosted baseline on the same features is expected to beat the LLM on AUC. The LLM earns its place with the narrative, attribution, construct recommendation and orthologue ranking, and any hallucinated target or PDB ID is an automatic eval failure.

✅ To Do

• ✅ Phase 1: acquire. 01_fetch_zenodo.py downloads, md5-verifies and extracts the 833 MB tarball, records the CC-BY-SA-4.0 licence and inventories Documentation/ (schema XSD and PDF, controlled vocabulary spreadsheet, contributors list)
• ✅ Phase 1: parse. 02_parse_tt_xml.py streams all 41 per-centre XML files in parallel with lxml.iterparse (element names verified against the XSD, not guessed) into six Parquet tables plus a deduplicated FASTA; the per-centre files were verified to hold exactly the same 335,771 targets as tt.xml.gz
• ✅ Phase 1: normalise. 03_normalise_status.py reads the vocabulary spreadsheet, asserts every value is mapped in config/status_map.yaml, writes the canonical ladder and a per-target max_stage summary, and reports unmapped values explicitly
• ✅ Phase 1: cluster. 04_cluster_sequences.sh runs MMseqs2 at 30% identity and resolves every distinct protein sequence to a cluster id
• ✅ Phase 1: query layer and acceptance. build_duckdb.py attaches DuckDB views over the Parquet directory and prints the acceptance report
• ✅ Phase 1: PDB resolution. fetch_pdb_metadata.py fills outcomes.resolution from RCSB: 11,607 of 11,802 distinct ids resolve (the rest are obsolete), 9,440 carry a resolution
• ✅ Phase 2: censoring. Two rules in scripts/censoring.py, both stored separately and configurable in config/labels.yaml: the specification’s per-centre wind-down (99.5th-percentile last activity, 180-day window, plus the global freeze) and a new bulk-closure detector. 19.03% of the archive is censored, inside the 12 to 22% gate
• ✅ Phase 2: labels. scripts/06_build_labels.py builds L1 (906,573 gate transitions over eight gates, 842,681 in loss), L2 (271,619 uncensored terminal outcomes, 3.86% deposited) and L3 (172,695 within-cluster preference pairs, 72,505 of them hard at >70% identity)
• ✅ Phase 2: features. scripts/05_derive_features.py and features_seq.py: composition (pI, GRAVY, charge, cysteines, aromaticity, low complexity), topology (transmembrane helices, signal peptide, FoldIndex disorder by terminus), taxonomy for 98.9% of targets, host and tag mined from the 1,501 shared protocol documents, and leave-one-out archive context. The raw sequence never enters a prompt
• ⬜ Phase 2: ESM-2 features. Disorder and embeddings on ZeroGPU, cached by seq_md5, to replace the local FoldIndex proxy
• ✅ Phase 2: splits. scripts/splits.py: cluster-held-out at exactly 80/10/10 with zero cluster and zero sequence leakage, leave-one-centre-out across five centres, and a temporal split that excludes the 7,846 targets straddling the 2014 boundary
• ✅ Phase 2: SFT corpus. scripts/07_build_sft.py: 120,000 training records in the specification’s five-task mix, 2,000 each for validation and test drawn from held-out clusters, 10 paraphrases per template, sorted by token length for MLX padding
• ✅ Phase 3: GBM baseline. baseline/gbm_baseline.py: LightGBM per gate and on the terminal outcome, across the cluster, temporal and declared-protocol configurations, with a leak demonstration as a negative control. Mean AUROC 0.839 cluster-held-out, terminal 0.866
• ✅ Phase 3: feature provenance. config/features.yaml separates prediction-time from post-hoc features, after the first baseline scored a meaningless 0.985
• ✅ Phase 3: licence and model source. meta-llama/Llama-3.1-8B-Instruct is already accessible with the existing HuggingFace token, and mlx-community/Meta-Llama-3.1-8B-Instruct-8bit is the same model pre-quantised, ungated, at 8.54 GB instead of 25 GB for a local convert
• ⬜ Phase 3: LoRA. Config, launcher and preflight are written and validated against the installed mlx-lm. Blocked on a reboot: 1.8 GB of swap is in use and the machine has been up five days
• ⬜ Phase 3: DPO. Optional preference stage on the L3 pairs if the SFT model is overconfident
• ⬜ Phase 3: fuse. --de-quantize needs roughly 16 GB more than the 15 GB free; serving from base plus adapter avoids it
• ✅ Phase 3: eval harness. eval/eval_calibration.py (Brier, 10-bin ECE, per-gate AUROC, bottleneck top-1, GBM delta, reliability tables) and eval/eval_generative.py (automatic hallucinated-identifier check plus a 40-case rubric form). Both run against the GBM today and against a served adapter with --llm-endpoint
• ⬜ Phase 3: grade the narratives. 40 cases in eval/generative_review.md, once a model exists
• ⬜ Phase 4: serve. Flask app with the Pipeline Rig front end, POST /predict contract, censoring visible at all times, live at faffabout.mdeller.com and listed on the mdeller.com launcher
• ⬜ Licence. Choose the code licence (the data is CC-BY-SA-4.0; Llama 3.1 has its own community licence to check before any model redistribution)

👤 Author

Marc C. Deller, D.Phil.
Structural biologist & drug discovery scientist