BUILD_PLAN = """
Verified steps (self.quit at first occurrence):
1. Daily fundamental universe, eligibility filter, daily adjusted price accumulation (quit: counts on first selection).
2. Monthly state: month-end px, LME, Dec ME, July annual snapshots with lag (quit: firms with complete annual data).
3. 25 characteristics + dMKT/dRf from FF, rank-normalisation, panel with lagged targets (quit: first full cross-section).
4. VFI + simulation (multi-seed, Table 2 moment rule) + base network in initialize (quit: moments and base loss).
5. Fine-tune / predict / decile weights / orders on shortened window (train 2020H1, val 2020H2, trade 2021Q1).
6. Full run 2002-12-01 -> today, verify orders.
Conventions fixed while building: r_{a_b} = P_{M-b}/P_{M-a} - 1 (FNW); beta needs >=200 of 252 daily obs; ivol/vol
need >=15 daily obs in prior month; NaN preferred stock/deferred taxes/LTD/inventory/payout items -> 0; core items
(at, sale, cogs, ib, gp, oiadp, gross_ppe, SH) NaN -> firm excluded; sim ME = v - d (ex-dividend value);
sim factors MKT (VW all), SMB (median ME split), HML (30/70 k/ME) re-sorted each June; sim beta/ivol/vol from
trailing 12 monthly returns; panel selection rule: minimise sum of squared relative deviations of (EP, mkt sd,
rf mean, rf sd, individual vol=28.5) from Table 2 data column.
Build log:
- Step 3 verified: first complete cross-section is Nov-2005 (needs 36 month-end prices from Nov-2002 and two June
  snapshots): n=955 firms (Morningstar coverage in the early 2000s is thinner than CRSP/Compustat; count grows later).
  AAPL row reconciled against FY2004 10-K and Oct-2005 prices.
- Data limitation found: the daily Fama-French risk-free rate is published to 0.01%/day precision, so the monthly
  RF (compounded daily RF) moves in coarse steps and dRf is 0 in most months with occasional +-0.002 jumps. Kept as
  the client decided (directive A: FF dataset); recorded for the final report. dMKT is unaffected.
- Step 4: VFI converges in ~40 outer iterations (Howard steps), k grid top = 17.0 (formula) vs 15 stated; 10 seeds
  simulated (5000 firms x 1400 months, burn 500), seed with the lowest moment score kept; 899 months x 5000 firms =
  4.5M simulated rows; base net trained one epoch per trading session over the first 5 sessions (ADAM state kept),
  to stay within the per-step time limit.
- Step 5 (fine-tune on shortened window, train 2020H1 / val 2020H2): first run showed validation target variance
  10.33 (expected ~0.01-0.05) driven by one firm-month: TOPS Aug-2020 adjusted price 0.072 -> 31.20 on 2020-08-10
  (x430 jump, reverse split not adjusted in the price series; research notebook cell 8). APVO also shows corrupted
  adjusted prices (~5,000,000 with split factor 586,080 and zero volume). Both blocked by Security ID
  (self._blocked_assets: "TOPT T0KDYN9C3IHX", "APVOV WCD405GN81GL") per the data-issues skill. All other |y|>2
  firm-months inspected (IBIO, UONE, VVPR, GNUS, CLIR, POLA, KODK, CCXI, APA, AR, SM, CDEV, ALT, CLDX, PEIX, CBAT,
  FTEK, CAPR, MGEN) are genuine multi-week 2020 small-cap rallies without snap-back (notebook cell 9) and are kept:
  no return clipping or winsorising is applied (the paper applies none).
- Step 5 verified (74b480833d2c11d6daa9318970c9985d): after blocking, var_val 0.0465, fine-tune 6 epochs, val MSE
  0.0515 -> 0.0499; first prediction Dec-2020 n=1685, 168 long / 169 short, sum_w 0, gross 1.0.
- Step 5b orders (539814ad91d280bedf0921a42da1e1ba, trade 2021-01 -> 2021-04-15): 1632 market-on-open orders placed
  08:00 ET on the first trading day of each month (Jan 4, Feb 1, Mar 1, Apr 1) and filled at the open; no
  invalid/rejected orders; old book liquidated ("Liquidated" tag) and new deciles entered the same morning,
  including the tight case where month-end selection (Apr 1 00:00) and rebalance (Apr 1 08:00) fall on one day.
- Step 6 first full run (10c6750fc802c79a5d4e149e9862b0e4): fine-tune at Dec-2020 on 130,096 train rows
  (2006-2015) and 88,751 validation rows (2016-2020), 8 epochs, val MSE 0.0367 -> 0.0292 (= val variance 0.0292);
  |y|>2 training/validation firm-months all genuine (ratios <= ~6x). Two platform issues found and handled in
  _rebalance: (a) one order attempt for a security without a price bar that morning (WOOF 2024-07-01) -> targets
  without a price are skipped and logged; (b) holdings whose data feed ends without a delisting record
  (acquisitions / ticker changes such as GRUB, GMLP, ZGNX, APRN, ATUS, GLT) cannot be priced or sold: the monthly
  liquidation orders were cancelled with price 0; now such holdings are left untouched and logged instead of
  re-submitting unpriceable orders. Explicit zero-weight targets replace liquidate_existing_holdings (verified
  identical result on the 2021Q1 window: a46f3184be1a3a20e2fc3e3594e0ca17).
- FINAL validation run f00162ff7a9c1b33f90e4df040f47de0 (2002-12-01 -> 2026-09-17): 37,722 market-on-open orders,
  all filled, none invalid/cancelled/rejected; 69 monthly rebalances Jan-2021 -> Sep-2026, ~170-210 names per leg;
  net -26.8%, Sharpe -0.95, beta 0.00, max DD 30.8%. Predicted returns: top decile ~2.4-4.0%/mo, bottom ~-0.8 to
  +0.1%/mo. WOOF (S9WD3HLRHX0L) had no price bar at five 2024 rebalances (weight ~0.1%) and was skipped.
- Client revision (after the final report): (1) backtest window 2021-09-01 -> 2026-09-01; the real panel
  (Dec-2002 onwards), the base network and the fine-tuned network are now built during warm-up
  (set_warm_up back to 2002-12-01: the daily fundamental universe selection runs through warm-up, returns no
  symbols until the final pre-start month-end selection, scheduled rebalances are guarded by is_warming_up);
  train 2006-2015 / validate 2016-2020 unchanged; first trade at the 2021-09-01 open. (2) Build scaffolding
  removed: _verify switch, all diagnostic self.log lines, outlier lists and counters kept only for logging.
  (3) Security initializer replaced by settings.seed_initial_prices = True.
"""

BRIEF = """
IMPLEMENTATION BRIEF - "Teach Machine Asset Pricing" (Chengyu Bai, SSRN 4598745, March 2024)

STRATEGY IN ONE PARAGRAPH
A feed-forward neural network (2 hidden layers, 27 inputs) is first pre-trained on a panel SIMULATED from a
production-based asset-pricing model (simplified Zhang 2005, Table 1 calibration), then fine-tuned on real US
stock data ("transfer learning", TL model). Each month the fine-tuned network predicts next-month returns for
all eligible stocks from 25 firm characteristics + 2 macro inputs (all rank-normalised to [-1,1] per month),
stocks are sorted into deciles on the prediction, and the portfolio goes long the top decile and short the
bottom decile, value-weighted, rebalanced monthly (Table 5, main result: 2-hidden-layer TL model).

1. UNIVERSE (Section 3.1)
- US common stocks on NYSE / AMEX / NASDAQ. QC: Morningstar Fundamental universe, security_reference.security_type
  == "ST00000001" and is_primary_share, exchange_id in NYSE/ASE(NYSE American)/NASDAQ.
- Exclude utilities (SIC 4900-4999) and financials (SIC 6000-6999). QC has no SIC codes -> exclude Morningstar
  sectors Utilities (207), Financial Services (103) and Real Estate (104; SIC 65xx is inside 6000-6999).
- Complete-case rule: a stock enters the month's cross-section only if ALL 25 characteristics are available
  (no imputation). No size or price filter (paper has none).

2. INPUTS (27), all cross-sectionally rank-normalised to [-1,1] each month (GKX 2020)
Annual accounting variables (Compustat annual in the paper -> Morningstar .twelve_months here) are taken from
the latest annual statement available at the end of June and held fixed until next June (Fama-French timing);
ME for the ratios is the previous December market cap (Fama-French). Lagged (t-1) items are the values stored
at the previous June snapshot. Monthly variables are updated each month-end.
Accounting (items 1-15 of Table 3 - reconstructed, see REQUEST 1):
  A2ME       total_assets / ME
  BEME       BE / ME ; BE = stockholders_equity + non_current_deferred_taxes_liabilities - preferred_stock
  CTO        total_revenue / lagged total_assets
  dCEQ       (ceq_t - ceq_t-1) / ceq_t-1 ; ceq = stockholders_equity - preferred_stock
  E2P        net_income_continuous_operations (ib) / ME
  Investment (at_t - at_t-1) / at_t-1
  NOP        (common_stock_dividend_paid + common_stock_payments - common_stock_issuance) / ME
  PCM        (total_revenue - cost_of_revenue) / total_revenue
  PM         operating_income / total_revenue
  Prof       gross_profit / BE
  Q          (total_assets + ME - ceq - non_current_deferred_taxes_liabilities) / total_assets
  ROA        ib / total_assets
  ROE        ib / BE
  SAT        total_revenue / total_assets
  Sales_g    total_revenue_t / total_revenue_t-1 - 1
Trading frictions (items 16-20):
  AT         total_assets
  LME        market cap at previous month-end
  capm_beta  slope of daily excess returns on SPY daily excess returns, trailing 12 months (Lewellen-Nagel 2006)
  ivol       std of residuals of daily returns on SPY over the previous month (Ang et al 2006)
  Total_vol  std of daily returns over the previous month (Ang et al 2006)
Past returns (items 21-25), compounded from month-end adjusted prices:
  r2_1 (month t-1), r6_2 (t-6..t-2), r12_2 (t-12..t-2), r12_7 (t-12..t-7), r36_13 (t-36..t-13)
Macro (items 26-27), first differences (Chen et al 2023):
  dMKT = MKT_t - MKT_t-1, MKT = SPY monthly total return minus rf ; dRf = rf_t - rf_t-1, rf = US Treasury
  1-month yield / 12 (USTreasuryYieldCurveRate one_month, from 2001-07).
Target: next-month simple return of the stock (paper: "next month's stock return", not excess).

3. SIMULATED PANEL AND BASE MODEL (Section 2, Appendix)
- Solve the model by value function iteration exactly as the Appendix: capital grid k_i = k_i-1 + c1 exp(c2 (i-2)),
  k1 = 0.01, c1 = 0.005, c2 = 0.1247952, N = 50; x and z on 5-point Rouwenhorst grids; Bellman (A1) with adjustment
  cost (5), SDF (6); convergence 1e-5; finer k' grid with linear interpolation. Parameters: Table 1 (alpha 0.3,
  delta 0.01, C_q 4, C_r 0.15, C_f 0, f 0.023, rho_x 0.95^(1/3), sigma_x 0.007/sqrt(1+rho_x^2+rho_x^4), xbar -3.15,
  rho_z 0.97, sigma_z 0.1, beta 0.999, gamma0 18, gamma1 -750).
- Simulate 5000 firms x 1400 months, discard the first 500 -> 900 months. Firm return R = v'/(v - d).
- Build the same 25 + 2 predictors from simulated quantities using the Table 3 mapping (k = assets and book
  equity, v = market value, y = sales, y - f = earnings, d = net payout, i and dk = investment, f = fixed cost,
  etc.), annual quantities aggregated over 12 months and updated once a year with the same 6-month lag; simulated
  market factor = value-weighted return of all simulated firms, simulated rf = 1/E[M] - 1; beta/ivol/vol from the
  simulated return series over the same windows. Rank-normalise each simulated month.
- Base model: feed-forward net, 2 hidden layers x 27 Swish units (trainable gamma), MSE, ADAM lr 0.001,
  batch 256, trained for a small number of epochs on the simulated panel (PyTorch, in initialize()).

4. FINE-TUNING ON REAL DATA (Section 3.2) - FIXED SPLIT, ONE FIT
- Paper: 1972-2022 = 15y train / 10y validation / 25y test, single split. Platform data (prices and Morningstar
  fundamentals from 1998, lags and 36-month returns first complete in 2001) forces the adaptation:
  train 2001-01..2010-12, validation 2011-01..2015-12, test (traded) 2016-01 .. today.
- All layers trainable, ADAM lr 0.0001, batch 256, early stopping with patience 3 epochs on validation MSE.
- The DD model (real data only) is the paper's comparison, not the traded strategy; not built.

5. PORTFOLIO CONSTRUCTION AND TIMING (Table 5)
- At each month-end from 2015-12 on: compute the 27 inputs, predict next-month return, sort into deciles,
  long decile 10 / short decile 1, value-weighted by LME within each leg.
- Gross exposure capped at 1x: +0.5 long leg, -0.5 short leg (paper's 100/100 scaled down).
- Orders placed at 08:00 ET on the first trading day after the month-end selection (market orders at the open),
  held one month; set_holdings targets, liquidating names that leave the legs.
- Starting cash 100,000,000 so the smallest value-weighted position is still several thousand dollars.

6. DATASETS
- US Equities (daily, adjusted) - prices, month-end returns; SPY as market proxy.
- US Fundamental Data (Morningstar) - universe, sector, exchange, all accounting fields (.twelve_months), market cap.
- US Treasury Yield Curve (USTreasuryYieldCurveRate, one_month) - risk-free rate.
- Backtest window: 2001-01-01 (panel accumulation, no trades) to today; first trade 2016-01.

DECISIONS
1. Main result = TL model with 2 hidden layers, long-short decile spread, value-weighted (Table 5). Rejected:
   long-only top decile (Figure 1), DD model, 1/3/4/5-layer variants, equal-weighted (Table 6/7).
2. Long-short sized +0.5 / -0.5 instead of the paper's implicit +1 / -1 (1x gross cap).
3. Sample dates adapted to platform history: train 2001-2010, validate 2011-2015, test 2016-today (paper 15/10/25).
   The split is FIXED (single fit), as the paper describes a single training/validation/test partition; rolling
   re-estimation rejected because the paper never mentions refits.
4. Items 1-15 of Table 3 are unreadable in the PDF; reconstructed from the paper's text/Table 7/citations and the
   Compustat item list in the Table 3 note as the 15 names above (A2ME, BEME, CTO, dCEQ, E2P, Investment, NOP, PCM,
   PM, Prof, Q, ROA, ROE, SAT, Sales_g). Alternatives that fit the citations but were dropped to reach 15: Lev, DPI2A,
   FC2Y, OA, D2A. See REQUEST 1.
5. Characteristic formulas follow Freyberger-Neuhierl-Weber (2020) as cited; Morningstar substitutes: ib ->
   net_income_continuous_operations, txditc/txdb -> non_current_deferred_taxes_liabilities, seq -> stockholders_equity,
   pstk -> preferred_stock, dvc -> common_stock_dividend_paid, prstkc -> common_stock_payments, sstk ->
   common_stock_issuance, oiadp -> operating_income, gp -> gross_profit, sale -> total_revenue, cogs -> cost_of_revenue.
6. SIC exclusions mapped to Morningstar sectors Utilities, Financial Services and Real Estate (SIC 6500-6599 is in
   the financial range). Rejected: excluding only Financial Services.
7. sigma_x read as 0.007/sqrt(1+rho_x^2+rho_x^4) (Table 1 prints "0 7/..."; Zhang 2005's quarterly value is 0.007).
8. capm_beta: trailing 12 months of daily returns vs SPY (paper cites Lewellen-Nagel 2006 without a window).
   Rejected: monthly returns over 60 months.
9. ivol: residual std from a market-only daily regression over the previous month (Ang et al 2006 use FF3; SMB/HML
   are not on the platform, so the market factor alone is used). Total_vol: daily return std, previous month.
10. Market factor = SPY total return minus rf; rf = 1-month Treasury yield / 12 (paper uses Kenneth French MKT and RF).
11. Past returns compounded (not summed) from month-end adjusted closes; r2_1 is the single previous month.
12. Target and macro returns are simple monthly returns; prediction target is raw next-month return (paper's wording),
    rejected: excess return.
13. Base-model epochs: paper says "a small number of epochs" without a number -> 5 epochs. Fine-tuning max 100 epochs
    with patience-3 early stopping. Single network, fixed random seed (paper mentions no ensembling).
14. Simulated risk factor: value-weighted market of all simulated firms; simulated beta and ivol from trailing 12 monthly
    simulated returns (simulation is monthly, no daily data exists in the model). Simulated annual items = 12-month sums
    of flows / end-of-year stocks, updated once a year with the real-data 6-month lag.
15. Complete-case rule applied to all 25 characteristics; missing values exclude the stock for that month (never
    imputed, never ranked).
16. Rebalance orders at 08:00 ET on the first trading day after month-end selection (daily data; the paper's
    end-of-month formation maps to the next open on the platform).

CLIENT DECISIONS (approved brief; these override everything above)
- Table 3 items 1-15 confirmed (real construction; model quantity):
  (1) A2ME = at / December ME (shrout x prc); k/v
  (2) BEME = BE / ME, BE = SH + txditc - PS, SH = seq, else ceq + PS, else at - lt; PS = pstkrv, else pstkl, else pstk; k/v
  (3) E2P = ib / ME; (y-f)/v
  (4) NOP = (dvc + prstkc - sstk) / ME; d/v
  (5) Q = (ME + dltt + pstkrv - invt - txdb) / ppegt (GROSS PP&E per formula); v/k
  (6) Sales_g = pct change in sale; pct change in y
  (7) Investment = pct change in at; pct change in k
  (8) dCEQ = pct change in BE; pct change in k
  (9) CTO = sale / lagged at; y_t / k_t-1
  (10) PCM = (sale - cogs)/sale; (y-f)/y
  (11) PM = oiadp / sale; (y - f - delta k)/y
  (12) Prof = gp / BE; y/k
  (13) ROA = ib / LAGGED at; (y-f)/k_t-1
  (14) ROE = ib / LAGGED BE; (y-f)/k_t-1
  (15) SAT = sale / at; y/k
- A. Use the platform Fama-French factors dataset (docs: quantconnect.com/docs/v2/writing-algorithms/datasets/quantconnect/fama-french)
  for MKT excess return and RF (dMKT, dRf), for capm_beta, and for ivol as the FF3 residual (Ang et al 2006). NOT SPY, NOT Treasury curve.
- B. Industry exclusions via asset_classification.sic: exclude 4900-4999 and 6000-6999 directly.
- C. Panel selection: simulate several panels with different seeds, compare cross-sectional moments with Table 2, keep the closest; record rule.
- D. Macro inputs enter as raw first differences (same for all firms); only the 25 firm characteristics are rank-normalised.
- E. Dates: train 2006-2015, validate 2016-2020, trade from January 2021 to today (no end date). Panel accumulation shifts the same way.
- F. Timing: monthly work at the START of the month: first trading day (date_rules.month_start) at 08:00 ET, inputs from data
  through the prior close, predict, sort, send orders (fill at that day's open).

REQUESTS (answered)
1. Table 3 items (1)-(15) of the paper (page 28, landscape table) could not be read from the PDF (OCR garbled) and
   no other copy is accessible (SSRN download blocked, no author code). Please confirm or correct the 15 accounting
   characteristic names and formulas listed above, or supply a readable copy of page 28 (text or Object Store key).
2. If you have Freyberger, Neuhierl and Weber (2020), "Dissecting Characteristics Nonparametrically", RFS 33(5),
   its Table 1 (characteristic definitions) would let me verify the exact formulas; otherwise I proceed with the
   definitions above.
"""

"""Structural data-generating process of Bai (2024), 'Teach Machine Asset Pricing'.

Solves the Zhang (2005)-type investment model by value function iteration (Appendix A of the
paper), simulates a panel of firms, and builds the 27 simulated predictors of Table 3 the same
way the real characteristics are built. Pure numpy, no LEAN dependency.
"""
import numpy as np


class ModelParams:
    """Table 1 calibration (monthly)."""

    def __init__(self) -> None:
        self.alpha = 0.3
        self.delta = 0.01
        self.cq = 4.0
        self.cr = 0.15
        self.cf = 0.0
        self.f = 0.023
        self.beta = 0.999
        self.gamma0 = 18.0
        self.gamma1 = -750.0
        self.rho_x = 0.95 ** (1.0 / 3.0)
        self.sigma_x = 0.007 / np.sqrt(1.0 + self.rho_x ** 2 + self.rho_x ** 4)
        self.xbar = -3.15
        self.rho_z = 0.97
        self.sigma_z = 0.1
        self.nx = 5
        self.nz = 5
        self.nk = 50
        self.k1 = 0.01
        self.c1 = 0.005
        self.c2 = 0.1247952
        self.n_fine = 1000
        self.tol = 1e-5
        self.howard_steps = 30
        self.max_iter = 5000


def rouwenhorst(n: int, rho: float, sigma: float, mean: float):
    """Rouwenhorst (1995) discretisation of y' = (1-rho) mean + rho y + sigma eps."""
    p = (1.0 + rho) / 2.0
    q = p
    theta = np.array([[p, 1.0 - p], [1.0 - q, q]])
    for i in range(3, n + 1):
        z = np.zeros((i, i))
        z[:-1, :-1] += p * theta
        z[:-1, 1:] += (1.0 - p) * theta
        z[1:, :-1] += (1.0 - q) * theta
        z[1:, 1:] += q * theta
        z[1:-1, :] /= 2.0
        theta = z
    psi = sigma / np.sqrt(1.0 - rho ** 2) * np.sqrt(n - 1.0)
    grid = np.linspace(mean - psi, mean + psi, n)
    return grid, theta


class ModelSolution:
    """Value function v(k,x,z) and policy k'(k,x,z) on the state grid."""

    def __init__(self, params: ModelParams) -> None:
        self.p = params
        p = params
        kg = np.empty(p.nk)
        kg[0] = p.k1
        for i in range(1, p.nk):
            kg[i] = kg[i - 1] + p.c1 * np.exp(p.c2 * (i - 1))
        self.kg = kg
        self.kf = np.interp(np.linspace(0.0, p.nk - 1.0, p.n_fine), np.arange(p.nk), kg)
        self.xg, self.qx = rouwenhorst(p.nx, p.rho_x, p.sigma_x, p.xbar)
        self.zg, self.qz = rouwenhorst(p.nz, p.rho_z, p.sigma_z, 0.0)
        gam = p.gamma0 + p.gamma1 * (self.xg - p.xbar)
        mx = p.beta * np.exp(gam[:, None] * (self.xg[:, None] - self.xg[None, :]))
        self.pm = self.qx * mx
        self.v = None
        self.kpol = None
        self.iterations = 0
        self.max_change = np.inf

    def _adjustment(self, i: np.ndarray, k: np.ndarray, y: np.ndarray = None) -> np.ndarray:
        p = self.p
        h = p.cr * np.abs(i) * (i < 0) + 0.5 * p.cq * (i / k) ** 2 * k
        if p.cf != 0.0 and y is not None:
            h = h + p.cf * y * (i != 0)
        return h

    def solve(self) -> None:
        p = self.p
        kg, kf, xg, zg = self.kg, self.kf, self.xg, self.zg
        nk, nf, nx, nz = p.nk, p.n_fine, p.nx, p.nz
        y = np.exp(xg[None, :, None] + zg[None, None, :]) * kg[:, None, None] ** p.alpha
        inv = kf[None, :] - (1.0 - p.delta) * kg[:, None]
        h = self._adjustment(inv, kg[:, None])
        pay = y[:, None, :, :] - (inv + h + p.f)[:, :, None, None]
        lo = np.clip(np.searchsorted(kg, kf) - 1, 0, nk - 2)
        w = (kf - kg[lo]) / (kg[lo + 1] - kg[lo])
        xi, zi = np.meshgrid(np.arange(nx), np.arange(nz), indexing="ij")
        v = np.zeros((nk, nx, nz))
        for it in range(p.max_iter):
            cont = np.einsum("ab,cd,kbd->kac", self.pm, self.qz, v)
            cont_f = cont[lo] * (1.0 - w)[:, None, None] + cont[lo + 1] * w[:, None, None]
            obj = pay + cont_f[None]
            pol = np.argmax(obj, axis=1)
            v_new = np.take_along_axis(obj, pol[:, None], axis=1)[:, 0]
            change = np.max(np.abs(v_new - v))
            v = v_new
            pay_pol = np.take_along_axis(pay, pol[:, None], axis=1)[:, 0]
            lo_p = lo[pol]
            w_p = w[pol]
            for _ in range(p.howard_steps):
                cont = np.einsum("ab,cd,kbd->kac", self.pm, self.qz, v)
                cont_p = cont[lo_p, xi[None], zi[None]] * (1.0 - w_p) + cont[lo_p + 1, xi[None], zi[None]] * w_p
                v = pay_pol + cont_p
            self.iterations = it + 1
            self.max_change = change
            if change < p.tol:
                break
        self.v = v
        self.kpol = kf[pol]

    def interp(self, arr: np.ndarray, k: np.ndarray, x: float, z: np.ndarray) -> np.ndarray:
        """Trilinear interpolation of arr(k,x,z) at vector k, scalar x, vector z."""
        xg, zg, kg = self.xg, self.zg, self.kg
        ix = int(np.clip(np.searchsorted(xg, x) - 1, 0, len(xg) - 2))
        wx = (x - xg[ix]) / (xg[ix + 1] - xg[ix])
        a = arr[:, ix, :] * (1.0 - wx) + arr[:, ix + 1, :] * wx
        ik = np.clip(np.searchsorted(kg, k) - 1, 0, len(kg) - 2)
        wk = (k - kg[ik]) / (kg[ik + 1] - kg[ik])
        iz = np.clip(np.searchsorted(zg, z) - 1, 0, len(zg) - 2)
        wz = (z - zg[iz]) / (zg[iz + 1] - zg[iz])
        return (a[ik, iz] * (1.0 - wk) * (1.0 - wz) + a[ik + 1, iz] * wk * (1.0 - wz)
                + a[ik, iz + 1] * (1.0 - wk) * wz + a[ik + 1, iz + 1] * wk * wz)


class SimulatedPanel:
    """Simulated firm panel: n_firms x n_periods monthly arrays (float32)."""

    def __init__(self, sol: ModelSolution, n_firms: int, n_periods: int, seed: int) -> None:
        self.sol = sol
        self.n_firms = n_firms
        self.n_periods = n_periods
        self.seed = seed
        p = sol.p
        rng = np.random.default_rng(seed)
        nT, nJ = n_periods, n_firms
        self.k = np.empty((nJ, nT), dtype=np.float32)
        self.y = np.empty((nJ, nT), dtype=np.float32)
        self.i = np.empty((nJ, nT), dtype=np.float32)
        self.d = np.empty((nJ, nT), dtype=np.float32)
        self.v = np.empty((nJ, nT), dtype=np.float32)
        self.x = np.empty(nT, dtype=np.float64)
        self.rf = np.empty(nT, dtype=np.float64)   # risk-free rate for period t -> t+1, known at t
        sd_x = p.sigma_x / np.sqrt(1.0 - p.rho_x ** 2)
        sd_z = p.sigma_z / np.sqrt(1.0 - p.rho_z ** 2)
        x = p.xbar + sd_x * rng.standard_normal()
        z = sd_z * rng.standard_normal(nJ)
        k = rng.uniform(sol.kg[0], sol.kg[-1], nJ)
        xlo, xhi = sol.xg[0], sol.xg[-1]
        zlo, zhi = sol.zg[0], sol.zg[-1]
        for t in range(nT):
            x = min(max(x, xlo), xhi)
            z = np.clip(z, zlo, zhi)
            k = np.clip(k, sol.kg[0], sol.kg[-1])
            y = np.exp(x + z) * k ** p.alpha
            kn = sol.interp(sol.kpol, k, x, z)
            inv = kn - (1.0 - p.delta) * k
            h = sol._adjustment(inv, k, y)
            d = y - inv - h - p.f
            v = sol.interp(sol.v, k, x, z)
            gam = p.gamma0 + p.gamma1 * (x - p.xbar)
            mu = (1.0 - p.rho_x) * p.xbar + p.rho_x * x
            em = p.beta * np.exp(gam * (x - mu) + 0.5 * gam ** 2 * p.sigma_x ** 2)
            self.k[:, t] = k
            self.y[:, t] = y
            self.i[:, t] = inv
            self.d[:, t] = d
            self.v[:, t] = v
            self.x[t] = x
            self.rf[t] = 1.0 / em - 1.0
            x = mu + p.sigma_x * rng.standard_normal()
            z = p.rho_z * z + p.sigma_z * rng.standard_normal(nJ)
            k = kn
        self.me = (self.v - self.d).astype(np.float32)       # ex-dividend market value
        ret = np.empty_like(self.v)
        ret[:, 0] = np.nan
        ret[:, 1:] = self.v[:, 1:] / self.me[:, :-1]           # gross return over period t (t-1 -> t)
        self.ret = ret
        w = self.me[:, :-1] / self.me[:, :-1].sum(axis=0, keepdims=True)
        mkt = np.full(nT, np.nan)
        mkt[1:] = (w * ret[:, 1:]).sum(axis=0) - 1.0
        rf_real = np.full(nT, np.nan)
        rf_real[1:] = self.rf[:-1]                              # rate earned over period t
        self.mkt = mkt
        self.rf_realised = rf_real
        self.mkt_ex = mkt - rf_real

    def moments(self, burn: int) -> dict:
        s = slice(burn, self.n_periods)
        ex = self.mkt_ex[s]
        rf = self.rf_realised[s]
        ind = np.nanstd(self.ret[:, s] - 1.0, axis=1)
        return {
            "ep": 1200.0 * np.mean(ex),
            "mkt_sd": 100.0 * np.sqrt(12.0) * np.std(ex),
            "rf": 1200.0 * np.mean(rf),
            "rf_sd": 100.0 * np.sqrt(12.0) * np.std(rf),
            "ind_vol": 100.0 * np.sqrt(12.0) * np.mean(ind),
        }


TARGET_MOMENTS = {"ep": 8.7, "mkt_sd": 15.7, "rf": 3.0, "rf_sd": 0.7, "ind_vol": 28.5}


def moment_score(m: dict) -> float:
    return float(sum(((m[k] - v) / v) ** 2 for k, v in TARGET_MOMENTS.items()))


def _vw(ret: np.ndarray, w: np.ndarray, mask: np.ndarray) -> float:
    ws = w[mask].sum()
    return float((ret[mask] * w[mask]).sum() / ws) - 1.0 if ws > 0 else 0.0


def build_features(sim: SimulatedPanel, burn: int, june_offset: int = 6):
    """Build the 27 predictors (Table 3 order) and next-month target for months burn..T-2.

    Returns X (N,27) float32 rank-normalised (cols 0-24) with raw dMKT/dRf (25,26), y (N,) float32.
    Accounting years end at month T = J - 6 for every June month J (J % 12 == june_offset);
    annual items become available at J and are used for months J..J+11 (Fama-French convention).
    """
    p = sim.sol.p
    nJ, nT = sim.n_firms, sim.n_periods
    k, y, inv, d, me, ret = sim.k, sim.y, sim.i, sim.d, sim.me, sim.ret
    mkt_ex, rf = sim.mkt_ex, sim.rf_realised
    logret = np.log(ret[:, 1:])
    cum = np.concatenate([np.zeros((nJ, 1), dtype=np.float32), np.cumsum(logret, axis=1)], axis=1)
    # ---- annual blocks keyed by June month J
    annual = {}
    for J in range(june_offset, nT, 12):
        T = J - 6
        if T - 23 < 0 or T + 1 >= nT:
            continue
        cur = slice(T - 11, T + 1)
        prev = slice(T - 23, T - 11)
        Y = y[:, cur].sum(axis=1)
        Yp = y[:, prev].sum(axis=1)
        D = d[:, cur].sum(axis=1)
        DEP = p.delta * k[:, cur].sum(axis=1)
        F = 12.0 * p.f
        AT = k[:, T + 1]
        ATp = k[:, T - 11]
        V = me[:, T]
        prof = Y - F
        annual[J] = {
            "A2ME": AT / V, "BEME": AT / V, "E2P": prof / V, "NOP": D / V, "Q": V / AT,
            "Sales_g": Y / Yp - 1.0, "Investment": AT / ATp - 1.0, "dCEQ": AT / ATp - 1.0,
            "CTO": Y / ATp, "PCM": prof / Y, "PM": (prof - DEP) / Y, "Prof": Y / AT,
            "ROA": prof / ATp, "ROE": prof / ATp, "SAT": Y / AT, "AT": AT,
        }
    order = ["A2ME", "BEME", "E2P", "NOP", "Q", "Sales_g", "Investment", "dCEQ", "CTO", "PCM",
             "PM", "Prof", "ROA", "ROE", "SAT", "AT"]
    # ---- SMB / HML with June re-sorts (2x3 on size median and BEME 30/70)
    smb = np.full(nT, np.nan)
    hml = np.full(nT, np.nan)
    junes = sorted(annual.keys())
    groups = None
    for t in range(1, nT):
        js = [J for J in junes if J <= t]
        if not js:
            continue
        J = js[-1]
        if t == J or groups is None:
            size = me[:, J]
            beme = annual[J]["BEME"]
            small = size <= np.median(size)
            lo_b, hi_b = np.percentile(beme, [30.0, 70.0])
            low = beme <= lo_b
            high = beme > hi_b
            mid = ~low & ~high
            groups = [(small & low), (small & mid), (small & high), (~small & low), (~small & mid), (~small & high)]
        w = me[:, t - 1]
        r = ret[:, t]
        pr = [_vw(r, w, g) for g in groups]
        smb[t] = (pr[0] + pr[1] + pr[2]) / 3.0 - (pr[3] + pr[4] + pr[5]) / 3.0
        hml[t] = (pr[2] + pr[5]) / 2.0 - (pr[0] + pr[3]) / 2.0
    # ---- monthly panel
    months = [t for t in range(max(burn, 36), nT - 1) if any(J <= t for J in junes)]
    n_rows = len(months) * nJ
    X = np.empty((n_rows, 27), dtype=np.float32)
    Yt = np.empty(n_rows, dtype=np.float32)
    scale = 2.0 / (nJ - 1.0)
    row = 0
    for t in months:
        J = max(Jx for Jx in junes if Jx <= t)
        a = annual[J]
        raw = np.empty((nJ, 25), dtype=np.float64)
        for c, name in enumerate(order):
            raw[:, c] = a[name]
        raw[:, 16] = me[:, t]                                    # LME
        win = slice(t - 11, t + 1)
        fr = ret[:, win] - 1.0 - rf[win][None, :]                # firm excess returns, 12 months
        xm = np.column_stack([np.ones(12), mkt_ex[win]])
        b1 = np.linalg.lstsq(xm, fr.T, rcond=None)[0]
        raw[:, 17] = b1[1]                                       # CAPM beta
        x3 = np.column_stack([np.ones(12), mkt_ex[win], smb[win], hml[win]])
        b3 = np.linalg.lstsq(x3, fr.T, rcond=None)[0]
        resid = fr.T - x3 @ b3
        raw[:, 18] = resid.std(axis=0)                           # idiosyncratic vol
        raw[:, 19] = (ret[:, win] - 1.0).std(axis=1)             # total vol
        raw[:, 20] = np.exp(cum[:, t] - cum[:, t - 1]) - 1.0     # r2_1
        raw[:, 21] = np.exp(cum[:, t - 1] - cum[:, t - 5]) - 1.0     # r6_2
        raw[:, 22] = np.exp(cum[:, t - 1] - cum[:, t - 11]) - 1.0    # r12_2
        raw[:, 23] = np.exp(cum[:, t - 6] - cum[:, t - 11]) - 1.0    # r12_7
        raw[:, 24] = np.exp(cum[:, t - 12] - cum[:, t - 35]) - 1.0   # r36_13
        ranks = np.argsort(np.argsort(raw, axis=0), axis=0)
        X[row:row + nJ, :25] = ranks * scale - 1.0
        X[row:row + nJ, 25] = mkt_ex[t] - mkt_ex[t - 1]
        X[row:row + nJ, 26] = rf[t] - rf[t - 1]
        Yt[row:row + nJ] = ret[:, t + 1] - 1.0
        row += nJ
    return X, Yt, len(months)

FIELDS = """
FAMA-FRENCH DATASET (QuantConnect first-party, docs /datasets/quantconnect/fama-french):
- add: self._ff = self.add_data(FamaFrench, "FF", Resolution.DAILY).symbol
- Start Jan 1998, daily, tz America/New_York, values arrive the day AFTER the session measured.
- Fields: momentum, value (=HML), is_estimate, plus Mkt-RF, SMB, RMW, CMA, RF (exact python names to introspect
  in research notebook). Every factor nullable (None) -> check before use.
- history: self.history(self._ff, n, Resolution.DAILY) (DataFrame) or self.history[FamaFrench](...)
- asset_classification.sic available for SIC exclusion (4900-4999, 6000-6999).

VERIFIED IN RESEARCH NOTEBOOK (cells 0-2):
- FamaFrench python attrs: market_excess_return, smb, hml, rmw, cma, momentum, risk_free_rate, is_estimate,
  value(=hml), time, end_time. DataFrame cols: cma,hml,isestimate,marketexcessreturn,momentum,riskfreerate,rmw,smb,value.
- obj.time = SESSION date, end_time = session + 1 calendar day (DataFrame index = end_time, so shift -1 day).
  Use history[FamaFrench] objects and key by o.time.date(). Daily RF ~ 0.0002.
- f.security_reference.exchange_id in {'NYS','NAS','ASE'} for NYSE/NASDAQ/AMEX (2012: 1169/1542/126).
- Cash-flow signs: common_stock_dividend_paid NEGATIVE, common_stock_payments NEGATIVE, common_stock_issuance POSITIVE
  -> NOP = (|div| + |payments| - issuance) / ME.
- Missing MultiPeriodField values are NaN (not 0); .has_value available. preferred_stock: 1065 NaN / 1629 zero / 607 pos
  over 3301 firms -> NaN preferred stock treated as 0 (also deferred taxes, long_term_debt, inventory, payout items).
- f.market_cap in dollars (AAPL 2012 ~5.4e11); f.adjusted_price split+dividend adjusted; asset_classification.sic int.
- period_ending_date.twelve_months / file_date.twelve_months give fiscal-year end and filing date of the annual values.
- qb.universe_history(universe, start, end) -> Series of Fundamental collections (one per day).

Morningstar field paths verified from skills (equity-fundamental-data, fundamental-balance-sheet).
All MultiPeriodField -> append .twelve_months (annual / TTM) per data-fidelity rule.

Top level:
  f.market_cap                      (price x shares outstanding, current)
  f.adjusted_price, f.price, f.dollar_volume, f.volume
  f.financial_statements.period_ending_date.twelve_months, .file_date.twelve_months
  f.security_reference.security_type == "ST00000001", f.security_reference.is_primary_share
  f.asset_classification.morningstar_sector_code  (compare MorningstarSectorCode.UTILITIES / FINANCIAL_SERVICES)

Balance sheet (f.financial_statements.balance_sheet.*):
  total_assets                      -> Compustat at
  stockholders_equity               -> seq
  preferred_stock                   -> pstk (par value)
  preferred_securities_outside_stock_equity (pref. treated as liability)
  non_current_deferred_taxes_liabilities -> txdb / deferred taxes (balance sheet)
  current_deferred_taxes_liabilities
  long_term_debt                    -> dltt
  long_term_debt_and_capital_lease_obligation
  current_debt                      -> dlc (approx)
  current_debt_and_capital_lease_obligation
  inventory                         -> invt
  net_ppe                           -> ppent
  gross_ppe                         -> ppegt
  common_stock                      (par value of common)
  retained_earnings, additional_paid_in_capital, treasury_stock
  cash_and_cash_equivalents         -> che
  current_assets, current_liabilities
  minority_interest
  goodwill_and_other_intangible_assets

Income statement (f.financial_statements.income_statement.*):
  total_revenue                     -> sale
  cost_of_revenue                   -> cogs
  gross_profit                      -> gp  (= revenue - cost of revenue)
  operating_income                  -> oiadp (income after cost of revenue and opex)
  net_income                        -> ni
  net_income_continuous_operations  -> ib  (before extraordinary / discontinued)
  net_income_common_stockholders    (ni - preferred dividends)
  selling_general_and_administration -> xsga
  research_and_development          -> xrd
  depreciation_and_amortization     -> dp
  interest_expense
  pretax_income, tax_provision, ebit, ebitda
  preferred_stock_dividends

Cash flow statement (f.financial_statements.cash_flow_statement.*):
  common_stock_dividend_paid        -> dvc (cash outflow, sign to check)
  cash_dividends_paid               -> dv (common + preferred)
  common_stock_payments             -> prstkc (repurchase outflow)
  common_stock_issuance             -> sstk (issuance inflow)
  net_common_stock_issuance
  capital_expenditure               -> capx
  operating_cash_flow, depreciation_and_amortization
"""

# region imports
from AlgorithmImports import *
from collections import deque
import numpy as np
from scipy.stats import rankdata
import torch
import nets
from dgp import ModelParams, ModelSolution, SimulatedPanel, build_features, moment_score
# endregion


class PanelMonth:
    """One monthly cross-section: predictors formed at the start of month m, realised return over m."""

    def __init__(self, m: int, symbols: List[Symbol], x: np.ndarray, lme: np.ndarray) -> None:
        self.m = m
        self.symbols = symbols
        self.x = x
        self.lme = lme
        self.y = None
        self.raw = None


class FirmState:
    """Per-firm accumulated state fed by the daily fundamental universe."""

    __slots__ = ("symbol", "daily", "month_px", "lme", "dec_me", "annual", "annual_prev", "last_seen")

    def __init__(self, symbol: Symbol) -> None:
        self.symbol = symbol
        self.daily = deque(maxlen=270)      # (session_idx, adjusted close)
        self.month_px = {}                  # month_idx -> adjusted close at end of month_idx - 1
        self.lme = None                     # market cap at end of previous month
        self.dec_me = {}                    # year -> market cap at end of December of that year
        self.annual = None                  # dict of annual items (snapshot taken in July)
        self.annual_prev = None             # previous July snapshot
        self.last_seen = -1


class TransferLearningStockReturns(QCAlgorithm):
    """Bai (2024) 'Teach Machine Asset Pricing' - transfer learning (TL2) long-short deciles."""

    def initialize(self) -> None:
        # sample split (directive E): train 2006-2015, validate 2016-2020; trading starts at the start date.
        # The real panel (Morningstar coverage from Dec-2002), the base model and the fine-tuned model are
        # built during warm-up, which replays the daily fundamental universe from Dec-2002 to the start date.
        self.set_start_date(2021, 9, 1)
        self.set_end_date(2026, 9, 1)
        self.set_warm_up(self.start_date - datetime(2002, 12, 1))
        self._train_start, self._train_end = 2006 * 12 + 1, 2015 * 12 + 12
        self._val_start, self._val_end = 2016 * 12 + 1, 2020 * 12 + 12
        self.set_cash(100_000_000)
        self.settings.min_absolute_portfolio_target_percentage = 0
        self.settings.minimum_order_margin_portfolio_percentage = 0
        self.settings.seed_initial_prices = True

        self._ff = self.add_data(FamaFrench, "FF", Resolution.DAILY).symbol
        spy = Symbol.create("SPY", SecurityType.EQUITY, Market.USA)
        self._hours = self.market_hours_database.get_exchange_hours(Market.USA, spy, SecurityType.EQUITY)

        self.universe_settings.resolution = Resolution.DAILY
        self._universe = self.add_universe(self._select)

        self._firms = {}                    # Symbol -> FirmState
        self._sessions = []                 # list of session dates (prior close of each selection day)
        self._session_idx = {}              # date -> index into _sessions
        self._weight_by_symbol = {}
        self._panel = {}                    # month index -> PanelMonth
        self._tuned = None

        # structural model -> simulated panel -> base network (trained one epoch per session)
        self._base_epochs = 5
        self._base_epochs_done = 0
        self._build_base_data()

        self.schedule.on(self.date_rules.month_start("SPY"), self.time_rules.at(8, 0), self._rebalance)

    # ------------------------------------------------------------------ base model (simulated data)
    def _build_base_data(self) -> None:
        """Solve the model, simulate candidate panels, keep the one closest to the Table 2 data
        moments (directive C), build its 27 predictors and create the base network."""
        torch.set_num_threads(4)
        sol = ModelSolution(ModelParams())
        sol.solve()
        best = None
        for seed in range(10):
            sim = SimulatedPanel(sol, 5000, 1400, seed)
            sc = moment_score(sim.moments(500))
            if best is None or sc < best[0]:
                best = (sc, sim)
        self._sim_x, self._sim_y, _ = build_features(best[1], 500)
        torch.manual_seed(0)
        self._model = nets.SwishNet(27, 2)
        self._base_opt = torch.optim.Adam(self._model.parameters(), lr=1e-3)

    def _train_base_epoch(self) -> None:
        nets.train(self._model, self._sim_x, self._sim_y, lr=1e-3, batch_size=256, max_epochs=1,
                   seed=self._base_epochs_done, opt=self._base_opt)
        self._base_epochs_done += 1
        if self._base_epochs_done == self._base_epochs:
            self._sim_x = None
            self._sim_y = None
            self._base_opt = None

    # ------------------------------------------------------------------ universe
    def _eligible(self, f: Fundamental) -> bool:
        sr = f.security_reference
        if sr.security_type != "ST00000001" or not sr.is_primary_share:
            return False
        if sr.exchange_id not in ("NYS", "NAS", "ASE"):
            return False
        sic = f.asset_classification.sic
        if 4900 <= sic <= 4999 or 6000 <= sic <= 6999:
            return False
        return True

    def _select(self, fundamentals: List[Fundamental]) -> List[Symbol]:
        today = self.time.date()
        session = None
        eligible = [f for f in fundamentals if f.adjusted_price > 0 and self._eligible(f)]
        if not eligible:
            return Universe.UNCHANGED
        # the fundamental row carries the previous session's close; key it by that session date
        session = self._previous_session(today)
        if session in self._session_idx:
            return Universe.UNCHANGED
        s_idx = len(self._sessions)
        self._sessions.append(session)
        self._session_idx[session] = s_idx
        if self._base_epochs_done < self._base_epochs:
            self._train_base_epoch()

        month_end = self._next_session(session).month != session.month
        m_idx = session.year * 12 + session.month      # index of the month that just ended

        for f in eligible:
            st = self._firms.get(f.symbol)
            if st is None:
                st = FirmState(f.symbol)
                self._firms[f.symbol] = st
            st.daily.append((s_idx, f.adjusted_price))
            st.last_seen = s_idx
            if month_end:
                st.month_px[m_idx] = f.adjusted_price
                st.lme = f.market_cap
                if session.month == 12:
                    st.dec_me[session.year] = f.market_cap
                elif session.month == 6:
                    st.annual_prev = st.annual
                    st.annual = self._annual_snapshot(f, st.dec_me.get(session.year - 1))

        if not month_end:
            return Universe.UNCHANGED

        # prune firms that left the feed long ago and stale month-end prices
        for sym in [s for s, st in self._firms.items() if s_idx - st.last_seen > 300]:
            del self._firms[sym]
        for st in self._firms.values():
            for k in [k for k in st.month_px if k < m_idx - 40]:
                del st.month_px[k]
            for k in [k for k in st.dec_me if k < session.year - 2]:
                del st.dec_me[k]

        # realised targets for the row formed at the start of month m_idx
        prev = self._panel.get(m_idx)
        if prev is not None:
            y = np.full(len(prev.symbols), np.nan, dtype=np.float32)
            for i, sym in enumerate(prev.symbols):
                st = self._firms.get(sym)
                if st is not None and m_idx in st.month_px and (m_idx - 1) in st.month_px:
                    y[i] = st.month_px[m_idx] / st.month_px[m_idx - 1] - 1.0
            prev.y = y

        # features for the coming month M = m_idx + 1
        row = self._build_month(eligible, session, s_idx, m_idx)
        if row is not None:
            self._panel[m_idx + 1] = row

        # fine-tune once the validation window has its last realised return (end of val_end month)
        if self._tuned is None and m_idx >= self._val_end:
            self._fine_tune()
        if self._tuned is None or row is None:
            return Universe.UNCHANGED

        # predict, sort into deciles, value-weight long top / short bottom (Table 5)
        pred = nets.predict(self._model, row.x)
        n = len(pred)
        order = np.argsort(pred)
        decile = np.empty(n, dtype=int)
        decile[order] = (np.arange(n) * 10) // n
        top = decile == 9
        bottom = decile == 0
        weights = {}
        for mask, sign in ((top, 0.5), (bottom, -0.5)):
            tot = float(row.lme[mask].sum())
            for i in np.flatnonzero(mask):
                weights[row.symbols[i]] = sign * float(row.lme[i]) / tot
        self._weight_by_symbol = weights
        if self.is_warming_up and self._next_session(session) < self.start_date.date():
            # warm-up month: keep the state, subscribe nothing (only the final pre-start selection is traded)
            return []
        return list(weights.keys())

    # ------------------------------------------------------------------ transfer learning
    def _fine_tune(self) -> None:
        """Fine-tune the base network (all layers, lr 1e-4, batch 256, patience 3) on the real panel:
        training months [_train_start, _train_end], validation months [_val_start, _val_end]."""
        def stack(lo: int, hi: int) -> tuple:
            xs, ys = [], []
            for m in range(lo, hi + 1):
                pm = self._panel.get(m)
                if pm is None or pm.y is None:
                    continue
                ok = np.isfinite(pm.y)
                xs.append(pm.x[ok])
                ys.append(pm.y[ok])
            if not xs:
                return np.empty((0, 27), dtype=np.float32), np.empty(0, dtype=np.float32)
            return np.concatenate(xs), np.concatenate(ys)

        x_tr, y_tr = stack(self._train_start, self._train_end)
        x_va, y_va = stack(self._val_start, self._val_end)
        nets.train(self._model, x_tr, y_tr, lr=1e-4, batch_size=256, max_epochs=100, seed=0,
                   x_val=x_va, y_val=y_va, patience=3)
        self._tuned = True
        for m in [m for m in self._panel if m <= self._val_end]:
            del self._panel[m]

    # ------------------------------------------------------------------ characteristics
    def _ff_arrays(self, s_idx: int) -> tuple:
        """Daily FF factors aligned to session index (NaN where missing) over the last 270 sessions."""
        n = 270
        mkt = np.full(n, np.nan)
        smb = np.full(n, np.nan)
        hml = np.full(n, np.nan)
        rf = np.full(n, np.nan)
        base = s_idx - n + 1
        for o in self.history[FamaFrench](self._ff, timedelta(days=420), Resolution.DAILY):
            j = self._session_idx.get(o.time.date())
            if j is None or j < base:
                continue
            if o.market_excess_return is None or o.risk_free_rate is None or o.smb is None or o.hml is None:
                continue
            k = j - base
            mkt[k] = float(o.market_excess_return)
            smb[k] = float(o.smb)
            hml[k] = float(o.hml)
            rf[k] = float(o.risk_free_rate)
        return mkt, smb, hml, rf

    def _build_month(self, eligible: List[Fundamental], session: datetime.date, s_idx: int, m_idx: int) -> Optional[PanelMonth]:
        """Cross-section of the 27 predictors formed at the end of month m_idx (for month m_idx + 1)."""
        n_days = 270
        base = s_idx - n_days + 1
        if base < 0:
            return None
        mkt, smb, hml, rf = self._ff_arrays(s_idx)
        # session -> month index, month-level macro series
        sess_month = np.array([d.year * 12 + d.month for d in self._sessions[base:s_idx + 1]])
        in_month = sess_month == m_idx
        in_prev = sess_month == m_idx - 1
        if in_month.sum() < 15 or in_prev.sum() < 15 or np.isnan(mkt[in_month]).any() or np.isnan(mkt[in_prev]).any():
            return None
        mkt_m = np.prod(1 + mkt[in_month] + rf[in_month]) - np.prod(1 + rf[in_month])
        mkt_p = np.prod(1 + mkt[in_prev] + rf[in_prev]) - np.prod(1 + rf[in_prev])
        rf_m = np.prod(1 + rf[in_month]) - 1
        rf_p = np.prod(1 + rf[in_prev]) - 1
        d_mkt = mkt_m - mkt_p
        d_rf = rf_m - rf_p
        beta_days = slice(n_days - 252, n_days)
        x_beta = mkt[beta_days]
        x_ivol = np.column_stack([np.ones(in_month.sum()), mkt[in_month], smb[in_month], hml[in_month]])

        symbols, rows, lmes = [], [], []
        for f in eligible:
            st = self._firms[f.symbol]
            a, p = st.annual, st.annual_prev
            if a is None or p is None or a["dec_me"] is None or a["pe"] == p["pe"]:
                continue
            need = [st.month_px.get(m_idx - k) for k in range(36)]
            if any(v is None for v in need):
                continue
            # daily return series on the session grid (NaN where absent)
            px = np.full(n_days, np.nan)
            for j, v in st.daily:
                if j >= base:
                    px[j - base] = v
            ret = px[1:] / px[:-1] - 1.0
            ret = np.concatenate([[np.nan], ret])
            r_m = ret[in_month]
            if np.isnan(r_m).sum() > in_month.sum() - 15:
                continue
            ok_m = ~np.isnan(r_m)
            if ok_m.sum() < 15:
                continue
            ex_beta = ret[beta_days] - rf[beta_days]
            ok_b = ~np.isnan(ex_beta) & ~np.isnan(x_beta)
            if ok_b.sum() < 200:
                continue
            xb = x_beta[ok_b]
            yb = ex_beta[ok_b]
            beta = np.cov(xb, yb, bias=True)[0, 1] / np.var(xb)
            xm = x_ivol[ok_m]
            ym = r_m[ok_m] - rf[in_month][ok_m]
            coef, _, _, _ = np.linalg.lstsq(xm, ym, rcond=None)
            ivol = np.std(ym - xm @ coef)
            tvol = np.std(r_m[ok_m])

            chars = self._characteristics(a, p, st.lme, need)
            if chars is None:
                continue
            chars[17] = beta
            chars[18] = ivol
            chars[19] = tvol
            symbols.append(f.symbol)
            rows.append(chars)
            lmes.append(st.lme)

        if len(symbols) < 50:
            return None
        raw = np.array(rows, dtype=np.float64)
        x = np.empty((len(symbols), 27), dtype=np.float32)
        n = len(symbols)
        for c in range(25):
            ranks = rankdata(raw[:, c], method="average")
            x[:, c] = (ranks - 1.0) / (n - 1.0) * 2.0 - 1.0
        x[:, 25] = d_mkt
        x[:, 26] = d_rf
        row = PanelMonth(m_idx + 1, symbols, x, np.array(lmes, dtype=np.float64))
        row.raw = raw
        return row

    @staticmethod
    def _characteristics(a: Dict[str, Any], p: Dict[str, Any], lme: float, px: List[float]) -> Optional[Any]:
        """Items 1-17 and 21-25 of Table 3 from the annual snapshots and month-end prices. None if incomplete."""
        def z(v: float) -> float:
            return 0.0 if v != v else v

        def be(s: Dict[str, Any]) -> float:
            ps = z(s["ps"])
            sh = s["seq"]
            if sh != sh:
                sh = s["ceq"] + ps if s["ceq"] == s["ceq"] else s["at"] - s["lt"]
            return sh + z(s["txditc"]) - ps

        core = ("at", "sale", "cogs", "ib", "gp", "oiadp", "ppegt")
        for s in (a, p):
            if any(s[k] != s[k] for k in core) or (s["seq"] != s["seq"] and s["ceq"] != s["ceq"]
                                                   and (s["lt"] != s["lt"])):
                return None
        dec_me = float(a["dec_me"])
        be_a, be_p = be(a), be(p)
        if dec_me <= 0 or a["ppegt"] <= 0 or a["sale"] <= 0 or p["sale"] <= 0 or a["at"] <= 0 or p["at"] <= 0 \
                or be_a <= 0 or be_p <= 0 or lme is None or lme <= 0:
            return None
        nop = abs(z(a["dvc"])) + abs(z(a["prstkc"])) - z(a["sstk"])
        q = (dec_me + z(a["dltt"]) + z(a["ps"]) - z(a["invt"]) - z(a["txditc"])) / a["ppegt"]
        c = np.empty(27)
        c[0] = a["at"] / dec_me                      # A2ME
        c[1] = be_a / dec_me                         # BEME
        c[2] = a["ib"] / dec_me                      # E2P
        c[3] = nop / dec_me                          # NOP
        c[4] = q                                     # Q
        c[5] = a["sale"] / p["sale"] - 1.0           # Sales_g
        c[6] = a["at"] / p["at"] - 1.0               # Investment
        c[7] = be_a / be_p - 1.0                     # dCEQ
        c[8] = a["sale"] / p["at"]                   # CTO
        c[9] = (a["sale"] - a["cogs"]) / a["sale"]   # PCM
        c[10] = a["oiadp"] / a["sale"]               # PM
        c[11] = a["gp"] / be_a                       # Prof
        c[12] = a["ib"] / p["at"]                    # ROA
        c[13] = a["ib"] / be_p                       # ROE
        c[14] = a["sale"] / a["at"]                  # SAT
        c[15] = a["at"]                              # AT
        c[16] = lme                                  # LME
        c[17] = c[18] = c[19] = np.nan               # beta, ivol, total vol (filled by caller)
        # px[k] = month-end price k months before the formation date; r_{a_b} = P_{-b} / P_{-a} - 1
        c[20] = px[0] / px[1] - 1.0                  # r2_1
        c[21] = px[1] / px[5] - 1.0                  # r6_2
        c[22] = px[1] / px[11] - 1.0                 # r12_2
        c[23] = px[6] / px[11] - 1.0                 # r12_7
        c[24] = px[12] / px[35] - 1.0                # r36_13
        c[25] = c[26] = 0.0
        if not np.all(np.isfinite(c[:17])) or not np.all(np.isfinite(c[20:25])):
            return None
        return c

    def _annual_snapshot(self, f: Fundamental, dec_me: float) -> Dict[str, Any]:
        """Latest annual (.twelve_months) statement items, Compustat names as keys. NaN = missing."""
        fs = f.financial_statements
        bs, inc, cf = fs.balance_sheet, fs.income_statement, fs.cash_flow_statement
        snap = {
            "at": bs.total_assets.twelve_months,
            "lt": bs.total_liabilities_net_minority_interest.twelve_months,
            "seq": bs.stockholders_equity.twelve_months,
            "ceq": bs.common_stock_equity.twelve_months,
            "ps": bs.preferred_stock.twelve_months,
            "txditc": bs.non_current_deferred_taxes_liabilities.twelve_months,
            "dltt": bs.long_term_debt.twelve_months,
            "invt": bs.inventory.twelve_months,
            "ppegt": bs.gross_ppe.twelve_months,
            "sale": inc.total_revenue.twelve_months,
            "cogs": inc.cost_of_revenue.twelve_months,
            "gp": inc.gross_profit.twelve_months,
            "oiadp": inc.operating_income.twelve_months,
            "ib": inc.net_income_continuous_operations.twelve_months,
            "dvc": cf.common_stock_dividend_paid.twelve_months,
            "prstkc": cf.common_stock_payments.twelve_months,
            "sstk": cf.common_stock_issuance.twelve_months,
            "pe": fs.period_ending_date.twelve_months,
            "dec_me": dec_me,
        }
        return snap

    def _previous_session(self, today: datetime.date) -> datetime.date:
        d = today - timedelta(days=1)
        while not self._hours.is_date_open(d):
            d -= timedelta(days=1)
        return d

    def _next_session(self, session: datetime.date) -> datetime.date:
        d = session + timedelta(days=1)
        while not self._hours.is_date_open(d):
            d += timedelta(days=1)
        return d

    # ------------------------------------------------------------------ trading
    def _rebalance(self) -> None:
        if self.is_warming_up or not self._weight_by_symbol:
            return
        targets = []
        for s, w in self._weight_by_symbol.items():
            sec = self.securities.get(s)
            if sec is None or sec.price <= 0:
                # cannot be priced this morning (no bar yet): skip rather than send an unpriceable order
                continue
            targets.append(PortfolioTarget(s, w))
        # flatten last month's names that left the book; a holding whose data feed has ended (price 0,
        # e.g. an acquisition without a delisting record) cannot be priced and is left as is
        for kvp in self.portfolio:
            if kvp.value.invested and kvp.key not in self._weight_by_symbol and self.securities[kvp.key].price > 0:
                targets.append(PortfolioTarget(kvp.key, 0))
        self.set_holdings(targets)

"""Feed-forward network with trainable-gamma Swish activations (Bai 2024, Section 3.2)."""
import numpy as np
import torch
import torch.nn as nn


class Swish(nn.Module):
    """sigma(x) = x / (1 + exp(-gamma x)) with trainable gamma (initialised at 1)."""

    def __init__(self) -> None:
        super().__init__()
        self.gamma = nn.Parameter(torch.ones(1))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return x * torch.sigmoid(self.gamma * x)


class SwishNet(nn.Module):
    """n_inputs -> [n_inputs] * n_hidden -> 1 (hidden width equals input width)."""

    def __init__(self, n_inputs: int, n_hidden: int) -> None:
        super().__init__()
        layers = []
        for _ in range(n_hidden):
            layers.append(nn.Linear(n_inputs, n_inputs))
            layers.append(Swish())
        layers.append(nn.Linear(n_inputs, 1))
        self.net = nn.Sequential(*layers)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x).squeeze(-1)


def predict(model: nn.Module, x: np.ndarray) -> np.ndarray:
    model.eval()
    with torch.no_grad():
        return model(torch.from_numpy(np.ascontiguousarray(x, dtype=np.float32))).numpy()


def mse(model: nn.Module, x: np.ndarray, y: np.ndarray) -> float:
    pred = predict(model, x)
    return float(np.mean((pred - y) ** 2))


def train(model: nn.Module, x: np.ndarray, y: np.ndarray, lr: float, batch_size: int, max_epochs: int,
          seed: int, x_val: np.ndarray = None, y_val: np.ndarray = None, patience: int = 3,
          opt: torch.optim.Optimizer = None) -> dict:
    """ADAM + MSE mini-batch training. With validation data: early stopping (patience epochs),
    restoring the best-validation weights. Without: fixed number of epochs. Pass opt to keep the
    ADAM state across successive calls."""
    torch.manual_seed(seed)
    gen = torch.Generator().manual_seed(seed)
    xt = torch.from_numpy(np.ascontiguousarray(x, dtype=np.float32))
    yt = torch.from_numpy(np.ascontiguousarray(y, dtype=np.float32))
    if opt is None:
        opt = torch.optim.Adam(model.parameters(), lr=lr)
    loss_fn = nn.MSELoss()
    n = xt.shape[0]
    best_val = np.inf
    best_state = None
    bad = 0
    history = []
    epochs_run = 0
    for epoch in range(max_epochs):
        model.train()
        perm = torch.randperm(n, generator=gen)
        total = 0.0
        for start in range(0, n, batch_size):
            idx = perm[start:start + batch_size]
            opt.zero_grad()
            loss = loss_fn(model(xt[idx]), yt[idx])
            loss.backward()
            opt.step()
            total += loss.item() * len(idx)
        epochs_run = epoch + 1
        train_loss = total / n
        if x_val is None:
            history.append((train_loss, None))
            continue
        val_loss = mse(model, x_val, y_val)
        history.append((train_loss, val_loss))
        if val_loss < best_val - 1e-12:
            best_val = val_loss
            best_state = {k: v.detach().clone() for k, v in model.state_dict().items()}
            bad = 0
        else:
            bad += 1
            if bad >= patience:
                break
    if best_state is not None:
        model.load_state_dict(best_state)
    return {"epochs": epochs_run, "history": history, "best_val": best_val}

NOTES = """
PAPER: "Teach Machine Asset Pricing", Chengyu Bai (SAIF), SSRN 4598745, draft March 26 2024. 33 pages.

p1-3 (abstract, intro):
- Framework: transfer learning. Feed-forward NNs first trained on SIMULATED data from a production-based
  model (simplified Zhang 2005), then further trained on real US stock data. "TL model" = simulated + real;
  "DD model" = real data only; same architecture and hyper-parameters.
- Inputs: firm characteristics and macroeconomic time series. Target: next month's stock return.
- Evaluation: out-of-sample R^2 monthly. Best TL R2 0.9%/month vs DD 0.81%.
- Portfolio: sort stocks into DECILE portfolios by predicted return; VALUE-WEIGHTED portfolio returns; top decile
  (highest predicted) minus bottom decile spread. Monthly.
- Robustness: NNs trained purely on simulated data also predict real returns; pooling simulated+real without TL is worse.

p4-6 (related lit, Section 2 production-based model):
- Closest paper: Chen et al. (2023) transfer learning framework (option pricing w/ Black-Scholes). Gu, Kelly, Xiu (2020) is the
  benchmark ML-in-asset-pricing work (evaluation measures, benchmarks). Paper structure: Sec 2 model, Sec 3 NN/TL, Sec 4 results, Sec 5 attribution.
- Section 2.1 model (Zhang 2005 simplified). Production: y_jt = exp(x_t + z_jt) k_jt^alpha, 0<alpha<1 (eq 1).
- Aggregate productivity AR(1): x_{t+1} = (1-rho_x) xbar + rho_x x_t + sigma_x eps^x (eq 2), eps iid normal.
- Idiosyncratic productivity: z_{j,t+1} = rho_z z_jt + sigma_z eps^z (eq 3), iid normal, independent of eps^x and across firms.

p7-9 (model cont., calibration, cross-sectional moments):
- Capital: k_{t+1} = i_t + (1-delta) k_t (eq 4).
- Adjustment cost h = 1{i!=0} C_f y + C_r |i| 1{i<0} + (C_q/2) (i/k)^2 k (eq 5).
- SDF: log M_{t+1} = log beta + gamma_t (x_t - x_{t+1}); gamma_t = gamma_0 + gamma_1 (x_t - xbar) (eq 6); beta in (0,1), gamma0>0, gamma1<0.
- Dividend d = y - i - h - f (eq 7), f fixed production cost. Bellman v(k;x,z) = max_i {d + E[M v']} (eq 8). Gross return R_{t+1} = v_{t+1}/(v_t - d_t); E[M R]=1 (eq 9).
- Calibration (monthly, Table 1): alpha=0.3, delta=0.01 (Zhang 2005); C_f=0 (Bloom 2009); C_r=0.15; C_q=4 (Cooper-Haltiwanger 2006);
  f=0.023 (Belo et al 2014, match value premium). rho_x = 0.95^(1/3); sigma_x = 0.7/sqrt(1+rho_x^2+rho_x^4) [note: quarterly sigma 0.7 -- likely 0.007 in Zhang units, check Table 1];
  xbar = -3.15 (avg firm size ~1). rho_z, sigma_z same as Zhang (2005) (Zhang: rho_z=0.97, sigma_z=0.10 monthly). Pricing kernel from Belo et al 2014: beta=0.999, gamma0=18, gamma1=-750.
- Solved by value function iteration (Appendix). Simulation for moments: 100 panels x 5000 firms x 1400 months, drop first 500.
- Table 2: model equity premium 10.6% vs data 8.7%; sd 22.2% vs 15.7%; rf 2.8% vs 3%; rf vol 1.4%. (Data from K. French website.)

p10-12 (Section 3 Empirical: data, training):
- Model moments: market SR 48.5%, indiv stock vol 31%; value premium (top-bottom BM decile) 5.5% vs 5.2% data; invest rate 12.8%, disinvest fraction 11.5%.
- 3.1 DATA: CRSP returns + Compustat Fundamental ANNUAL. Sample 1972-2022. NYSE/AMEX/NASDAQ stocks. EXCLUDE utilities (SIC 4900-4999)
  and financials (SIC 6000-6999). Avg > 1500 stocks per year. Only keep stocks whose characteristics are COMPLETE (no imputation, following Chen et al 2023).
- Predictors: 25 firm-level characteristics from Freyberger, Neuhierl, Weber (2020) [list in Table 3]; 2 macro variables: market excess return and
  risk-free rate (French website), transformed by FIRST-ORDER DIFFERENCE (Chen et al 2023). Only characteristics the production model can generate
  (no intangibles, no trading volume).
- Annual variables updated at END OF JUNE (Fama-French convention). Monthly variables (e.g., cumulative returns) updated end of each month.
- Characteristics normalized each period to [-1,1] by cross-sectional RANK (Gu et al 2020).
- Simulated data: panel 5000 firms x 900 months; pick panel close to Table 2 moments; simulated firms sorted into portfolios to build risk factors
  for beta and idio vol; simulated data aggregated annually to match real availability. Table 3 lists predictors and construction.
- 3.2 TRAINING: Base model = trained on simulated only; TL = base then fine-tuned on real; DD = real only. Feed-forward NN, up to 5 hidden layers
  (NN1..NN5), each hidden layer has same number of neurons as input layer (27 inputs -> 27 neurons). Activation Swish sigma(x)= x/(1+exp(-gamma x)),
  gamma trainable. ADAM, lr 0.001 for base & DD, lr 0.0001 for TL. Loss MSE. Mini-batch 256. Early stopping: patience 3 epochs on validation.
  Base model trained on simulated data for a small number of epochs. Fine-tuning method (all layers trainable), footnote 1.
- SAMPLE SPLIT: first 15 years training, 10 years validation, last 25 years out-of-sample test (1972-1986 train, 1987-1996 val, 1997-2022 test?). [Check whether
  the split is fixed or rolling - continue reading.]

p13-15 (3.3 OOS performance, Section 4 attribution):
- R2_oos = 1 - sum (r - rhat)^2 / sum r^2 over test set (eq 11; GKX2020 definition, no demeaning). Best TL R2 0.9%/month (Table 4);
  TL beats DD for 1-4 layers; DD5 beats TL5. Subsamples by market cap: TL better for large caps.
- PORTFOLIO: sort stocks into DECILE portfolios by predicted return (monthly). Monthly rebalanced, VALUE-WEIGHTED returns.
  Long-short = long top decile (highest predicted), short bottom decile. Table 5. Best TL model = 2 HIDDEN LAYERS (top decile VW 2.56%/mo).
  Best DD = 4 layers. Paper FOCUSES on the 2-layer TL model (Figure 1 = cumulative VW returns of top decile for TL2, DD2, momentum 12-2).
  => MAIN RESULT: TL model with 2 hidden layers, decile sort, value-weighted, long-short top minus bottom (Table 5); Fig 1 shows long-only top decile.
- Momentum benchmark: cumulative return months t-12 to t-2 (Fama-French 1996).
- 4.1 Base model: base models trained until validation loss stops improving (different from main; main trains base for few epochs).
  Base-model L/S returns nonnegative; annual top/bottom spread 6% for 1 and 5 layers. Equal-weighted shows wider spread (Table 6).

p16-18 (4.2-4.4, conclusion):
- 4.2 Pooled sim+real training (no TL) is worse than DD -> improvement not due to sample size.
- 4.3 Variable importance (GKX: set variable to zero, drop in in-sample R2). Focus on 2-layer nets. Top 3 for both TL and DD: LME (mkt cap),
  dMKT (change in market excess return), r2-1 (short-term reversal). Disagreements: AT, CTO, long-term reversal, ROE, capm_beta, A2ME.
  Variable names visible: LME, dMKT, r2_1, AT, CTO, ROE, capm_beta, A2ME, Sales_g, Prof, SAT.
- 4.4 Univariate sorts on characteristics (Table 7), equally weighted per FNW2020, L/S decile 10 minus decile 1, t>3.
- Conclusion: repeats; main portfolio result = decile sort on TL predictions, top-bottom spread; top decile persistently higher returns.

p19-21 (Appendix numerical method, references start):
- VFI on grid: capital grid k_i = k_{i-1} + c1 exp(c2 (i-2)), k1=0.01, c1=0.005, c2=0.1247952, N=50, kN=15. Finer grid K' for optimal k', linear interp.
- x and z discretized by Rouwenhorst (1995), 5 points each. Bellman (A1): v(k,x,z)=max_k' {e^{x+z} k^alpha - i - h - f + sum M(x'|x) v(k',x',z') Qz Qx},
  M(x'|x) = beta exp((gamma0+gamma1(x-xbar))(x-x')). Convergence max |v^n - v^{n-1}| < 1e-5.
- Simulation: random initial x, z, k for 5000 firms; truncate x,z to grid range; k' from policy, v from 3-D interp; iid normal shocks; 1400 periods, discard 500 -> 900 months.
  Predictors built per Table 3 from simulated variables. One panel with moments close to reality suffices.
- References include Ang et al 2006 (idio vol), Balakrishnan 2010, Ball et al 2015, Rosenberg-Reid-Lanstein.

p22-24 (references): Chen, Cheng, Liu, Tang (2023) "Teaching Economics to the Machines" SSRN 4642167 (TL framework, first-difference macro,
  complete-case rule). Freyberger-Neuhierl-Weber (2020) RFS 33(5) (characteristic definitions). Gu-Kelly-Xiu (2020) RFS. Cooper-Gulen-Schill 2008 (asset growth),
  Bhandari 1988 (leverage), Basu 1977 (E/P), Boudoukh et al 2007 (payout), Lakonishok et al 1994, Jegadeesh 1990, Jegadeesh-Titman 1993, De Bondt-Thaler 1985,
  Haugen-Baker 1996, Lewellen-Nagel 2006, Kaplan-Zingales 1997, Gorodnichenko-Weber 2016, D'Acunto et al 2018, Gandhi-Lustig 2015. No source code link found so far.

p25-27 (refs end, Table 1, Table 2):
- Table 1 (confirmed): alpha 0.3; delta 0.01; C_q 4.0; C_r 0.15; C_f 0; f 0.023; rho_x 0.95^(1/3); xbar -3.15; sigma_x = 0.007/sqrt(1+rho_x^2+rho_x^4)
  (table renders "0 7/ sqrt(1+rho_x^2+rho_x^4)" - Zhang 2005 uses quarterly 0.007; treat as 0.007); rho_z 0.97; sigma_z 0.1; beta 0.999; gamma0 18; gamma1 -750.
- Table 2 note: French data 1985-2022; 100 panels x 5000 firms x 600 months for moments. Data: EP 8.7, sd 15.7, rf 3.0, rf sd 0.7, SR 43, ind vol 25-32, value prem 5.2,
  inv rate 12.2, neg inv 10.4, inaction 8.1, pos spike 18.6, neg spike 1.8.

p28 (Table 3 "Matching model quantities to reality" - rotated landscape table, OCR garbled; partially recovered):
- Table note: CRSP items shrout, prc. Compustat Fundamentals Annual items: at, txditc, seq, ceq, lt, pstkrv, pstkl, pstk, ib, dvc, prstkc, sstk, dltt,
  invt, txdb, ppent, sale, cogs, oiadp, gp. Market factor and rf from French website.
- Recovered model-quantity <-> characteristic pairs (fragments): k_t/v_t with Bhandari (1988) => Lev (leverage: (dltt+dlc? / ME) -- FNW: Lev = (dltt+dlc)/(dltt+dlc+seq));
  k_t/v_t with Rosenberg-Reid-Lanstein / Davis et al (2000) => BEME; (y_t - f)/v_t with Basu(1977) => E2P; (dvc + prstkc - sstk)/ME with Boudoukh et al (2007) => NOP (net payout);
  d_t/v_t ; (txdb...)/ppegt ; v_t/k_t => Q (Tobin's Q); Sales_g with Lakonishok et al (1994); change in k_t with Cooper et al (2008) => Investment (asset growth);
  dCEQ/k_t with Richardson et al (2005) => OA? ; y_t/k_{t-1} with Haugen-Baker (1996) => ATO or SAT (sales/assets); (y_t - f)/y_t => PM (Soliman 2008) ;
  Gorodnichenko-Weber (2016) => FC2Y (fixed costs to sales); D'Acunto et al (2018) => PCM (price-cost margin); Balakrishnan et al (2010) => ROA;
  Ball et al (2015) => Prof / OL; Haugen-Baker (1996) also => ROE/CTO.
- Variable names named in text: LME, dMKT, r2_1, AT, CTO, ROE, capm_beta, A2ME, Sales_g, Prof, SAT; also long-term reversal, idio vol (Ang et al 2006), beta.
- 25 firm characteristics + 2 macro (dMKT = first diff of market excess return, dRF = first diff of risk-free rate) = 27 inputs.
- FULL LIST OF 25 NOT RECOVERABLE FROM OCR -> try web for HTML version; otherwise REQUEST from client.

p29-30 (Table 3 continued - items 16-27 CLEAN; Tables 4, 5):
 Trading frictions: (16) AT total assets = at [model k_t] Gandhi-Lustig 2015; (17) LME market cap = PREVIOUS MONTH shrout x prc [v] FF1992;
 (18) capm_beta estimated by real data (Lewellen-Nagel 2006); (19) ivol idiosyncratic vol (Ang et al 2006); (20) Total_vol total volatility (Ang et al 2006).
 Past returns: (21) r2_1 return 1 month before prediction (Jegadeesh 1990); (22) r6_2 return from 6 to 2 months before (JT1993); (23) r12_2 return 12 to 2 months before (FF1996);
 (24) r12_7 return 12 to 7 months before (Novy-Marx 2012); (25) r36_13 return 36 to 13 months before (De Bondt-Thaler 1985).
 Macro: (26) dMKT change in market excess return (Welch-Goyal 2008); (27) dRf change in Treasury bill rate.
 => items 1-15 (page 28, garbled) are the accounting characteristics: candidates from fragments: A2ME, BEME, E2P, Q, Lev, NOP(payout), Sales_g, Investment, OA, ATO/SAT, CTO, PM, FC2Y, PCM, ROA, ROE, Prof, D2A.
- Table 4 R2oos (%): TL All: 0.90,0.90,0.67,0.79,0.69 (1-5 layers); DD All: 0.71,0.66,0.52,0.62,0.81.
- Table 5 (VW decile monthly %, TL): 2-layer: L 0.61 ... H 2.56, H-L 1.95 (best). 4-layer H-L 1.91. DD best 4-layer H-L 1.85.
  Table 5 caption: "Stocks are sorted into portfolios by their predicted returns. Decile portfolios are formed and rebalanced each month. Value-weighted portfolio returns."

p31-33 (Fig 1, Table 6, Fig 2, Table 7):
- Fig 1: cumulative return of top decile, TL2 vs DD2 vs momentum (r12_2 top decile), monthly rebalanced.
- Table 6: Base models (sim only) and pooled-data DD; VW decile returns. Not the main result.
- Table 7: L/S decile 10-1 EW sorts on characteristics with |t|>3; lists characteristic NAMES: A2ME, BEME, Q, Sales_g, Investment, dCEQ, Prof, SAT, LME, r2_1, r36_13.
  => "dCEQ" is a predictor (change in common equity / ..., Richardson et al 2005). So confirmed accounting predictors include: A2ME, BEME, Q, Sales_g, Investment, dCEQ, Prof, SAT,
  plus (from text/fragments) CTO, ROE, ROA, E2P, Lev, NOP, PM, FC2Y, PCM, OA?, D2A?, AT(16). Need 15 accounting names for items 1-15.
END OF PAPER (33 pages). No source-code link anywhere in the paper.
"""