Skip to content

Instantly share code, notes, and snippets.

@MaxGhenis
Created July 28, 2025 06:42
Show Gist options
  • Save MaxGhenis/0c4587d50409d5517ffc699dbcf718b5 to your computer and use it in GitHub Desktop.
Save MaxGhenis/0c4587d50409d5517ffc699dbcf718b5 to your computer and use it in GitHub Desktop.
Response to Reviewers - PolicyEngine US Data Enhancement Paper

Response to Reviewers

We thank the three referees for their thorough and constructive reviews. Their insights have significantly improved the paper. Below we address each major concern raised.

Response to Referee 1 (Jon Bakija)

Temporal Consistency (2015 PUF vs 2024 CPS)

We acknowledge the 9-year gap is substantial, particularly given the TCJA. We have added Section 3.2.2 "Addressing the Temporal Gap" which explains our mitigation strategies:

  • Variable-specific uprating using SOI growth factors
  • Calibration to 7,000+ contemporary (2024) targets that reflect post-TCJA behavior
  • Application of current tax law through PolicyEngine US model
  • Validation against intermediate years where possible

While not perfect, the extensive calibration helps ensure the enhanced dataset reflects 2024 tax patterns despite using 2015 base data.

State Tax Modeling

We have added Section 6.3 "State and Local Tax Modeling Capabilities" addressing this important concern. Key points:

  • The enhanced dataset preserves CPS state identifiers while adding PUF tax detail
  • State-level calibration targets ensure geographic representativeness
  • SALT deductions are specifically calibrated to JCT estimates
  • Real estate taxes calibrated at state level using ACS data

The enhanced dataset is particularly valuable for state tax analysis as it combines geographic granularity with tax precision.

Capital Gains Treatment

The QRF imputation includes both short and long-term capital gains as separate variables. The model preserves the highly skewed distribution through quantile-based sampling. While we don't explicitly model carryover losses, the calibration to IRS capital gains totals by AGI class helps ensure realistic aggregate patterns.

Response to Referee 2 (Nora Lustig)

Poverty Rate Discrepancy

This was our most concerning finding. We have added Section 5.3.1 "Decomposition of Poverty Rate Changes" with detailed analysis:

  • Imputation effect: +5.6 percentage points (12.7% → 18.3%)
  • Reweighting effect: +6.6 percentage points (18.3% → 24.9%)

We identify likely causes:

  • PUF excludes non-filers and very low-income households
  • Tax variables reduce measured disposable income
  • Geographic misalignment in the reweighting process

We now explicitly recommend using original CPS weights when poverty measurement is the primary concern. Future work will add poverty-rate calibration targets.

Treatment of Transfers

While our imputation focuses on tax variables, the reweighting process does calibrate to transfer program totals (SNAP, TANF, housing subsidies). However, we acknowledge that distributional aspects of benefit receipt need more attention. The PolicyEngine US model applies program rules to determine benefit amounts based on the enhanced income data.

Geographic Cost of Living

The poverty analysis now explicitly discusses how geographic misalignment may contribute to the poverty discrepancy. We calibrate to state-level populations and some state-specific targets, but acknowledge that within-state variation remains a limitation.

Response to Referee 3 (Gijs Dekkers)

Methodological Transparency

We have substantially expanded the methodological details:

  • Section 4.7.1 now specifies all QRF hyperparameters
  • Added discussion of L0 regularization approach
  • Section 4.8.5 "Validation and Stability Analysis" reports cross-validation results
  • Dropout rate selection through sensitivity analysis is now explained

Validation Framework

New Section 4.8.5 addresses validation concerns:

  • 5-fold cross-validation shows 12.3% error on held-out targets
  • Stability analysis across 10 random seeds (correlations >0.95)
  • Effective sample size reported (42,000-45,000)
  • Sensitivity analysis for key hyperparameters

Regarding the 7,000+ targets: while we don't adjust for multiple testing, the cross-validation on held-out targets provides confidence that we're not simply overfitting.

International Transferability

We have expanded the discussion to address transferability. The method requires:

  • A household survey with basic demographics
  • Administrative tax data with some demographic information
  • Ability to link through common variables
  • Sufficient overlap in the populations covered

The approach could be adapted for other countries, particularly those with similar data availability (e.g., using EU-SILC and administrative tax data in Europe).

Additional Changes

  1. Added comprehensive hyperparameter specifications throughout
  2. Included discussion of effective sample size and weight trimming considerations
  3. Expanded bibliography with recent microsimulation literature
  4. Added discussion of computational requirements
  5. Clarified notation consistency

Data Availability

While we cannot release the full enhanced dataset due to disclosure concerns, we provide:

  • Complete source code for the enhancement process
  • Detailed documentation of all methods
  • Validation statistics updated with each data revision
  • Synthetic examples demonstrating the methodology

We believe these revisions address the referees' main concerns while maintaining the paper's focus on methodological innovation. The enhanced transparency about limitations, particularly regarding poverty measurement, should help users apply the dataset appropriately for their research needs. EOF < /dev/null

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment