Hidden From Reporting Tag - Implementation Plan

hidden_from_reporting_tag_c38f8bc8.plan.md

name	Hidden From Reporting Tag
overview	Add a `$Hidden From Reporting` standard data tag and make all reporting surfaces treat tagged fields as if they were deleted (not present in the dictionary).
todos	id content status register-tag Create packs/standard_data/catalog/hidden_from_reporting_tag.rb with TagDefinition registration pending id content status not-tagged-as Add `not_tagged_as` class method to CustomerData::Field as complement to `tagged_as` pending id content status field-scope Add `visible_in_reporting` scope to CustomerData::Field using `not_tagged_as` pending id content status dictionary-method Add `reportable_fields` method to CustomerData::Dictionary pending id content status update-availability Update MarketingDataAvailability#marketing_data_field_for_type_slot to use reportable_fields pending id content status update-helper Update MarketingDataHelper#setup_for_dictionary to use reportable_fields pending id content status update-detail Update DetailReportCustomerData#customer_data_field_for_type_index to use reportable_fields pending id content status update-summary Update SummaryReportCustomerData#dictionary_fields to use reportable_fields pending id content status tests Add unit and integration tests for scope, dictionary method, and all four reporting paths pending id content status post-deploy Create post-deploy script template for tagging specific fields pending
isProject	false

Hidden From Reporting Tag

Architecture Summary

When a CustomerData::Field is hard-deleted, these four independent field-lookup caches stop returning it, which causes all reporting surfaces to exclude the field:

flowchart TD
    subgraph fieldSources [Field Lookup Caches]
        A["MarketingDataAvailability\n#marketing_data_field_for_type_slot\ndict.fields.index_by(&:internal_name)"]
        B["MarketingDataHelper\n#setup_for_dictionary\ndict.fields.index_by(&:internal_name)"]
        C["DetailReportCustomerData\n#customer_data_field_for_type_index\ndict.fields.build_hash"]
        D["SummaryReportCustomerData\n#dictionary_fields\ndict.fields.to_a"]
    end

    subgraph consumers [Reporting Surfaces]
        E[ColumnCatalog / ReportAdapterCatalog\nES + warehouse reports]
        F[Legacy Detail Reports]
        G[Legacy Summary Reports]
        H[GraphQL reportQueryColumns]
    end

    A --> E
    B --> E
    B --> H
    C --> F
    D --> G

Loading

Our goal: make tagged fields disappear from these same four caches without actually deleting them.

---

Part A: Register the Tag in StandardData

Create a new catalog registration file following the exact pattern of existing tags like $Lead and $Consumer Property.

• New file: packs/standard_data/catalog/hidden_from_reporting_tag.rb
• Register one StandardData::TagDefinition:
  • tag_name: '$Hidden From Reporting'
  • data_type: :any (applies to all field types, like $Consumer Property)
  • description: and display_name: as appropriate
• The CatalogLoader auto-discovers *.rb in packs/standard_data/catalog/, so no wiring needed.

---

Part B: Exclude Tagged Fields from Reporting

Strategy: Single scope on CustomerData::Field, single method on Dictionary

Add one ActiveRecord scope and one dictionary convenience method, then update the four cache-building call sites.

Step 1a: Add not_tagged_as to CustomerData::Field

In packs/customer_data/app/models/customer_data/field.rb, add a general-purpose complement to the existing tagged_as scope. While tagged_as returns fields that have ALL of the specified tags, not_tagged_as excludes fields that have ANY of the specified tags:

# @param tag_names [Array<String>] tag names to exclude
# @return [ActiveRecord::Relation] Returns fields that have NONE of the specified tags
def not_tagged_as(*tag_names)
  return all if tag_names.empty?

  where.not(
    id: joins(:tags).where(customer_data_tags: { name: tag_names }).select(:id)
  )
end

This uses a NOT IN subquery on the existing HABTM :tags association, so it needs no new indexes (the join table PK is (field_id, tag_id) and customer_data_tags has a unique index on (dictionary_id, name)).

Step 1b: Add visible_in_reporting scope using not_tagged_as

scope :visible_in_reporting, -> { not_tagged_as('$Hidden From Reporting') }

This keeps the reporting-specific logic as a thin, readable wrapper over the reusable not_tagged_as primitive.

Step 2: Add reportable_fields to CustomerData::Dictionary

In packs/customer_data/app/models/customer_data/dictionary.rb, add:

def reportable_fields
  fields.visible_in_reporting
end

Step 3: Update the four cache-building sites

Each site changes dict.fields to dict.reportable_fields:

File	Method	Change
marketing_data_availability.rb	marketing_data_field_for_type_slot (line 69)	dict.fields.index_by -> dict.reportable_fields.index_by
marketing_data_helper.rb	setup_for_dictionary (line 19)	marketing_data_dictionary.fields.index_by -> marketing_data_dictionary.reportable_fields.index_by
detail_report_customer_data.rb	customer_data_field_for_type_index (line 407)	dict.fields.build_hash -> dict.reportable_fields.build_hash
summary_report_customer_data.rb	dictionary_fields (line 374)	effective_customer_dictionary.fields.to_a -> effective_customer_dictionary.reportable_fields.to_a

These four changes make tagged fields disappear from all downstream reporting surfaces (ColumnCatalog, ReportAdapterCatalog, GraphQL reportQueryColumns, legacy detail/summary reports, ES queries) because every one of those consumers gets its field information from one of these four caches.

What is NOT affected (intentionally)

• Management UI (ManageCustomerData GraphQL resolvers): still uses dictionary.fields directly -- hidden fields remain visible and editable.
• LookupFieldCatalog (lookup tables / spreadsheets): uses @dictionary.fields -- lookup tables can still reference hidden fields. (Decide if this should also filter.)
• Consumer profile import (tagged_as("$Consumer Property")): unrelated tag, unaffected.
• Data ingest / JS tag / call processing: writes to fields by slot regardless of tags.

---

Part C: Apply the Tag to Fields (post-deploy script)

Create a post-deploy script to tag specific fields. The mechanism uses the existing Field#standard_data_tag_names= API:

field.standard_data_tag_names = field.standard_data_tag_names.to_a + ['$Hidden From Reporting']

(Which fields to tag is a separate decision -- the script template will be ready.)

---

Testing Strategy

• Unit: CustomerData::Field spec for not_tagged_as -- excludes fields with any of the specified tags, returns all when no tags given
• Unit: CustomerData::Field spec for visible_in_reporting scope -- field with/without the $Hidden From Reporting tag
• Unit: CustomerData::Dictionary spec for reportable_fields
• Unit: MarketingDataAvailability spec -- field tagged $Hidden From Reporting returns nil from marketing_data_field_available
• Unit: MarketingDataHelper spec -- setup_for_dictionary excludes hidden field from @fields_hash
• Integration: Detail/summary report specs -- hidden field columns are not visible
• Catalog: StandardData::Catalog spec -- $Hidden From Reporting tag is registered

---

Decision Points

1. Should LookupFieldCatalog also exclude hidden fields? Currently it iterates @dictionary.fields for lookup table column pickers. If yes, same one-line change.
2. Should the management UI indicate a field is hidden from reporting? The GraphQL customerDataFields resolver could expose tag info for UI treatment.