Home Assistant Blueprint for monitoring a 3D printer using an AI Task

3D Print - AI Health Check.yaml

blueprint:
  name: AI 3D Printer Status Check
  description: |
    Monitors a camera on a repeating cadence **while** a state sensor equals a target value (case‑insensitive).
    Each cycle runs a Home Assistant **AI Task** on a frame and exposes the AI result + snapshot path to an optional
    user script for platform‑specific notifications.

    • Local AI = attach **camera** directly to the AI Task (fast, high quality).
    • Remote AI = attach **snapshot** file via Local Media (requires allowlisted media path; see inputs).
    • Optional confidence threshold (fixed number or entity) gates the built‑in text alert.
    • Built‑in **debug** and **failure** messages are text‑only; image sending is delegated to your script.
    • Snapshot naming: overwrite a single file or write timestamped files (no automatic cleanup).

    Notes:
    - Image sending is delegated to **script_on_result**, keeping this blueprint notify‑platform agnostic.
    - The script receives **nested objects** for clarity and long‑term stability (see the Scripts section definitions).
  domain: automation

  input:
    core:
      name: Core inputs
      icon: mdi:camera
      collapsed: false
      description: >
        Required entities and cadence.
      input:
        camera_entity:
          name: Camera
          selector:
            entity:
              domain: camera
        stage_sensor:
          name: State sensor
          description: Entity whose state controls when monitoring is active
          selector:
            entity:
              domain: sensor
        active_state:
          name: Active state value
          description: Case‑insensitive match that enables monitoring (e.g., "Printing")
          default: "Printing"
          selector:
            text: {}
        interval_minutes:
          name: Interval (minutes)
          default: 15
          selector:
            number:
              min: 1
              max: 120
              step: 1
              mode: slider

    ai_settings:
      name: AI settings
      icon: mdi:robot
      collapsed: false
      description: Choose how to attach imagery and which AI Task to use. Remote AI requires Local Media access.
      input:
        local_model:
          name: Use local AI (attach camera directly)
          description: >
            ON: attach the camera live frame via media‑source (good for local backends like Ollama).
            OFF: attach a snapshot file via Local Media (recommended for remote providers).
          default: true
          selector:
            boolean: {}
        ai_task_entity:
          name: AI Task entity (optional)
          description: If set, this AI Task will be used instead of the system default.
          default: ""
          selector:
            entity:
              domain: ai_task
        snapshot_path_base:
          name: Local Media filesystem path
          description: >
            Filesystem path for your Local Media folder (NOT a URL). Commonly **/media**, sometimes **/config/media**.
            Must be listed under **homeassistant.allowlist_external_dirs** and the subfolder must exist.
          default: "/media"
          selector:
            text: {}
        snapshot_subfolder:
          name: Subfolder (under Local Media)
          description: >
            Optional subfolder name. If blank, defaults to `ai_task/<camera_object_id>` to avoid collisions.
          default: ""
          selector:
            text: {}

    thresholds:
      name: Thresholds (optional)
      icon: mdi:tune-variant
      collapsed: true
      description: >
        Gate the built‑in text alert by minimum AI confidence. Your result script still receives all results.
      input:
        confidence_threshold:
          name: Confidence threshold (fixed)
          description: Notify only when normalized confidence ≥ this value (0–100). Leave blank to disable.
          selector:
            number:
              min: 0
              max: 100
              step: 1
        confidence_threshold_entity:
          name: Confidence threshold entity (number or sensor)
          description: If set, this entity's numeric value (0–100) takes precedence over the fixed value.
          default: ""
          selector:
            entity:
              domain:
                - number
                - sensor

    scripts:
      name: Script (optional)
      icon: mdi:script-text-outline
      collapsed: true
      description: |
        Provide a script to handle platform‑specific notifications or archival.
        The same script is called for success **and** failure; on failure, an `error` object is included.

        <details>
        <summary><strong>script_on_result payload (nested objects)</strong></summary>

        ```yaml
        # On success:
        ai_result:
          has_problem: bool
          problem_type: string|null
          advice: string|null
          confidence:
            raw: number|null          # as returned by the model (0.85 or 85)
            normalized: number|null   # 0–100; null if model omitted confidence
        snapshot:
          abs_path: string            # absolute file path
          rel_path: string            # relative under Local Media (e.g., subfolder/filename.jpg)
          provider_mode: string       # "local" or "remote"
        script_context:
          camera_entity: string
          ai_task_entity: string|null
          ts: string                  # ISO8601 timestamp
          threshold:
            active: boolean
            value: number|null
          notified: boolean           # whether built‑in alert fired this cycle
        error:
          message: string             # not provided if there is no error
        ```
        </details>
      input:
        script_on_result:
          name: Script on result (success or failure)
          description: Optional script called each cycle; receives nested objects as above.
          default: ""
          selector:
            entity:
              domain: script

    housekeeping:
      name: Housekeeping (optional)
      icon: mdi:broom
      collapsed: true
      description: >
        Control snapshot file naming.
        This blueprint does not delete files. If you choose timestamped files, you'll need to handle cleanup yourself
        (e.g., via another automation/script or an external tool like a cron job).
      input:
        overwrite_snapshot:
          name: Overwrite single file (latest.jpg)
          description: If ON, writes `<subfolder>/latest.jpg` each run. OFF writes timestamped files (no auto-cleanup).
          default: true
          selector:
            boolean: {}

    notifications:
      name: Built‑in notifications (optional)
      icon: mdi:message-outline
      collapsed: true
      description: Text‑only status messages from the blueprint itself.
      input:
        notify_action:
          name: Custom notify action (optional)
          description: >
            An action (or sequence) to run when the blueprint emits a text notification. If provided, it will be
            used instead of the simple notify service below. This action can reference variables like `ai`,
            `norm_conf`, `abs_path`, `rel_path`, etc.
          default: []
          selector:
            action: {}
        notify_service:
          name: Notify service for text pings (fallback)
          description: e.g., `notify.mobile_app_<device>` or `notify.persistent_notification`. Leave blank to disable if not using a custom action.
          default: ""
          selector:
            text: {}

    debugging:
      name: Debug (optional)
      icon: mdi:bug
      collapsed: true
      description: Extra telemetry each cycle.
      input:
        debug_mode:
          name: Debug mode
          default: false
          selector:
            boolean: {}

# Triggers
triggers:
  - trigger: state
    entity_id: !input stage_sensor
  - trigger: homeassistant
    event: start

mode: restart

# Variables
variables:
  cam: !input camera_entity
  stage: !input stage_sensor
  interval: !input interval_minutes
  active_val: !input active_state

  use_local: !input local_model
  ai_entity: !input ai_task_entity
  snapshot_base: !input snapshot_path_base
  snapshot_subfolder: !input snapshot_subfolder

  thr_fixed: !input confidence_threshold
  thr_entity: !input confidence_threshold_entity

  script_result: !input script_on_result

  overwrite: !input overwrite_snapshot

  notify_act: !input notify_action
  notify_svc: !input notify_service
  debug_enabled: !input debug_mode

  has_notify_action: "{{ (notify_act | default([])) | length > 0 }}"
  has_notify_service: "{{ (notify_svc | string) | length > 0 }}"

  entity_provided: "{{ (thr_entity | string) | length > 0 }}"
  thr_from_entity: "{{ states(thr_entity) | float('nan') if entity_provided else float('nan') }}"
  thr_from_fixed: "{{ thr_fixed | float('nan') if (thr_fixed is not none) else float('nan') }}"
  conf_threshold: >
    {% if entity_provided %}{{ thr_from_entity }}{% else %}{{ thr_from_fixed }}{% endif %}
  threshold_active: "{{ conf_threshold == conf_threshold }}"
  both_thresholds_provided: "{{ entity_provided and (thr_fixed is not none) }}"

  # Resolve subfolder (default: ai_task/<camera object_id>) and filenames
  cam_obj: "{{ (cam | string).split('.')[-1] }}"
  folder: >
    {% set sf = (snapshot_subfolder | string) %}
    {% if sf|length > 0 %}{{ sf.strip('/') }}{% else %}ai_task/{{ cam_obj }}{% endif %}

  # Shared AI instructions & schema (generic wording)
  ai_instr: >-
    You are inspecting a 3D printer's camera frame. Determine if the print is going wrong.
    Common failures include spaghetti (stringy filament), poor first‑layer adhesion, severe warping/corner lift,
    nozzle clog with under‑extrusion, layer shift, knocked‑over part, filament blob near the hotend, or the part
    detaching from the bed. If there is meaningful risk of failure without intervention, set has_problem to true
    and provide brief, actionable advice.
  ai_schema: >-
    {
      "has_problem": {"selector": {"boolean": {}}, "required": true},
      "problem_type": {"selector": {"text": {}}},
      "confidence": {"selector": {"number": {}}, "description": "0–100 (fraction like 0.85 will be interpreted as 85%)."},
      "advice": {"selector": {"text": {}}}
    }

actions:
  # Warn once if both threshold inputs are filled — entity will be used.
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ both_thresholds_provided }}"
        sequence:
          - action: persistent_notification.create
            data:
              title: "AI image health — threshold conflict"
              message: >
                Both a fixed confidence threshold and a threshold entity were provided.
                The entity value will be used. Clear one of them to remove this message.

  # Only proceed if currently active (case‑insensitive)
  - condition: template
    value_template: "{{ (states(stage) | lower) == (active_val | lower) }}"

  # Jitter
  - delay:
      seconds: "{{ range(0, 5) | random }}"

  # Loop while active
  - repeat:
      until:
        - condition: template
          value_template: "{{ (states(stage) | lower) != (active_val | lower) }}"
      sequence:
        # Build paths and snapshot once per cycle (used for scripts & remote AI)
        - variables:
            ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
            filename: >-
              {% if overwrite %}latest.jpg{% else %}{{ ts }}.jpg{% endif %}
            rel_path: "{{ folder }}/{{ filename }}"
            abs_path: "{{ snapshot_base.rstrip('/') ~ '/' ~ rel_path }}"
            media_source_id: "media-source://media_source/local/{{ rel_path }}"
        - action: camera.snapshot
          target:
            entity_id: !input camera_entity
          data:
            filename: "{{ abs_path }}"
        - delay: "00:00:02"

        # Run AI Task (local uses camera; remote uses snapshot file)
        - choose:
            - conditions:
                - condition: template
                  value_template: "{{ use_local | bool }}"
              sequence:
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ (ai_entity | string) | length > 0 }}"
                      sequence:
                        - action: ai_task.generate_data
                          continue_on_error: true
                          data:
                            entity_id: !input ai_task_entity
                            task_name: "image health check"
                            instructions: "{{ ai_instr }}"
                            structure: "{{ ai_schema | from_json }}"
                            attachments:
                              media_content_id: "media-source://camera/{{ cam }}"
                              media_content_type: image/jpeg
                          response_variable: ai
                    - conditions:
                        - condition: template
                          value_template: "{{ (ai_entity | string) | length == 0 }}"
                      sequence:
                        - action: ai_task.generate_data
                          continue_on_error: true
                          data:
                            task_name: "image health check"
                            instructions: "{{ ai_instr }}"
                            structure: "{{ ai_schema | from_json }}"
                            attachments:
                              media_content_id: "media-source://camera/{{ cam }}"
                              media_content_type: image/jpeg
                          response_variable: ai
            - conditions:
                - condition: template
                  value_template: "{{ not (use_local | bool) }}"
              sequence:
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ (ai_entity | string) | length > 0 }}"
                      sequence:
                        - action: ai_task.generate_data
                          continue_on_error: true
                          data:
                            entity_id: !input ai_task_entity
                            task_name: "image health check (snapshot)"
                            instructions: "{{ ai_instr }}"
                            structure: "{{ ai_schema | from_json }}"
                            attachments:
                              media_content_id: "{{ media_source_id }}"
                              media_content_type: image/jpeg
                          response_variable: ai
                    - conditions:
                        - condition: template
                          value_template: "{{ (ai_entity | string) | length == 0 }}"
                      sequence:
                        - action: ai_task.generate_data
                          continue_on_error: true
                          data:
                            task_name: "image health check (snapshot)"
                            instructions: "{{ ai_instr }}"
                            structure: "{{ ai_schema | from_json }}"
                            attachments:
                              media_content_id: "{{ media_source_id }}"
                              media_content_type: image/jpeg
                          response_variable: ai

        # If AI failed, call the single script with error (and optionally text‑notify)
        - choose:
            - conditions:
                - condition: template
                  value_template: "{{ ai is not defined }}"
              sequence:
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ (script_result | string) | length > 0 }}"
                      sequence:
                        - action: !input script_on_result
                          data:
                            error:
                              message: "no assistant content returned"
                            snapshot:
                              abs_path: "{{ abs_path }}"
                              rel_path: "{{ rel_path }}"
                              provider_mode: "{{ 'local' if (use_local | bool) else 'remote' }}"
                            script_context:
                              camera_entity: "{{ cam }}"
                              ai_task_entity: "{{ ai_entity if (ai_entity | string) | length > 0 else none }}"
                              ts: "{{ now().isoformat() }}"
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ has_notify_action | bool }}"
                      sequence: !input notify_action
                    - conditions:
                        - condition: template
                          value_template: "{{ (not has_notify_action) and has_notify_service }}"
                      sequence:
                        - action: !input notify_service
                          data:
                            title: "AI image health — processing failed"
                            message: >-
                              The AI task did not return a valid assistant message this cycle.
                              Try a different instruct/JSON‑friendly model. (Monitoring continues.)

        # If AI succeeded, optionally send built‑in text alert (threshold‑aware) and call result script
        - variables:
            raw_conf: >-
              {% if ai is defined and (ai.data.confidence is defined or ai.data.get('confidence') is not none) %}
                {{ ai.data.confidence if (ai.data.confidence is defined) else ai.data.get('confidence') }}
              {% else %}{{ none }}{% endif %}
            norm_conf: >-
              {% if raw_conf is not none %}
                {% set c = raw_conf | float(0) %}{{ (c * 100) if c <= 1 else c }}
              {% else %}{{ none }}{% endif %}
            problem_ok: "{{ ai is defined and (ai.data.has_problem | default(false)) }}"
            threshold_ok: >-
              {% if not threshold_active %}true{% else %}
                {% if norm_conf is none %}false{% else %}{{ (norm_conf | float(0)) >= (conf_threshold | float(0)) }}{% endif %}
              {% endif %}
            should_notify: "{{ problem_ok and threshold_ok }}"
        - choose:
            - conditions:
                - condition: template
                  value_template: "{{ should_notify | bool }}"
              sequence:
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ has_notify_action | bool }}"
                      sequence: !input notify_action
                    - conditions:
                        - condition: template
                          value_template: "{{ (not has_notify_action) and has_notify_service }}"
                      sequence:
                        - action: !input notify_service
                          data:
                            title: "⚠️ Image Issue Detected"
                            message: >-
                              Type: {{ ai.data.problem_type | default('unknown') }}
                              Confidence: {{ (norm_conf | float(0)) | round(1) if (norm_conf is not none) else '—' }}%
                              Advice: {{ ai.data.advice | default('Check the print ASAP.') }}
                              (via AI Task)
        - choose:
            - conditions:
                - condition: template
                  value_template: "{{ (script_result | string) | length > 0 and ai is defined }}"
              sequence:
                - action: !input script_on_result
                  data:
                    ai_result:
                      has_problem: "{{ ai.data.has_problem | default(false) }}"
                      problem_type: "{{ ai.data.problem_type | default(none) }}"
                      advice: "{{ ai.data.advice | default(none) }}"
                      confidence:
                        raw: "{{ raw_conf if (raw_conf is not none) else none }}"
                        normalized: "{{ norm_conf if (norm_conf is not none) else none }}"
                    snapshot:
                      abs_path: "{{ abs_path }}"
                      rel_path: "{{ rel_path }}"
                      provider_mode: "{{ 'local' if (use_local | bool) else 'remote' }}"
                    script_context:
                      camera_entity: "{{ cam }}"
                      ai_task_entity: "{{ ai_entity if (ai_entity | string) | length > 0 else none }}"
                      ts: "{{ now().isoformat() }}"
                      threshold:
                        active: "{{ threshold_active | bool }}"
                        value: "{{ conf_threshold if threshold_active else none }}"
                      notified: "{{ should_notify | bool }}"

        # Debug (text‑only)
        - choose:
            - conditions:
                - condition: template
                  value_template: "{{ debug_enabled | bool }}"
              sequence:
                - choose:
                    - conditions:
                        - condition: template
                          value_template: "{{ has_notify_action | bool }}"
                      sequence: !input notify_action
                    - conditions:
                        - condition: template
                          value_template: "{{ (not has_notify_action) and has_notify_service }}"
                      sequence:
                        - action: !input notify_service
                          data:
                            title: "AI image health — debug"
                            message: >-
                              {% if ai is not defined %}
                              AI failed this cycle (no assistant content).
                              {% else %}
                              Has Problem={{ ai.data.has_problem | default(false) }},
                              Confidence (raw)={{ raw_conf | default('—') }},
                              Confidence (normalized)={{ (norm_conf | float(0)) | round(2) if (norm_conf is not none) else '—' }},
                              Threshold Active={{ threshold_active }}, 
                              Threshold={{ (conf_threshold | float(0)) | round(1) if threshold_active else '—' }},
                              Notified={{ should_notify | bool }}
                              {% endif %}

        # Wait until next cycle; loop ends when state stops matching
        - delay:
            minutes: "{{ interval | int }}"