Sensor Staleness — Current State and Improvement Proposals¶

This is the second occurrence of this bug class in the project. The first (battery_soc_entity, fixed in v2.0.2) is the precedent for the proposals below.

Mechanism (in brief)¶

HAWebSocketClient._parse_ts(state)
  → prefers state["last_updated"], falls back to state["last_changed"]
  → stored in _state_ts[entity_id]

get_state_age_sec(entity_id)
  → now(UTC) − _state_ts[entity_id]   (None if never received)

is_stale(entity_id, max_age_sec)
  → age is None  OR  age > max_age_sec

Three constants cover every staleness check in the project (const.py):

Dead feature: get_state_value(entity_id, max_age_sec=None) has an optional staleness parameter that is never passed a value at any call site in the codebase (verified: grep -n "get_state_value(" *.py). Every actual staleness check goes through the separate, explicit is_stale() calls listed below.

Two obvious shortcuts — and why only one holds up¶

Before the numbers: two ideas suggest themselves when you see this problem. Both were checked against the actual integration source on the production system. One does not hold up at all; the other only halfway.

Idea A: "Just use last_reported instead of last_updated"¶

Home Assistant maintains three timestamps per entity:

last_reported is in fact included in the REST JSON (verified live), so switching would be trivial. It still gains nothing here. All three fields only advance when the integration actually calls async_write_ha_state(). They measure "when did the integration last write" — not "when did it last reach the source".

And for the daily forecasts, the Solcast integration deliberately does not write (custom_components/solcast_solar/sensor.py, _handle_coordinator_update):

if self._update_policy == SensorUpdatePolicy.DEFAULT and not (
        self._coordinator.date_changed or self._coordinator.data_updated):
    return          # ← no async_write_ha_state()

get_sensor_update_policy() (same file) grants EVERY_TIME_INTERVAL to only nine keys — among them ENTITY_FORECAST_REMAINING_TODAY and ENTITY_POWER_NOW. Everything else, including the "today" and "tomorrow" daily totals, gets DEFAULT.

Measurement on the production system (18:14 UTC), all sensors from the same integration and the same coordinator:

The integration is demonstrably alive — it writes every five minutes. Only the daily total is not rewritten, because nothing changed semantically. For prognose_heute all three timestamps are identical: last_reported would reproduce the false alarm exactly. The same holds for the price sensor (sensor.deye8k_current_electricity_price: last_changed = last_reported = 16:00:00).

Idea B: "Doesn't the sensor just go unavailable when the source is gone?"¶

available is a purely opt-in property of each integration. The two integrations miniEMS depends on decide it in opposite ways:

Solarman/Deye (custom_components/solarman/entity.py:38) — the inverter:

def available(self) -> bool:
    return self.coordinator.last_update_success and self.coordinator.device.state.value > -1

→ Yes. Modbus connection lost ⇒ unavailable. A reliable, immediate signal.

Solcast (custom_components/solcast_solar/sensor.py:690):

def available(self) -> bool:
    return self._attr_available      # = (self._sensor_data is not None)

_sensor_data comes from the forecast cache persisted to disk, not from an API call. API down, quota exhausted, internet gone — the cache stays populated, the sensor stays available and serves yesterday's forecast indefinitely. → No, it never goes unavailable.

The unavailable half is already implemented in miniEMS: get_state_value() returns None for unavailable/unknown/"" (ha_ws_client.py:67), and _build_sensor_warnings turns that into "Sensor unavailable". For the Deye sensors this is therefore the dependable signal — the age check there is essentially redundancy.

What follows from both ideas¶

These are two orthogonal failure modes that today's age check conflates:

1. Connection dead → unavailable. Already solved, reliable for all hardware sensors.
2. Connection alive, data semantically outdated (Solcast forecast from yesterday) → neither unavailable nor any timestamp catches this. Only a semantic check helps.

Current state — every call site in the project¶

Legend for the last column: ✅ = both failure modes (connection dead / data outdated) covered · ⚠️ = only one of the two · ❌ = no effective detection at all.

battery_soc_entity itself has had no age check since v2.0.2 — only a presence check (get_state_value(...) is None). That is the already-solved instance of this bug class.

Worth noting: #4 and #6 come from the same integration and share the same constant even though their cadences have nothing in common — it fits #4 (5 h 50 min measured flat period against an 8 h threshold) while being too tight for #6. That one and the same value fits one sensor and produces false alarms on another from the same integration is demonstration enough that "age" cannot be the load-bearing measure here.

Root cause¶

Three constants effectively cover five distinct update-cadence classes:

The deeper cause, however, is not the constants. For class (d) no age limit can be structurally correct: the same forecast value is fresh at 09:00 and still correct at 23:00 — the timestamp's age says nothing about validity in either case. Raising the threshold merely postpones the problem.

Improvement proposals¶

1. Daily forecasts: check the date, not the age ✅¶

Instead of asking "how old is the timestamp?", ask the semantically correct question: "Is the timestamp from today?"

For class (d) this is answerable exactly, because the Solcast integration is guaranteed to write on every date rollover (coordinator.py):

self.tasks[TASK_LISTENERS] = async_track_utc_time_change(
    self.hass, self._update_integration_listeners, minute=range(0, 60, 5), second=0
)

_update_integration_listeners therefore runs every five minutes and sets:

current_day = dt.now(self.solcast.options.tz).day
self._date_changed = current_day != self._last_day

When date_changed is true, the early-return condition in _handle_coordinator_update no longer applies and all DEFAULT-policy sensors are rewritten. This yields a hard guarantee:

As long as the Solcast integration is running, prognose_heute and prognose_morgen carry a timestamp from the current day within five minutes of local midnight. A timestamp from yesterday necessarily means the integration has stopped.

This makes the check self-calibrating: no new constant, no new config field, no additional entity, and it stays correct across year boundaries and DST changes.

Two implementation notes:

• Use the local timezone. Solcast evaluates the date rollover in local time (dt.now(self.solcast.options.tz)), whereas _parse_ts() returns UTC. Under CEST (UTC+2), local midnight is 22:00 UTC the previous day — a date comparison in UTC would be wrong for two hours. The comparison must happen in the HA installation's local timezone.
• Small grace period. Between 00:00 and 00:05 local time the timestamp is still legitimately from yesterday. A tolerance of ~15 minutes after midnight avoids a daily false alarm inside that five-minute window.

Affects #5 and #6. Resolves the reported false alarm and closes the gap in the v2.0.1 grid-charge fallback.

2. Hardware sensors: unavailable is already the right signal¶

For #1, #2 and #7 the Solarman integration reports unavailable on connection loss — faster and more reliable than any age check, and already evaluated in get_state_value(). The 300-second check there is not wrong, but largely redundant; it can remain as a second line of defence.

3. Electricity price: no semantic equivalent — a small margin suffices ✅¶

For #3 there is neither a date reference nor a liveness proxy: the price sensor is written only on tier changes (last_changed = last_reported, confirmed live).

The upper bound is, however, known exactly: the tariff is purely time-driven, the schedule is fixed, and the longest window is exactly 6 h (06:00–12:00). No market dynamics can change that. So no generous buffer is needed — only enough clearance to avoid the "threshold == longest window" exact tie:

PRICE_MAX_AGE_SEC = 21720   # 6 h + 2 min – longest tariff window is exactly 6 h

A higher value would only degrade detection time without covering any real case.

The cleaner solution would be a plausibility check against the entity's activation_rules schedule — that schedule is present and describes the tariff profile completely. It requires access to entity attributes, which miniEMS currently has nowhere; this is a touchpoint with the grid-friendly charging roadmap, which proposes exactly that access as its first step.

4. Solcast data freshness (#4, #5, #6) ✅¶

For the three Solcast checks, only the "integration dead" failure mode is addressed so far. There is a second one that none of the signals above detects:

The Solcast API is unreachable (quota exhausted, internet gone, service disrupted) while the disk cache keeps serving unchanged.

In that state everything looks healthy:

The most serious case is #4, because that value feeds _should_grid_charge directly: the grid-charge decision can run on a days-old forecast without any warning appearing. Only once the cache no longer holds data for the current day — after roughly seven days — does _sensor_data become None and the sensor flip to unavailable, at which point the existing presence check applies. The days before that are blind.

Candidate for the gap: the value of sensor.solcast_pv_forecast_zeitpunkt_letzter_api_abruf — explicitly its content, not its timestamp (see the box below). Proven in the source (solcastapi.py:561):

@property
def last_updated(self) -> dt | None:
    """When the data was last updated.

    Returns:
        dt | None: The last successful forecast fetch.
    """
    return self.data[LAST_UPDATED].astimezone(self.tz) if self.data.get(LAST_UPDATED) is not None else None

The value comes from the persisted dataset and is therefore exactly the age of the forecast data, independent of how often HA rewrites the sensors. Verified live: 2026-08-15T08:31:25 at a measurement time of 18:14 UTC ⇒ forecast data 9 h 43 min old.

Implementation obstacle: this is a datetime sensor. get_state_value() does float(raw) and returns None for an ISO timestamp (ha_ws_client.py:69-72) — it would need a datetime path in the client plus a config field for the entity, analogous to load_consumption_entity in v2.0.3. A sensible threshold should be derived from the actual fetch rhythm rather than guessed; that measurement, across several days, is still missing.

5. Plausibility check on the curve shape (#4) — dropped¶

Instead of asking when the value last arrived, check whether it behaves the way it must. remaining_today has a mandatory daily shape: reset once, then decay to zero.

Measured across two full days (HA recorder, 356 data points, 13–15 Aug 2026):

That yields three refinements to the obvious formulation "zero at night, otherwise falling, one rise in the morning":

1. The rise sits on local midnight, not in the morning. It is the date rollover, not sunrise.
2. Only the evening dark window is zero. In the pre-dawn hours the value sits at its daily maximum, constant for 5 h 50 min — a rule "dark ⇒ 0" would raise a false alarm every night between 00:00 and 05:50 local.
3. Monotonicity only holds between forecast fetches. Solcast revises intraday; the measured steps were small at +0.35 and +0.13 kWh, but they are systematic and should be expected to be considerably larger on a changeable day. A strict monotonicity rule produces false alarms; it needs a tolerance, or an exemption at fetch moments (known from the value in proposal 4).

What the check delivers: it detects a frozen, implausible or miswired sensor during daylight within minutes instead of eight hours — over the course of the day the value moves every 5 minutes by amounts far above the noise floor (up to 0.47 kWh per interval measured). That is a genuine improvement in detection latency over the pure age check, and it covers faults none of the other measures can see.

What it does not deliver: it does not close the gap from proposal 4. The curve shape is produced by the cached spline plus the clock — with a days-dead API, the reset, night plateau, monotonic decay and evening zero all remain intact. Every single criterion of this check would pass. For data freshness, the value of zeitpunkt_letzter_api_abruf remains the only dependable signal.

Not part of this page¶

This page describes the current state and proposals only — none of the points above are implemented. The actual implementation (date check for daily forecasts, price margin) is a separate step.

Explicitly still without a finished solution is proposal 4 (Solcast data freshness,

4/#5/#6): the signal source is identified and proven in the integration source, but the¶

client extension, the config field and an empirically grounded threshold are all missing.

Proposal 5 (plausibility check) and proposal 1 (date check) each improve detection of integration failure — proposal 5 markedly so in speed. Neither covers data freshness; that remains the one open point of this page, and it affects #4 of all things, the only Solcast value with direct influence on a control decision.

Relation to the day-rollover roadmap¶

While reviewing this page a second defect surfaced, covered in Tageswechsel & Energiezählung (German): miniEMS cuts the day 4 min 54 s ahead of the inverter. The two pages are more closely related than the topics suggest.

What the day-rollover rework does not fix here¶

The delimitation first, so no false expectation arises: switching to total counters with a self-computed delta fixes not one row of the table above. The Solcast sensors have no total counterpart, and the price and power sensors are untouched by energy accounting. The two efforts are independently implementable.

What it contributes nonetheless¶

1. Proposal 1 rests on a precondition proven there. The date check only works if miniEMS's own day boundary reliably sits on local midnight — otherwise the Solcast date would be compared against a shifted boundary. That is exactly what the day-rollover page measures and confirms (TZ=Europe/Berlin in the container, cut observed live at 21:59:38 → 22:00:08 UTC). The precondition therefore need not be re-derived here — but if anything changes about the day boundary there, proposal 1 must be re-checked.

2. The same figure of thought, twice. Both pages arrive independently at the same rule:

Do not trust the derived artifact — move one level closer to the raw source and compute the quantity you need yourself.

There: don't inherit the foreign day cut, read the monotonic lifetime counter and compute the daily delta yourself. Here: don't check the write timestamp of the derived sensor, read the source's own fetch time (proposal 4). Understand one and you understand the other.

3. The monotonicity safeguard generalises. The day-rollover page requires of the total counter: a monotonically increasing value must never run backwards — if it does, that is a reportable event, not a computation case. The same discipline applies to zeitpunkt_letzter_api_abruf once proposal 4 is implemented: its value is monotonically increasing too, and a backwards jump would be an integration defect, not a number to interpret.

4. Coordination during implementation. Both efforts need new config fields and thus a schema migration (migration.py, CONFIG_SCHEMA_VERSION). If they land in the same release, they should share one schema version rather than two consecutive migration steps. Both would additionally touch config_loader.py and templates/settings.html.