Virtual Serial Proxy — allow components to share UART data with HA

[FredM67]

Context

This idea emerged from a discussion with @bdraco on PR esphome/esphome#9027 (emontx component). The emontx component reads UART data to parse JSON and update sensors, but also needs to forward a copy of that data bidirectionally to Home Assistant for a configuration panel (HACS integration).

bdraco's observations (from PR #9027 review)

bdraco: "It looks the purpose of this component is to proxy serial data to Home Assistant. It likely could use the new serial_proxy without having to implement anything at all new in ESPHome."

After discussion, the fundamental issue was identified:

bdraco: "Yeah that's a problem for sure. You can't share [the UART] so you'd have to implement all the logic in Home Assistant instead of in ESPHome."

bdraco: "We can't merge something that uses the HomeassistantActionRequest message directly outside of the Home Assistant path because that would make it a public API and open the door to other things using it."

bdraco suggested a longer-term solution:

bdraco: "Longer term, for the config panel approach we need something like

message DeviceEventResponse {
    string event_type = 1;    // e.g. "esphome.emontx_raw"
    repeated KeyValuePair data = 2;
}

that could be used with a more generic stable API [...] Someone would need to champion all that work and make the changes in aioesphomeapi and get the arch change through the Home Assistant process as well."

Problem Statement

A common pattern exists where an ESPHome component:

1. Owns a UART — reads, parses, and processes data (e.g., updating sensors)
2. Wants to also forward a copy of that raw data bidirectionally to Home Assistant

This cannot use serial_proxy because both components would try to read from the same UART (reads are destructive — bytes consumed by one are gone for the other).

This cannot use HomeassistantActionRequest because that's restricted to the homeassistant component (internal API, not public).

Adding component-specific protobuf messages doesn't scale ("we can't start allocating protobuf messages to specific components").

Proposed Solution: Virtual Serial Proxy

Instead of creating new protobuf messages, extend the existing serial_proxy with a VIRTUAL port type that decouples the API communication from the physical UART.

A component that owns the UART creates a virtual serial proxy and pushes data into it. Home Assistant subscribes to it using the exact same protocol as a physical serial proxy. Zero new protobuf messages, zero aioesphomeapi changes.

Architecture

SerialProxyBase (Component)                    [extracted from existing code]
├── API subscription management
├── push_to_client(data, len)
├── on_client_data() — virtual method
├── Pre-allocated outgoing protobuf message
│
├── SerialProxy (+ UARTDevice)                 [existing, refactored]
│   ├── loop() reads UART → push_to_client()
│   ├── on_client_data() → write_array()
│   ├── configure(), flush_port(), modem pins
│
└── VirtualSerialProxy                          [new, ~30 lines]
    ├── on_client_data() → invokes callback
    ├── flush_port() → NOT_SUPPORTED
    └── configure() → no-op

Changes Required

1. Proto (api.proto) — one line addition

enum SerialProxyPortType {
  SERIAL_PROXY_PORT_TYPE_TTL = 0;
  SERIAL_PROXY_PORT_TYPE_RS232 = 1;
  SERIAL_PROXY_PORT_TYPE_RS485 = 2;
  SERIAL_PROXY_PORT_TYPE_VIRTUAL = 3;   // NEW
}

No new messages. All existing subscribe/write/data messages work unchanged.

2. C++ — Extract SerialProxyBase from SerialProxy

Move the API communication code (subscription management, push_to_client(), instance indexing, pre-allocated message) into a base class. The existing SerialProxy inherits from it + UARTDevice. A new VirtualSerialProxy inherits from it alone.

class SerialProxyBase : public Component {
 public:
  void setup() override;

  uint32_t get_instance_index() const;
  void set_instance_index(uint32_t index);
  void set_name(const char *name);
  void set_port_type(api::enums::SerialProxyPortType port_type);

  api::APIConnection *get_api_connection();
  void serial_proxy_request(api::APIConnection *conn, api::enums::SerialProxyRequestType type);

  /// Send data to subscribed API client
  void push_to_client(const uint8_t *data, size_t len);

  /// Receive data from API client — subclass implements
  virtual void on_client_data(const uint8_t *data, size_t len) = 0;

  /// Remote UART configuration — subclass may override
  virtual void configure(uint32_t baudrate, bool flow_control,
                         uint8_t parity, uint8_t stop_bits, uint8_t data_size) {}

  /// Flush — subclass may override
  virtual uart::UARTFlushResult flush_port() {
    return uart::UART_FLUSH_RESULT_NOT_SUPPORTED;
  }

 protected:
  uint32_t instance_index_{0};
  api::APIConnection *api_connection_{nullptr};
#ifdef USE_API
  api::SerialProxyDataReceived outgoing_msg_;
#endif
  const char *name_{nullptr};
  api::enums::SerialProxyPortType port_type_{};
};

3. C++ — VirtualSerialProxy (~30 lines)

using virtual_proxy_data_cb_t = void (*)(void *context, const uint8_t *data, size_t len);

class VirtualSerialProxy : public SerialProxyBase {
 public:
  void dump_config() override;

  void on_client_data(const uint8_t *data, size_t len) override {
    if (this->on_data_cb_ != nullptr)
      this->on_data_cb_(this->on_data_context_, data, len);
  }

  void set_on_data_callback(virtual_proxy_data_cb_t cb, void *context) {
    this->on_data_cb_ = cb;
    this->on_data_context_ = context;
  }

 protected:
  virtual_proxy_data_cb_t on_data_cb_{nullptr};
  void *on_data_context_{nullptr};
};

4. application.h — widen pointer type

// Was: StaticVector<serial_proxy::SerialProxy *>
StaticVector<serial_proxy::SerialProxyBase *, SERIAL_PROXY_COUNT> serial_proxies_;

5. api_connection.cpp — rename write_from_client → on_client_data

Same dispatch logic via App.get_serial_proxies(), just calls the virtual method.

6. Python — add VIRTUAL port type, make uart_id optional for virtual

7. aioesphomeapi — no changes needed

Virtual proxies appear in DeviceInfoResponse.serial_proxies with port_type=VIRTUAL (value 3). Clients subscribe, write, and receive data using the exact same messages as physical proxies.

Usage Example

A component that reads UART and wants to share data with HA:

# Component internally creates a virtual serial proxy
emontx:
  config_panel: true

// In setup(): register callback for data from HA
this->virtual_proxy_->set_on_data_callback(
    [](void *ctx, const uint8_t *data, size_t len) {
      static_cast<EmonTx *>(ctx)->on_proxy_data_(data, len);
    }, this);

// In loop(): after reading a UART line, forward copy to HA
if (this->virtual_proxy_ != nullptr &&
    this->virtual_proxy_->get_api_connection() != nullptr) {
  this->virtual_proxy_->push_to_client(
      reinterpret_cast<const uint8_t *>(this->buffer_.data()),
      this->buffer_pos_);
}

The HA-side integration discovers the virtual proxy and subscribes using the standard serial proxy API — no custom services needed.

Standalone YAML Usage

A virtual serial proxy could also be configured directly in YAML for other use cases:

serial_proxy:
  - name: "My Data Channel"
    port_type: VIRTUAL
    id: my_virtual_proxy

Other components could then reference my_virtual_proxy to push/receive data.

Benefits

Aspect	Status
New protobuf messages	Zero — reuses existing serial proxy protocol
aioesphomeapi changes	Zero — virtual proxies are just serial proxies with port_type=3
Generic / reusable	Yes — any UART-owning component can use this pattern
Bidirectional	Yes — subscribe, write, data all work
Code footprint	Minimal — base class extraction ~100 lines, VirtualSerialProxy ~30 lines
Heap allocation	None in hot path — push_to_client uses pre-allocated protobuf message

Comparison with Alternatives

Approach	Proto changes	aioesphomeapi	HA integration	Generic
Virtual Serial Proxy	1 enum value	None	None	Yes
Generic DeviceEvent message	New messages	New support	New handling	Yes
Extend Event entity	Modified message	Modified support	Modified handling	Partially
Component-specific proto	New messages per component	Per component	Per component	No

Who Benefits

• emontx — config panel bidirectional communication (the immediate use case, PR [emontx] emonTx component esphome#9027)
• Any future component that parses UART data but also needs to share raw data with HA
• Any HA integration that needs a generic data channel to an ESPHome device

CC @bdraco @kbx81