stream

Models for mio.stream_daq

class mio.models.stream.ADCScaling(*, ref_voltage: float = 1.1, bitdepth: int = 8, battery_div_factor: float = 5.0, vin_div_factor: float = 11.3)

Configuration for the ADC scaling factors

battery_div_factor: float
bitdepth: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ref_voltage: float
scale_battery_voltage(voltage_raw: float) → float

Scale raw input ADC voltage to Volts

Parameters:

voltage_raw – Voltage as output by the ADC

Returns:

Scaled voltage

Return type:

float

scale_input_voltage(voltage_raw: float) → float

Scale raw input ADC voltage to Volts

Parameters:

voltage_raw – Voltage as output by the ADC

Returns:

Scaled voltage

Return type:

float

vin_div_factor: float
class mio.models.stream.RuntimeMetadata(*, buffer_recv_index: int = -1, buffer_recv_unix_time: float = -1.0, black_padding_px: int = -1, reconstructed_frame_index: int = -1)

Runtime metadata for data streams.

black_padding_px: int
buffer_recv_index: int
buffer_recv_unix_time: float
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

reconstructed_frame_index: int
class mio.models.stream.StreamBufferHeader(*, linked_list: int, frame_num: int, buffer_count: int, frame_buffer_count: int, write_buffer_count: int, dropped_buffer_count: int, timestamp: int, write_timestamp: int, pixel_count: int, battery_voltage_raw: int, input_voltage_raw: int, runtime_metadata: RuntimeMetadata = <factory>)

Refinements of BufferHeader for StreamDevice

property adc_scaling: ADCScaling | None

ADCScaling applied to voltage readings

property battery_voltage: float

Scaled battery voltage in Volts.

battery_voltage_raw: int
classmethod csv_header_cols(header_format: StreamBufferHeaderFormat) → list[str]

Return the standardized column names for CSV output.

This ensures consistent column ordering across all StreamBufferHeader instances when writing to CSV files.

Parameters:

header_format – The StreamBufferHeaderFormat instance to get column ordering from

Returns:

Column names in the order they should appear in CSV output

Return type:

list[str]

classmethod from_format(vals: Sequence, format: StreamBufferHeaderFormat, construct: bool = False, runtime_metadata: RuntimeMetadata = None) → Self

Instantiate a stream buffer header from linearized values (eg. in an ndarray or list), an associated format that tells us what index the model values are found in that data, and runtime metadata container.

Parameters:

  • vals (list, numpy.ndarray) – Indexable values to cast to the header model

  • format (BufferHeaderFormat) – Format used to index values

  • construct (bool) – If True , use model_construct() to create the model instance (ie. without validation, but faster). Default: False

  • runtime_metadata (RuntimeMetadata, optional) – Runtime metadata to attach to the header.

Returns:

StreamBufferHeader

property input_voltage: float

Scaled input voltage in Volts.

input_voltage_raw: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_dump_all(warning: bool = False) → dict

Return a dictionary of the model values, including runtime metadata if available.

Returns:

Dictionary of model values

Return type:

dict

model_post_init(context: Any, /) → None

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self – The BaseModel instance.

  • context – The context.

pixel_count: int
runtime_metadata: RuntimeMetadata
class mio.models.stream.StreamBufferHeaderFormat(*, id: Annotated[str, _PydanticGeneralMetadata(pattern='[\\w\\-\\/#]+')], mio_model: Annotated[str, AfterValidator(func=_is_identifier)] = None, mio_version: str = '0.10.1.dev6+g8c206e3', linked_list: int, frame_num: int, buffer_count: int, frame_buffer_count: int, write_buffer_count: int, dropped_buffer_count: int, timestamp: int, write_timestamp: int, pixel_count: int, battery_voltage_raw: int, input_voltage_raw: int)

Refinements of BufferHeaderFormat for StreamDevice

Parameters:

  • pixel_count (int) – Number of pixels in the buffer.

  • battery_voltage (int) – Battery voltage. This is currently raw ADC value. Mapping to mV will be documented in device documentation.

  • vin_voltage (int) – Input voltage. This is currently raw ADC value. Mapping to mV will be documented in device documentation.

battery_voltage_raw: int
input_voltage_raw: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

pixel_count: int
class mio.models.stream.StreamDevConfig(*, id: Annotated[str, _PydanticGeneralMetadata(pattern='[\\w\\-\\/#]+')], mio_model: Annotated[str, AfterValidator(func=_is_identifier)] = None, mio_version: str = '0.10.1.dev6+g8c206e3', device: Literal['OK', 'UART'], bitstream: Path | None = None, port: str | None = None, baudrate: int | None = None, frame_width: int, frame_height: int, fs: int = 20, preamble: bytes, header_len: int, pix_depth: int = 8, buffer_block_length: int, block_size: int, num_buffers: int, reverse_header_bits: bool = False, reverse_header_bytes: bool = False, reverse_payload_bits: bool = False, reverse_payload_bytes: bool = False, dummy_words: int = 0, adc_scale: ADCScaling | None = ADCScaling(ref_voltage=1.1, bitdepth=8, battery_div_factor=5.0, vin_div_factor=11.3), runtime: StreamDevRuntime = StreamDevRuntime(serial_buffer_queue_size=10, frame_buffer_queue_size=5, image_buffer_queue_size=5, queue_put_timeout=5, plot=StreamPlotterConfig(keys=['timestamp', 'buffer_count', 'frame_buffer_count'], update_ms=1000, history=500), csvwriter=CSVWriterConfig(buffer=100), ntp_server=None, ntp_max_offset_seconds=0.01))

Format model used to parse DAQ configuration yaml file (examples are in ./config) The model attributes are key-value pairs needed for reconstructing frames from data streams.

Parameters:

  • device (str) – Interface hardware used for receiving data. Current options are “OK” (Opal Kelly XEM 7310) and “UART” (generic UART-USB converters). Only “OK” is supported at the moment.

  • bitstream (str, optional) – Required when device is “OK”. The configuration bitstream file to upload to the Opal Kelly board. This uploads a Manchester decoder HDL and different bitstream files are required to configure different data rates and bit polarity. This is a binary file synthesized using Vivado, and details for generating this file will be provided in later updates.

  • port (str, optional) – Required when device is “UART”. COM port connected to the UART-USB converter.

  • baudrate (Optional[int]) – Required when device is “UART”. Baudrate of the connection to the UART-USB converter.

  • frame_width (int) – Frame width of transferred image. This is used to reconstruct image.

  • frame_height (int) – Frame height of transferred image. This is used to reconstruct image.

  • fs (int) – Framerate of acquired stream

  • preamble (str) – 32-bit preamble used to locate the start of each buffer. The header and image data follows this preamble. This is used as a hex but imported as a string because yaml doesn’t support hex format.

  • header_len (int, optional) – Length of header in bits. (For 32-bit words, 32 * number of words) This is useful when not all the variable/words in the header are defined in MetadataHeaderFormat. The user is responsible to ensure that header_len is larger than the largest bit position defined in MetadataHeaderFormat otherwise unexpected behavior might occur.

  • pix_depth (int, optional) – Bit-depth of each pixel, by default 8.

  • buffer_block_length (int) – Defines the data buffer structure. This value needs to match the Miniscope firmware. Number of blocks per each data buffer. This is required to calculate the number of pixels contained in one data buffer.

  • block_size (int) – Defines the data buffer structure. This value needs to match the Miniscope firmware. Number of 32-bit words per data block. This is required to calculate the number of pixels contained in one data buffer.

  • num_buffers (int) – Defines the data buffer structure. This value needs to match the Miniscope firmware. This is the number of buffers that the source microcontroller cycles around. This isn’t strictly required for data reconstruction but useful for debugging.

  • reverse_header_bits (bool, optional) – If True, reverse the bits within each byte of the header. Default is False.

  • reverse_header_bytes (bool, optional) – If True, reverse the byte order within each 32-bit word of the header. This is used for handling endianness in systems where the byte order needs to be swapped. Default is False.

  • reverse_payload_bits (bool, optional) – If True, reverse the bits within each byte of the payload. Default is False.

  • reverse_payload_bytes (bool, optional) – If True, reverse the byte order within each 32-bit word of the payload. This is used for handling endianness in systems where the byte order needs to be swapped. Default is False.

  • dummy_words (int, optional) – Number of 32-bit dummy words in the header. This is used to stabilize clock recovery in FPGA Manchester decoder. This value does not have a meaning for image recovery.

  • ..todo:: – Move port (for USART) to a user config area. This should make this pure device config.

adc_scale: ADCScaling | None
baudrate: int | None
bitstream: Path | None
block_size: int
buffer_block_length: int
property buffer_npix: list[int]

List of pixel counts per buffer for a complete frame.

A frame is split across multiple buffers. This returns a list where each element is the number of pixels in that buffer. The last buffer may have fewer pixels (the remainder).

device: Literal['OK', 'UART']
dummy_words: int
classmethod ensure_exists(value: Path | None) → Path | None

If a bitstream file has been provided, ensure it exists

frame_height: int
frame_width: int
fs: int
header_len: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context: Any, /) → None

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self – The BaseModel instance.

  • context – The context.

num_buffers: int
pix_depth: int
port: str | None
preamble: bytes
classmethod preamble_to_bytes(value: str | bytes | int) → bytes

Cast preamble to bytes.

Parameters:

value (str, bytes, int) – Recast from str (in yaml like preamble: "0x12345" ) or int (in yaml like preamble: 0x12345

Returns:

bytes

property px_per_buffer: int

Number of pixels per buffer

classmethod resolve_relative(value: Path) → Path

If we are given a relative path to a bitstream, resolve it relative to the device path

reverse_header_bits: bool
reverse_header_bytes: bool
reverse_payload_bits: bool
reverse_payload_bytes: bool
runtime: StreamDevRuntime
class mio.models.stream.StreamDevRuntime(*, serial_buffer_queue_size: int = 10, frame_buffer_queue_size: int = 5, image_buffer_queue_size: int = 5, queue_put_timeout: int = 5, plot: StreamPlotterConfig | None = StreamPlotterConfig(keys=['timestamp', 'buffer_count', 'frame_buffer_count'], update_ms=1000, history=500), csvwriter: CSVWriterConfig | None = CSVWriterConfig(buffer=100), ntp_server: str | None = None, ntp_max_offset_seconds: float = 0.01)

Runtime configuration for StreamDevice

Included within StreamDevConfig to separate config that is not unique to the device, but how that device is controlled at runtime.

csvwriter: CSVWriterConfig | None
frame_buffer_queue_size: int
image_buffer_queue_size: int
model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ntp_max_offset_seconds: float
ntp_server: str | None
plot: StreamPlotterConfig | None
queue_put_timeout: int
serial_buffer_queue_size: int