Multimodal Time-Series Data Management

In TimechoDB's Table Model, OBJECT is an advanced data type for managing multimodal time-series data. It associates the original content of unstructured objects such as images, audio/video, and documents with timestamps, tags, and ordinary time-series fields in the same record, so operational context and multimedia evidence can be written, queried, and managed together.

1. OBJECT Data Type

1.1 Definition

OBJECT is a large-object data type in the Table Model, supported since V2.0.8. The original binary content is stored in separate files. Its write amplification factor is always 1, with low space amplification, and it uses merge & release on write.

OBJECT is suitable for larger object files that need to be uploaded in segments. For small numbers, short text, enumerations, or values requiring frequent expression evaluation, use ordinary scalar types.

For details about the differences between OBJECT and BLOB, see Data Type.

1.2 Type Compatibility

Column type	Supported input type
OBJECT	OBJECT

An OBJECT column accepts only OBJECT values. It does not accept STRING, TEXT, or BLOB values directly. Incompatible values cause an error. SQL writes must construct an OBJECT value with to_object(...); clients must use the corresponding OBJECT write API.

1.3 Storage and Access

The original binary content of an OBJECT is stored under ${data_dir}/object. To avoid oversized requests, an object can be split into segments and written in order.

A direct query of an OBJECT column returns a size summary such as (Object) 5 B. Use READ_OBJECT to retrieve the original content; it returns BLOB. An offset and length can be specified to read the content in segments.

1.4 Related Configuration

The following settings apply to OBJECT. restrict_object_limit can only be changed before the service is started for the first time.

Setting	Description	Type	Default	Applies
max_object_file_size_in_byte	Maximum size of one object file, in bytes; supported since V2.0.8.1	long	4294967296 (4 GiB)	Hot reload
restrict_object_limit	Whether to restrict filesystem paths derived from OBJECT-related names; supported since V2.0.8.1	boolean	false	Before first service startup only

Choose a production value for max_object_file_size_in_byte based on object size, disk I/O, network transfer, backup window, and recovery objectives.

When restrict_object_limit=true and a table contains an OBJECT column, note the following:

1. Values in TAG columns, table names, and field names must not be . or .., and must not contain ./ or ./-style path traversal sequences (including ./ and ../); otherwise metadata creation fails. Names containing characters unsupported by the filesystem cause data writes to fail.
2. On a case-insensitive filesystem, device identifiers such as d1 and D1 may map to the same file path, causing object files to overwrite each other and corrupting data.
3. Object file paths use this format:

${dataregionid}/${tablename}/${tag1}/${tag2}/.../${field}/${timestamp}.bin

1.5 Failed Deletion Audit

FAILED_TTL_DELETION, supported since V2.0.11.1, records failed TTL deletion events. It is recorded only when a failure occurs.

The default whitelist for auditable_control_event_type includes FAILED_TTL_DELETION. To record this event, enable security auditing and ensure that the item remains on the whitelist. Removing it from the whitelist disables this event. Security auditing is disabled by default (enable_audit_log=false).

Failure scenario	Trigger	Log format	Description	Number of logs
Object file scan failure	An exception such as disk I/O error or insufficient permissions occurs while scanning and deleting expired Object files.	TTL object scan failed: errors=[ClassName1 xN1(msg1),ClassName2 xN2(msg2)]	Exceptions during one table scan are aggregated by type, with the count and first message retained.	One per affected table per scan.
TTL check startup failure	An exception occurs before per-table checks begin, or Settle Compaction fails while selecting tasks.	TTL check failed to start: {msg} error=ClassName(exceptionMsg)	Records the failed stage and cause; no table is recorded because the tables have not yet been selected.	One per failed check startup.
Settle Compaction execution failure	Settle Compaction completes with isSuccess=false.	TTL settle compaction failed: error=...	Records the cause and TTL tables involved, collected by the Selector while scanning devices.	One per involved TTL table per failed execution.

Table Model audit logs are stored in __audit.audit_log. For general audit settings and queries, see Security Audit.

2. Data Modeling

2.1 Preparation

Connect using a Table Model session to run the examples. The user needs the appropriate permissions to create databases and tables, write data, and query it; see Authority Management.

The SQL examples use database database1, table table1, and OBJECT measurement column s1. Create these in a test environment and run the examples in order. For client connection methods, see CLI.

2.2 Create a Table with an OBJECT Column

In the Table Model, TAG columns identify a device and FIELD columns hold time-varying measurements. Define an OBJECT column as a FIELD. TAG and ATTRIBUTE columns use STRING, and the TIME column uses TIMESTAMP.

CREATE DATABASE database1;
USE database1;

CREATE TABLE table1 (
  time TIMESTAMP TIME,
  region STRING TAG,
  plant_id STRING TAG,
  device_id STRING TAG,
  model_id STRING ATTRIBUTE,
  maintenance STRING ATTRIBUTE,
  temperature FLOAT FIELD,
  humidity FLOAT FIELD,
  status Boolean FIELD,
  arrival_time TIMESTAMP FIELD,
  s1 OBJECT FIELD COMMENT 'object type'
);

This table follows the table1 structure in Sample Data, with s1 added for object content. The write examples use 'Beijing', '1001', and 'tag1' for the region, plant_id, and device_id TAG columns.

Data retention: If no TTL is specified when creating the table, the database TTL applies. A table can also have its own TTL. TTL expiration is determined by data-point timestamps; expired data cannot be queried, and expired OBJECT files are scanned and deleted by the system. See Data Retention.

2.3 Add an OBJECT Column to an Existing Table

Use ALTER TABLE to add an OBJECT measurement column to an existing table:

ALTER TABLE table1
ADD COLUMN IF NOT EXISTS s1 OBJECT FIELD COMMENT 'object type';

For the full description of table structure and column categories, see Table Management.

3. Writing Data

3.1 Write Function

The to_object function has been supported since V2.0.8 to construct an OBJECT value for writing. Split large objects into segments to keep individual requests manageable.

Syntax:

to_object(isEOF, offset, content)

Parameter	Type	Description
isEOF	BOOLEAN	Whether this is the final segment of the object
offset	INT64	Starting byte offset of this segment in the complete object
content	Hexadecimal	Binary content represented as X'...'

General form:

INSERT INTO tableName(time, columnName)
VALUES (timeValue, to_object(isEOF, offset, content));

3.2 Write in One Segment

Write device tags and object content to the same record:

INSERT INTO table1(time, region, plant_id, device_id, s1)
VALUES (now(), 'Beijing', '1001', 'tag1', to_object(true, 0, X'696F746462'));

true marks the final segment and 0 is the starting offset. The example payload 0x696f746462 contains 5 bytes and demonstrates binary read/write.

3.3 Write in Multiple Segments

Split the same content into 696F, 7464, and 62, and write it at timestamp 1:

-- First segment: write 2 bytes; the object is not complete.
INSERT INTO table1(time, region, plant_id, device_id, s1)
VALUES (1, 'Beijing', '1001', 'tag1', to_object(false, 0, X'696F'));

-- Second segment: continue at offset 2 and write 2 bytes.
INSERT INTO table1(time, region, plant_id, device_id, s1)
VALUES (1, 'Beijing', '1001', 'tag1', to_object(false, 2, X'7464'));

-- Final segment: continue at offset 4 and write the last byte.
INSERT INTO table1(time, region, plant_id, device_id, s1)
VALUES (1, 'Beijing', '1001', 'tag1', to_object(true, 4, X'62'));

All segments use the same timestamp, TAG values, and OBJECT column. The completed content is 0x696f746462.

Notes:

1. A partially written object is returned as null; it becomes queryable after all segments have been written.
2. The offset for a write must equal the number of bytes already written, or the write fails.
3. After partial data has been written, writing again with offset=0 clears the earlier partial content and starts over. Use the correct offset to resume.
4. On Windows, explicitly provide every TAG value when writing OBJECT data. Otherwise, creating a directory containing NUL may fail with exception 741.
5. INSERT INTO QUERY does not currently support Object values.

For other write and update syntax, see Write & Update.

4. Querying Data

4.1 Query an Object Summary

The following query uses the object written in section 3.3. A direct query of an OBJECT column returns a size summary:

SELECT s1
FROM table1
WHERE device_id = 'tag1' AND time = 1;

+------------+
|          s1|
+------------+
|(Object) 5 B|
+------------+

4.2 Read the Original Content

READ_OBJECT, supported since V2.0.8, reads the binary content of an OBJECT and returns BLOB.

Syntax:

READ_OBJECT(object [, offset, length])

Parameter	Type	Description
object	OBJECT	Required object to read
offset	INT64	Optional read offset; defaults to 0. A value less than 0 or greater than or equal to the complete object length causes an exception.
length	INT64	Optional read length; defaults to the complete object length. Values greater than 2^31 - 1 cause an error. If the requested length exceeds the remaining content, the remaining content is returned. A negative value reads all remaining content from the offset.

Read the complete content:

SELECT READ_OBJECT(s1)
FROM table1
WHERE device_id = 'tag1' AND time = 1;

+------------+
|       _col0|
+------------+
|0x696f746462|
+------------+

Read the first 3 bytes:

SELECT READ_OBJECT(s1, 0, 3)
FROM table1
WHERE device_id = 'tag1' AND time = 1;

+--------+
|   _col0|
+--------+
|0x696f74|
+--------+

The complete result should be 0x696f746462, matching the segmented write in section 3.3. For large objects, read in segments with offset and length, observing the per-read length limit.

4.3 Length, Type Conversion, and Null Filling

Get object length

Since V2.0.9.1, the length function returns the number of bytes in an OBJECT as INT32.

length(input)

Type conversion

Since V2.0.8.1, CAST can convert OBJECT to STRING:

SELECT CAST(s1 AS STRING)
FROM table1;

Null filling

Fill method	Supported
PREVIOUS	Yes; uses the previous non-null value in the column
NEXT	Supported since V2.0.11.1; uses the next non-null value in the column
LINEAR	No
CONSTANT	No

If a fill method does not support OBJECT, the OBJECT column remains unchanged and no exception is thrown. For fill parameters and examples, see FILL Clause.

5. Native APIs

5.1 Java Native API

The Java Table Model writes Tablet data through ITableSession.insert(Tablet tablet). Since V2.0.8, this interface supports segmented OBJECT writes. Use the following method to add an object segment to an OBJECT column in a Tablet:

void addValue(int rowIndex, int columnIndex, boolean isEOF, long offset, byte[] content)

Parameter	Description
rowIndex	Row index in the Tablet
columnIndex	Column index in the Tablet
isEOF	Whether this is the final object segment
offset	Starting offset of this segment in the complete object
content	Bytes in this segment

The total length of all byte[] segments must equal the original object size, or the written data size will be incorrect. Submit segments in order. The meanings of offset and isEOF are the same as for SQL writes.

Example write call

Add the object content to the Tablet's OBJECT column, then call insert to submit it:

Tablet tablet = new Tablet(...);
tablet.addValue(rowIndex, objectColumnIndex, true, 0, content);
session.insert(tablet);

Here, rowIndex, objectColumnIndex, and content are the row index, OBJECT column index, and bytes to write. true, 0 writes the entire object from the beginning in one segment. Constructor arguments such as the table schema are omitted.

A direct query of an OBJECT column returns a summary string such as (Object) XX.XX KB from Field.getStringValue, Field.getObjectValue, SessionDataSet.DataIterator.getObject, and SessionDataSet.DataIterator.getString.

For client dependencies, session creation, and reading result sets, see Java Native API.

5.2 Python Native API

Since V2.0.8.3, the Python client supports TSDataType.OBJECT in Tablet batch writes and Session value serialization.

5.2.1 OBJECT APIs

API	Parameters	Description and return value
encode_object_cell	is_eof: bool, offset: int, content: bytes	Encodes one OBJECT cell in wire format and returns bytes.
decode_object_cell	cell: bytes, at least 9 bytes long	Parses an encoded OBJECT cell and returns (is_eof, offset, payload).
Tablet.add_value_object	row_index: int, column_index: int, is_eof: bool, offset: int, content: bytes	Adds an OBJECT segment at the specified row and column; returns None.
Tablet.add_value_object_by_name	column_name: str, row_index: int, is_eof: bool, offset: int, content: bytes	Adds an OBJECT segment by column name; returns None.
NumpyTablet.add_value_object	Same as Tablet.add_value_object	Adds a segment to an OBJECT column in a NumpyTablet; column data is an ndarray; returns None.
Field.get_object_value	data_type: TSDataType	For OBJECT, decodes the query value as UTF-8 and returns str.
Field.get_string_value	None	Decodes an OBJECT query value as UTF-8 and returns a string.
Field.get_binary_value	None	Gets binary data for TEXT, STRING, or BLOB and returns bytes or None; calling it directly on an OBJECT column raises an error.

A direct query of an OBJECT column returns its size summary. To read the original bytes, query with READ_OBJECT and use Field.get_binary_value() to get the bytes from the resulting BLOB.

5.2.2 OBJECT Cell Encoding

The wire format of an OBJECT cell is:

|[eof 1B]|[offset 8B BE]|[payload]|

• eof occupies 1 byte and indicates whether this is the final object segment.
• offset occupies 8 bytes in big-endian order and gives the starting offset of this segment in the object.
• payload contains the bytes in this segment.

encode_object_cell(is_eof, offset, content) creates this encoding. decode_object_cell(raw_cell) parses it and returns the EOF flag, offset, and payload. An encoded cell must be at least 9 bytes long.

5.2.3 Write and Read Examples

Use the Python client to create Tablet rows and write an OBJECT in one or multiple segments. For a one-segment write, set is_eof=True and offset=0. For segmented writes, use is_eof=False for intermediate segments and True for the final segment; each offset must equal the number of bytes already written. Query the value with READ_OBJECT to retrieve its bytes as BLOB.

# One-segment object write: final segment at offset zero.
# Use the OBJECT helpers shown above to populate the OBJECT column,
# then submit the Tablet with the Table Session insert API.

# Segmented OBJECT write: submit intermediate segment(s), then the final segment.
# Each segment must use the next expected byte offset.

5.3 C and C++ Native APIs

SessionC does not currently provide ts_tablet_add_value_object; use the C++ Session API to write OBJECT data. For C++ sessions and the general Tablet API, see C++ Native API.

6. Importing and Exporting Data

6.1 Import TsFiles Containing OBJECT Data

Since V2.0.9.2, Table Model import-data supports OBJECT data in TsFiles. Prepare both the TsFile and its associated object files before import.

If no object directory is specified, the tool looks for object files in a subdirectory with the same name as the TsFile:

/data/import/
├── sensor_v1.tsfile
└── sensor_v1/
    ├── regionID/tableName/tag1/tag2/field/timestamp1.bin
    └── regionID/tableName/tag1/tag2/field/timestamp2.bin

Run the following command from the installation directory to import into database1. The example explicitly sets the post-import handling of source files to none, so source files are neither deleted nor moved after success or failure:

tools/import-data.sh -sql_dialect table -ft tsfile \
  -s /data/import/sensor_v1.tsfile -db database1 -os none -of none

If object files are in an external directory, specify their root with -o (--object-file-paths), supported since V2.0.9.2:

tools/import-data.sh -sql_dialect table -ft tsfile \
  -s /data/import/sensor_v1.tsfile -db database1 \
  -o /mnt/object_storage/ -os none -of none

Under /mnt/object_storage/, prepare the associated files in the regionID/tableName/tag1/tag2/field/timestamp.bin hierarchy. On Windows, use tools\windows\import-data.bat. For all options, see Data Import.

6.2 Export with TsFileBackup

Since V2.0.9.2, tsfile-backup.sh/bat can export OBJECT data as TsFiles with an associated object directory. The tool uses the Pipe framework to generate and send CREATE PIPE SQL.

Before use, obtain tsfile-remote-sink-<version>-jar-with-dependencies.jar from the Timecho team and place it at a path accessible to the server, such as on every DataNode host.

The following example uses SCP to export database1.table1 to a remote server. Run it from the installation directory and replace the host, user, and destination with actual values. If -tpw has no value, enter the remote password when prompted:

tools/tsfile-backup.sh -sql_dialect table \
  -db database1 -table table1 -t /remote/archive/ \
  -th 192.168.1.100 -tu backup_user -tpw

On Windows, use tools\windows\tsfile-backup.bat. OBJECT-related options include:

Option	Description
--plugin_jar	Path to the Pipe plugin JAR
--rate_limit	Sending rate limit in bytes per second
--object-parallelism	Maximum parallelism for sending OBJECT files
--object-batch-size	Total bytes in each object-file upload batch, to control memory use and the size of each SCP transfer

To specify the plugin JAR, add --plugin_jar /local/lib/tsfile-remote-sink-<version>-jar-with-dependencies.jar to the command.

For SCP handshake errors, connection failures, or frequent Pipe restarts, reduce --object-parallelism as needed, or increase MaxStartups in the target host's SSH configuration. Apply the change with sshd reload or sshd restart. For database connection, time range, and other general options, see Data Export.

6.3 Export with Pipe

Since V2.0.9.2, Table Model Pipe supports two OBJECT export methods:

Method	Description
Local export	Uses the built-in tsfile-local-sink to export to the local filesystem of the database server.
Remote transfer	Registers the tsfile-remote-sink JAR as a plugin and transfers files to a remote server over SSH/SCP.

The following example exports database1.table1 to a local directory on the server. The export directory contains the TsFile and its same-named object subdirectory, as shown in section 6.1. The rate limit is 10485760 bytes per second.

CREATE PIPE tsfile_export_local
WITH SOURCE (
  'source' = 'iotdb-source',
  'database-name' = 'database1',
  'table-name' = 'table1'
)
WITH PROCESSOR (
  'processor' = 'do-nothing-processor'
)
WITH SINK (
  'sink' = 'tsfile-local-sink',
  'sink.local.target-path' = '/data/backup/export_2024',
  'sink.rate-limit-bytes-per-second' = '10485760'
);

Check the task status:

SHOW PIPE tsfile_export_local;

Remote SCP export

Obtain tsfile-remote-sink-<version>-jar-with-dependencies.jar from the Timecho team and place it at a path accessible to the server, such as on every DataNode host. Register the plugin:

CREATE PIPEPLUGIN tsfile_remote_sink
AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink'
USING URI 'file:///path/to/tsfile-remote-sink-<version>-jar-with-dependencies.jar';

Then create the remote export task, replacing the plugin path, host, user, password, and directory with actual values. The Pipe sink uses the plugin name specified at registration, tsfile_remote_sink.

CREATE PIPE tsfile_export_scp
WITH SOURCE (
  'source' = 'iotdb-source',
  'database-name' = 'database1',
  'table-name' = 'table1'
)
WITH PROCESSOR (
  'processor' = 'do-nothing-processor'
)
WITH SINK (
  'sink' = 'tsfile_remote_sink',
  'sink.file-mode' = 'scp',
  'sink.scp.host' = '192.168.1.100',
  'sink.scp.port' = '22',
  'sink.scp.user' = 'backup_user',
  'sink.scp.password' = '<remote-host-password>',
  'sink.scp.remote-path' = '/remote/archive/',
  'sink.rate-limit-bytes-per-second' = '10485760'
);

Remote transfer parameters:

Parameter	Description	Default
sink.scp.host	Remote host IP; required	None
sink.scp.port	Remote SSH port	22
sink.scp.user	Remote SSH user; required	None
sink.scp.password	Remote SSH password; required	None
sink.scp.remote-path	Remote destination directory; required	None
sink.rate-limit-bytes-per-second	Transfer rate limit in bytes per second; values less than or equal to 0 mean unlimited	0
sink.scp.object-parallelism	Maximum parallelism for sending OBJECT files	min(cpu/4, 16)
sink.scp.object-batch-size-bytes	Maximum size of OBJECT files sent by one asynchronous thread; the parameter description specifies MB as the unit	200

The name sink.scp.object-batch-size-bytes and its documented unit do not match; the actual unit needs verification. This page does not set this parameter.

Task management

-- Show all tasks
SHOW PIPES;

-- Show a task
SHOW PIPE tsfile_export_scp;

-- Stop and start a task
STOP PIPE tsfile_export_scp;
START PIPE tsfile_export_scp;

-- Drop a task
DROP PIPE IF EXISTS tsfile_export_scp;

To modify task parameters, use:

ALTER PIPE [IF EXISTS] <PipeId>
MODIFY SOURCE (...);

For other Pipe source, processor, and sink options and task management, see Data Sync and SQL Data Sync.

7. Frequently Asked Questions

• Why does a direct query show (Object) 5 B instead of binary data? A direct query returns the object size summary. Use READ_OBJECT to retrieve its content as BLOB.
• Why is a partially written value null? An object is visible only after the final segment is written with isEOF=true.
• Why does a segmented write fail? Check that all segments use the same timestamp, TAG values, and OBJECT column, and that each offset equals the number of bytes already written.
• Why does writing a STRING or BLOB to an OBJECT column fail? The column accepts only OBJECT values. Construct the value using to_object(...) or the client OBJECT API.
• Why are object files missing after TTL expiration? Expired object files are removed by background scanning. If deletion fails, enable security auditing and check FAILED_TTL_DELETION events in __audit.audit_log.