Storing timeseries data

Xively offers a data persistence layer on top of the broker's messaging service. This allows data published by devices to be stored for later retrieval. Time series data can be used for purposes like creating visualizations of historical values or for analytics.

Time series data published by devices to Xively using the MQTT protocol are retrievable via timeseries API endpoints. The timeseries service resides at timeseries.xively.com.

Setting an MQTT channel to "timeseries"

To persist timeseries data in the Xively database, ensure the following:

  • The data is sent on an MQTT chanel that has been set as a timeSeries channel
  • The payload of MQTT messages is formatted in one of the accepted input formats below (CSV or JSON)

As all timeSeries channels are standard MQTT channels, the publishing process is exactly the same as in other scenarios. However, in addition to usual message delivery, the properly-formatted data sent to timeSeries channels will also be stored for later retrieval.

📘

Normal MQTT rules apply

It is possible to publish messages of any format to a timeSeries channel, but only properly formatted data are stored for later retrieval.

Other types of payloads are still delivered as usual, but they are not stored.

👍

The Xively embedded client auto-formats for time series

Tip: If your device is using Xively's embedded IoT Client, or any other Xively client, SenML formatting can be automatically applied to time series messages.

❗️

Duplicate timestamps

Messages that are sent with identical timestamps will NOT overwrite each other - they will all be stored as separate datapoints, using the same timestamp.

Accepted input formats

There are 2 accepted formats in which messages can be published, in order to store their values.

  1. CSV
  2. JSON, formatted in SenML (useful for publishing multiple datapoints in a single payload)

Input format 1: CSV

CSV structure

Datapoints can be stored so long as they are published in the following format:

g

Field

Description

Notes

Timestamp

Timestamp associated with the value to be stored.
Supported formats: the ISO 8601 date (yyyy-MM-ddTHH:mm:ssZ) or number of milliseconds since Unix epoch

Optional
If left empty, filled by Xively Message Broker with the timestamp of the message arrival

Category

Arbritrary category the value belongs to

Required

Value

Value to be stored

Optional

String

Arbritrary string associated with value to be stored

Optional

Timestamp (ISO 8601 format or ms since Unix epoch),Category,Value,String

// Send a float datapoint of the category "Voltage"
,Voltage,1.25,

// Send a string datapoint of the category "Display" with an enforced timestamp
2015-11-04T15:55:46.5429603Z,Display,,Normal
1446652546000,Display,,Normal

// Send both a float and string datapoint of the category "Display"
,Display,1.25,Normal

Input format 2: JSON (SenML)

SenML (Sensor Markup Language) is a low-footprint format for representing simple sensor measurements. For the complete SenML specification, refer to the SenML memo.

Each representation carries a single SenML object that consist of the following elements:

  • Root variables: Base attributes of the SenML object. Each attribute type must be unique.
  • Measurement parameters: A mandatory array of one or more entries. May contain multiple entries of each entry type.
  • Sensor Names: A sensor name consist of the Base Name and the Name value supplied in the parameter array. The resulting sensor name must uniquely identify and differentiate the sensor from all others.

Note: The sensor name must consist only of characters out of the set "A" to "Z", "a" to "z", "0" to "9", "-", ":", ".", or "_" and it MUST start with a character out of the set "A" to "Z", "a" to "z", or "0" to "9".

Supported Units
For the complete unit registry, refer to section 10.1 of the SenML memo.

SenML root variables

Attribute

Description

Type

Notes

bn

The Base Name is a string that is prepended to the names found in the entries.

string

Optional

  • The Name value is concatenated to the Base Name value to get the name of the sensor.
  • If the object is a representation resulting from the request of a URI, then in the absence of the Base Name attribute, this URI is used as the default value of Base Name.

bt

A Base Time that is added to the time found in an entry.

number

Optional
Defaults to 0 if not present.
Date and time values are measured in seconds since the epoch.

bu

A Base Unit that is assumed for all entries, unless otherwise indicated.

number

Optional
Assumed for all entries, unless otherwise indicated in the parameter array.

ver

Version number of media type format.

positive integer

Optional
Defaults to 1 if not present.

e

A parameter Array that carries the payload for device data.

array

Required
Important:

  • Objects must contain exactly one e attribute.
  • The supplied parameter array must contain at least one measurement or parameter.

The "e" parameter array

Parameter

Description

Type

Notes

n

Name of the sensor or parameter.

string

Required

  • A sensor name must uniquely identify and differentiate the sensor from all others.
  • The Name value is concatenated to the Base Name value to get the name of the sensor.

u

Units for a measurement value.

string

Optional
If not supplied, the Base Unit is assumed.

v

Value of the entry.

floating point

Optional (see note below)

sv

String Value of the entry.

string

Optional (see note below)

bv

Boolean Value of the entry.

boolean

Optional (see note below)

s

Integrated Sum of the values over time.

floating point

Optional

t

Time when value was recorded.

The ISO 8601 formatted date (yyyy-MM-ddTHH:mm:ssZ)

Optional
Defaults to 0 if not present.

ut

A time in seconds that represents the maximum time before the sensor will provide an updated reading for a measurement.

The ISO 8601 formatted date (yyyy-MM-ddTHH:mm:ssZ)

Optional
Defaults to 0 if not present.

🚧

Value types

Important: Only one parameter representing a value can be supplied for each SenML payload: v, sv or bv.

{"e":[{ "n": "urn:dev:ow:10e2073a01080063", "v":23.5 }]}
{"e":[
        { "n": "voltage", "u": "V", "v": 120.1 },
        { "n": "current", "t": -5, "v": 1.2 },
        { "n": "current", "t": -4, "v": 1.30 },
        { "n": "current", "t": -3, "v": 0.14e1 },
        { "n": "current", "t": -2, "v": 1.5 },
        { "n": "current", "t": -1, "v": 1.6 },
        { "n": "current", "t": 0,   "v": 1.7 }],
    "bn": "urn:dev:mac:0024befffe804ff1/",
    "bt": 1480871899,
    "ver": 1,
    "bu": "A"
   }

Updated 3 years ago

Storing timeseries data


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.