Query data

You can use the data management service to capture sensor data from any machine and sync that data to the cloud. Then, you can follow the steps on this page to query it using SQL or MQL. For example, you can configure data capture for several sensors on one machine, or for several sensors across multiple machines, to report the ambient operating temperature. You can then run queries against that data to search for outliers or edge cases, to analyze how the ambient temperature affects your machines’ operation.

Query using Viam

Prerequisites

Captured sensor data
Follow the guide to capture sensor data.

Query data using Viam

Once your data has synced, you can query your data using SQL or MQL.

You must have the owner role in order to query data on Viam.

1

Query with SQL or MQL

Navigate to the Query page. Then, select either SQL or MQL from the Query mode dropdown menu on the right-hand side.

2

Run your query

This example query returns the last 5 readings from any components named my-sensor in your organization:

SELECT * FROM readings
WHERE component_name = 'sensor-1' LIMIT 5
[
    { "$match": { "component_name": "sensor-1" } },
    { "$limit": 5 }
]
Click to see an example that filters by component name and column names.
SELECT time_received, data, tags FROM readings
WHERE component_name = 'sensor-1' LIMIT 2
[
{
  "time_received": "2024-07-30 00:04:02.144 +0000 UTC",
  "data": {
    "readings": {
      "units": "μg/m³",
      "pm_10": 7.6,
      "pm_2.5": 5.7
    }
  },
  "tags": [
    "air-quality"
  ]
},
{
  "time_received": "2024-07-30 00:37:22.192 +0000 UTC",
  "data": {
    "readings": {
      "pm_2.5": 9.3,
      "units": "μg/m³",
      "pm_10": 11.5
    }
  },
  "tags": [
    "air-quality"
  ]
}
]
[
  {
    $match: {
      component_name: "sensor-1"
    }
  },{
    $limit: 2
  }, {
    $project: {
        time_received: 1,
        data: 1,
        tags: 1
    }
  }
]
[
{
  "time_received": "2024-07-30 00:04:02.144 +0000 UTC",
  "data": {
    "readings": {
      "units": "μg/m³",
      "pm_10": 7.6,
      "pm_2.5": 5.7
    }
  },
  "tags": [
    "air-quality"
  ]
},
{
  "time_received": "2024-07-30 00:37:22.192 +0000 UTC",
  "data": {
    "readings": {
      "pm_2.5": 9.3,
      "units": "μg/m³",
      "pm_10": 11.5
    }
  },
  "tags": [
    "air-quality"
  ]
}
]
Click to see an example that returns a count of records that match a component name from a specific location.
SELECT count(*) FROM readings
WHERE component_name = 'sensor-1'
[
  {
    "_1": 111550
  }
]
[
  {
    $match: {
      component_name: "sensor-1"
    }
  },{
    $count: "count"
  }
]
{ count: 111550 }
3

Review results

Click Run query when ready to perform your query and get matching results. You can view your query results in a table or as a JSON array below your query.

import asyncio

from viam.rpc.dial import DialOptions, Credentials
from viam.app.viam_client import ViamClient


# Configuration constants – replace with your actual values
API_KEY = ""  # API key, find or create in your organization settings
API_KEY_ID = ""  # API key ID, find or create in your organization settings
ORG_ID = ""  # Organization ID, find or create in your organization settings


async def connect() -> ViamClient:
    """Establish a connection to the Viam client using API credentials."""
    dial_options = DialOptions(
        credentials=Credentials(
            type="api-key",
            payload=API_KEY,
        ),
        auth_entity=API_KEY_ID
    )
    return await ViamClient.create_from_dial_options(dial_options)


async def main() -> int:
    viam_client = await connect()
    data_client = viam_client.data_client

    tabular_data_mql = await data_client.tabular_data_by_mql(
        organization_id=ORG_ID,
        query=[
            {
                "$match": {
                    "component_name": "sensor-1"
                },
            }, {
                "$limit": 5
            }
        ]
    )
    print(f"Tabular Data: {tabular_data_mql}")

    tabular_data_sql = await data_client.tabular_data_by_sql(
        organization_id=ORG_ID,
        sql_query="SELECT * FROM readings WHERE component_name = 'sensor-1' LIMIT 5"
    )
    print(f"Tabular Data: {tabular_data_sql}")

    viam_client.close()
    return 0

if __name__ == "__main__":
    asyncio.run(main())
Click to see an example that filters by component name and column names.
tabular_data_mql_filter = await data_client.tabular_data_by_mql(
    organization_id=ORG_ID,
    query=[
        {
            "$match": {
                "component_name": "sensor-1"
            },
        }, {
            "$limit": 2
        }, {
            "$project": {
                "time_received": 1,
                "data": 1,
                "tags": 1
            }
        }
    ]
)
print(f"Tabular Data: {tabular_data_mql_filter}")

tabular_data_sql_filter = await data_client.tabular_data_by_sql( organization_id=ORG_ID, sql_query="SELECT time_received, data, tags FROM readings " "WHERE component_name = 'sensor-1' LIMIT 2" ) print(f"Tabular Data: {tabular_data_sql_filter}")

Click to see an example that returns a count of records that match a component name from a specific location.
tabular_data_mql_count = await data_client.tabular_data_by_mql(
    organization_id=ORG_ID,
    query=[
        {
            "$match": {
                "component_name": "sensor-1"
            },
        }, {
            "$count": "count"
        }
    ]
)
print(f"Tabular Data: {tabular_data_mql_count}")

tabular_data_sql_count = await data_client.tabular_data_by_sql( organization_id=ORG_ID, sql_query="SELECT count(*) FROM readings " "WHERE component_name = 'sensor-1'" ) print(f"Tabular Data: {tabular_data_sql_count}")

package main

import (
	"context"
	"fmt"

	"go.viam.com/rdk/app"
	"go.viam.com/rdk/logging"
)

func main() {
	apiKey := ""
	apiKeyID := ""
	orgID := ""

	logger := logging.NewDebugLogger("client")
	ctx := context.Background()
	viamClient, err := app.CreateViamClientWithAPIKey(
		ctx, app.Options{}, apiKey, apiKeyID, logger)
	if err != nil {
		logger.Fatal(err)
	}
	defer viamClient.Close()

	dataClient := viamClient.DataClient()

	// Create MQL stages as map slices
	mqlStages := []map[string]interface{}{
		{"$match": map[string]interface{}{"component_name": "sensor-1"}},
		{"$limit": 5},
	}

	tabularDataMQL, err := dataClient.TabularDataByMQL(ctx, orgID, mqlStages, &app.TabularDataByMQLOptions{})
	if err != nil {
		logger.Fatal(err)
	}

	fmt.Printf("Tabular Data: %v\n", tabularDataMQL)

	tabularDataSQL, err := dataClient.TabularDataBySQL(ctx, orgID, "SELECT * FROM readings WHERE component_name = 'sensor-1' LIMIT 5")
	if err != nil {
		logger.Fatal(err)
	}

	fmt.Printf("Tabular Data: %v\n", tabularDataSQL)
}
Click to see an example that filters by component name and column names.
// Create MQL stages as map slices
mqlStages := []map[string]interface{}{
	{"$match": map[string]interface{}{"component_name": "sensor-1"}},
	{"$limit": 2},
	{"$project": map[string]interface{}{
		"time_received": 1,
		"data": 1,
		"tags": 1,
	}},
}

tabularDataMQL, err := dataClient.TabularDataByMQL(ctx, orgID, mqlStages, &app.TabularDataByMQLOptions{}) if err != nil { logger.Fatal(err) }

fmt.Printf("Tabular Data: %v\n", tabularDataMQL)

tabularDataSQL, err := dataClient.TabularDataBySQL(ctx, orgID, "SELECT time_received, data, tags FROM readings " + "WHERE component_name = 'sensor-1' LIMIT 2") if err != nil { logger.Fatal(err) }

fmt.Printf("Tabular Data: %v\n", tabularDataSQL)

Click to see an example that returns a count of records that match a component name from a specific location.
mqlStages = []map[string]interface{}{
	{"$match": map[string]interface{}{"component_name": "sensor-1"}},
	{"$count": "count"},
}

tabularDataMQLCount, err := dataClient.TabularDataByMQL(ctx, orgID, mqlStages, &app.TabularDataByMQLOptions{}) if err != nil { logger.Fatal(err) }

fmt.Printf("Tabular Data: %v\n", tabularDataMQLCount)

tabularDataSQLCount, err := dataClient.TabularDataBySQL(ctx, orgID, "SELECT count(*) FROM readings WHERE component_name = 'sensor-1'") if err != nil { logger.Fatal(err) }

fmt.Printf("Tabular Data: %v\n", tabularDataSQLCount)

import { createViamClient } from "@viamrobotics/sdk";

// Configuration constants – replace with your actual values
let API_KEY = "";  // API key, find or create in your organization settings
let API_KEY_ID = "";  // API key ID, find or create in your organization settings
let ORG_ID = "";  // Organization ID, find or create in your organization settings

async function main(): Promise<void> {
    // Create Viam client
    const client = await createViamClient({
        credentials: {
            type: "api-key",
            authEntity: API_KEY_ID,
            payload: API_KEY,
        },
    });

    const tabularDataMQL = await client.dataClient.tabularDataByMQL(
        ORG_ID,
        [
            { "$match": { "component_name": "sensor-1" } },
            { "$limit": 5 }
        ],
        false
    );
    console.log(tabularDataMQL);

    const tabularDataSQL = await client.dataClient.tabularDataBySQL(
        ORG_ID,
        "SELECT * FROM readings WHERE component_name = 'sensor-1' LIMIT 5"
    );
    console.log(tabularDataSQL);
}

// Run the script
main().catch((error) => {
    console.error("Script failed:", error);
    process.exit(1);
});
Click to see an example that filters by component name and column names.
const tabularDataMQLFilter = await client.dataClient.tabularDataByMQL(
    ORG_ID,
    [
        { "$match": { "component_name": "sensor-1" } },
        { "$limit": 2 },
        { "$project": {
            "time_received": 1,
            "data": 1,
            "tags": 1
        }}
    ]
);
console.log(tabularDataMQLFilter);

const tabularDataSQLFilter = await client.dataClient.tabularDataBySQL( ORG_ID, "SELECT time_received, data, tags FROM readings " + "WHERE component_name = 'sensor-1' LIMIT 2" ); console.log(tabularDataSQLFilter);

Click to see an example that returns a count of records that match a component name from a specific location.
const tabularDataMQLCount = await client.dataClient.tabularDataByMQL(
    ORG_ID,
    [
        { "$match": { "component_name": "sensor-1" } },
        { "$count": "count" }
    ]
);
console.log(tabularDataMQLCount);

const tabularDataSQLCount = await client.dataClient.tabularDataBySQL( ORG_ID, "SELECT count(*) FROM readings WHERE component_name = 'sensor-1'" ); console.log(tabularDataSQLCount);

Query using third-party tools

Prerequisites

Captured sensor data
Follow the guide to capture sensor data.
Viam CLI

You must have the Viam CLI installed to configure querying with third-party tools.

To download the Viam CLI on a macOS computer, install brew and run the following commands:

brew tap viamrobotics/brews
brew install viam

To download the Viam CLI on a Linux computer with the aarch64 architecture, run the following commands:

sudo curl -o /usr/local/bin/viam https://storage.googleapis.com/packages.viam.com/apps/viam-cli/viam-cli-stable-linux-arm64
sudo chmod a+rx /usr/local/bin/viam

To download the Viam CLI on a Linux computer with the amd64 (Intel x86_64) architecture, run the following commands:

sudo curl -o /usr/local/bin/viam https://storage.googleapis.com/packages.viam.com/apps/viam-cli/viam-cli-stable-linux-amd64
sudo chmod a+rx /usr/local/bin/viam

You can also install the Viam CLI using brew on Linux amd64 (Intel x86_64):

brew tap viamrobotics/brews
brew install viam

Download the binary and run it directly to use the Viam CLI on a Windows computer.

If you have Go installed, you can build the Viam CLI directly from source using the go install command:

go install go.viam.com/rdk/cli/viam@latest

To confirm viam is installed and ready to use, issue the viam command from your terminal. If you see help instructions, everything is correctly installed. If you do not see help instructions, add your local go/bin/* directory to your PATH variable. If you use bash as your shell, you can use the following command:

echo 'export PATH="$HOME/go/bin:$PATH"' >> ~/.bashrc

For more information see install the Viam CLI.

A third-party tool for querying data, such as mongosh
Download the mongosh shell or another third-party tool that can connect to a MongoDB data source to follow along. See the mongosh documentation for more information.

Configure data query

If you want to query data from third party tools, you have to configure data query to obtain the credentials you need to connect to the third party service.

1

Authenticate with the CLI

Authenticate using a personal access token:

viam login

For alternative authentication methods, see Authenticate.

2

Find your organization ID

To create a database user allowing you to access your data, find your organization ID:

viam organizations list
3

Configure a new database user

Configure a new database user for the Viam organization’s MongoDB Atlas Data Federation instance, which is where your machine’s synced data is stored.

Provide your organization’s org-id from step 2, and a password for your database user.

viam data database configure --org-id=<YOUR-ORGANIZATION-ID> --password=<NEW-DBUSER-PASSWORD>

This command configures a database user for your organization for use with data query, and sets the password. If you have run this command before, this command instead updates the password to the new value you set.

4

Determine the connection URI

Determine the connection URI (also known as a connection string) for your organization’s MongoDB Atlas Data Federation instance by running the following command with the organization’s org-id from step 2:

viam data database hostname --org-id=abcd1e2f-a1b2-3c45-de6f-ab123456c123
# Example output
MongoDB Atlas Data Federation instance hostname: data-federation-abcd1e2f-a1b2-3c45-de6f-ab123456c123-0z9yx.a.query.mongodb.net
MongoDB Atlas Data Federation instance connection URI: mongodb://db-user-abcd1e2f-a1b2-3c45-de6f-ab123456c123:YOUR-PASSWORD-HERE@data-federation-abcd1e2f-a1b2-3c45-de6f-ab123456c123-0z9yx.a.query.mongodb.net/?ssl=true&authSource=admin

This command returns:

  • hostname: the MongoDB Atlas Data Federation instance hostname

  • connection URI: the MongoDB Atlas Data Federation instance connection uniform resource indicator. This is the connection URI to your organization’s MongoDB Atlas Data Federation instance, which is of the form:

    mongodb://<USERNAME>:<YOUR-PASSWORD>@<HOSTNAME>/?ssl=true&authSource=admin
    

Most MQL-compatible database clients require the connection URI, along with your user credentials, to connect to this server.

Some MQL-compatible database client instead require a hostname and database name, along with your user credentials, to connect to this server.

You will need the connection URI to query your data in the next section.

Query data using third-party tools

You can use third-party tools, such as the mongosh shell or MongoDB Compass, to query captured sensor data.

1

Connect to your Viam organization’s data

Run the following command to connect to your Viam organization’s MongoDB Atlas instance from mongosh using the connection URI you obtained during query configuration:

mongosh "mongodb://db-user-abcd1e2f-a1b2-3c45-de6f-ab123456c123:YOUR-PASSWORD-HERE@data-federation-abcd1e2f-a1b2-3c45-de6f-ab123456c123-0z9yx.a.query.mongodb.net/?ssl=true&authSource=admin"
2

Query data from a compatible client

Once connected, you can run SQL or MQL statements to query captured data directly.

The following query searches the sensorData database and readings collection, and gets sensor readings from an ultrasonic sensor on a specific robot_id where the recorded distance measurement is greater than .2 meters.

The following MQL query performs counts the number of sensor readings where the distance value is above 0.2 using the MongoDB query language:

use sensorData
db.readings.aggregate(
    [
        { $match: {
            'robot_id': 'abcdef12-abcd-abcd-abcd-abcdef123456',
            'component_name': 'my-ultrasonic-sensor',
            'data.readings.distance': { $gt: .2 } } },
        { $count: 'numStanding' }
    ] )
[ { numStanding: 215 } ]

The following query uses the MongoDB $sql aggregation pipeline stage:

use sensorData
db.aggregate(
[
    { $sql: {
        statement: "select count(*) as numStanding from readings \
            where robot_id = 'abcdef12-abcd-abcd-abcd-abcdef123456' and \
            component_name = 'my-ultrasonic-sensor' and (CAST (data.readings.distance AS DOUBLE)) > 0.2",
        format: "jdbc"
    }}
] )
[ { '': { numStanding: 215 } } ]
Need to query by date? Click here.
Query by date

When using MQL to query your data by date or time range, you can optimize query performance by avoiding the MongoDB $toDate expression, using the BSON date type instead.

For example, use the following query to search by a date range in the mongosh shell, using the JavaScript Date() constructor to specify an explicit start timestamp, and use the current time as the end timestamp:

// Switch to sensorData database:
use sensorData

// Set desired start and end times:
const startTime = new Date('2024-02-10T19:45:07.000Z')
const endTime = new Date()

// Run query using $match:
db.readings.aggregate(
    [
        { $match: {
            time_received: {
                $gte: startTime,
                $lte: endTime }
        } }
    ] )

For information on connecting to your Atlas instance from other MQL clients, see the MongoDB Atlas Connect to your Cluster Tutorial.

On top of querying sensor data with third-party tools, you can also query it with the Python SDK or visualize it.

Supported query languages

MQL

Viam supports the MongoDB Query Language for querying captured data from MQL-compatible clients such as mongosh or MongoDB Compass.

Supported aggregation operators

Viam supports the following MongoDB aggregation operators:

  • $addFields
  • $bucket
  • $bucketAuto
  • $count
  • $densify
  • $fill
  • $geoNear
  • $group
  • $limit
  • $match
  • $project
  • $redact
  • $replaceRoot
  • $replaceWith
  • $sample
  • $set
  • $setWindowFields
  • $skip
  • $sort
  • $sortByCount
  • $unset
  • $unwind

SQL

You can query data with SQL queries using the MongoDB Atlas SQL dialect, which supports standard SQL query syntax in addition to Atlas-specific capabilities such as FLATTEN and UNWIND.

SQL queries are subject to the following limitations:

  • If a database, table, or column identifier meets any of the following criteria, you must surround the identifier with backticks (`) or double quotes ("):
  • To include a single quote character in a string literal, use two single quotes (use o''clock to represent the literal o'clock).
  • The date data type is not supported. Use timestamp instead.

For a full list of limitations, see the MongoDB Atlas SQL Interface Language Reference.