Create Custom cnquery Providers

While cnquery includes providers for many common platforms, you might need to query systems or services that aren't yet supported. Custom providers let you extend cnquery to query any platform, API, or data source.

You can create your own custom providers for any MQL runtime, including cnquery and cnspec. Providers can be written in any language or framework as long as they adhere to the plugin structure and API.

Provider structure

Provider plugins are located by default in either the system or user location. They consist of a folder containing a set of required files. For example, for a provider mypro, you would see the following structure:

./mypro/                    Folder containing the provider and additional files
├── mypro                   Executable used to start the provider plugin
├── mypro.json              Provider metadata
└── mypro.resources.json    Resources and fields schema

• Provider folder The folder that must contain at least the binary, metadata, and schema. Additionally, providers may contain other custom files.

• Provider binary The binary is spawned when a new provider instance is created.

  It must adhere to the plugin behavior defined in go-plugin. This means it must create a blocking process that communicates via gRPC with the caller and implements the provider plugin proto API.

  Due to these specifications, providers can be created in any language or stack, as long as they offer a callable binary (that is executable on the target system) and communicate via the gRPC interface.

• Provider metadata This file contains information about the provider like its name, UID, version, connections, and connectors. It is also used to build the CLI interface. The structure is defined in the Provider struct.

• Resources and fields schema The schema contains all resources and fields that are offered by this provider. This includes version constraints for compatibility, field types, and basic documentation. It is defined in the Schema message in the resources proto. Provider schemas can be auto-generated (see scaffolding below).

Providers are distributed as tar.xz files which contain the above structure. They can be installed using these compressed archives via:

cnquery providers install -f provider.tar.xz

Providers can also be installed manually by creating the above structure in the user system provider location.

Provider scaffolding

To ease the creation of providers, cnquery includes a scaffolding utility. The scaffolding tool is designed for Go and requires Go 1.21 or later.

To install it, first clone the cnquery repository and then install the tool:

git clone https://github.com/mondoohq/cnquery.git
cd cnquery
go install ./apps/provider-scaffold

To create a new provider (in this example, mypro), run:

provider-scaffold --path mypro --provider-id mypro --provider-name "My Provider" --go-package github.com/myuser/mql-provider-mypro/mypro

The scaffolding tool currently generates Go-based providers. Contributions to extend it to other languages are welcome.

Builtin providers

If you prefer not to have separate provider binaries that are spawned, you can build any MQL runtime with Go-based providers directly into the binary. This means that no additional files are installed or updated, and no additional processes are spawned.

To manually configure builtin providers, you can modify the list of builtinProviders. Once added, you will see it in the list of builtin providers for any binary you build, for example:

> cnquery providers

→ builtin (found 3 providers)

  core 11.0.98
  mock 9.0.0 with connectors: mock, upstream
  mypro 0.0.1 with connectors: mytarget

...

Any MQL runtime can be built into a binary that contains these providers. Note that builtin providers cannot be updated without creating a new build, and they increase the file size of the runtime. However, this approach can be advantageous when you want an overall smaller footprint and tighter security profile.