The latest major release of cnquery and cnspec introduced the biggest change since their inception! We spent months reworking the foundation of both projects to prepare them for a growing community and a lasting future.

We have always wanted our users to be able to create their own providers (pluggable components that add capabilities) to expand the projects however they like. The explosive growth in the types of technologies we support has been a strong indication for this model.

At the same time, we wanted to make sure that both cnquery and cnspec remain nimble enough for any environment (including embedded devices and other tiny runtimes). It became clear to us that both projects needed to change.

In fact, this update modified 2,357 files and changed a total of 536,512 lines of code. We reduced the size of both binaries by 90%. We not only introduced extensible providers, but also fully modularized cnquery and cnspec and rebuilt them around efficient caching.

Extensibility

From the beginning, both cnquery and cnspec were built with extensibility at their core. However, because these projects were built as monoliths, including the binaries we generated, it was extremely difficult to contribute to them because any contribution quickly became a change to the ever-growing core project.

Testing took a long time. Change requests were hard. Contributors had to write everything in Go. It had to be built and distributed by Mondoo. Also, private providers were virtually impossible (unless you built your own binary).

With the release of v9, this finally changed.

Both cnquery and cnspec are now built around a fully extensible ecosystem of plugin providers.

With v9, cnquery now uses Terraform's plugin library as the foundation for all built-in and custom providers. We chose this mechanism for creating and maintaining plugins because it's well tested and so widely used across production environments.

Provider plugins are now self-contained processes that communicate with their runtime using gRPC and protocol buffers. Now anyone can write, publish, and maintain their own provider. You're free to use any programming language you like, as long as the result is an executable file that implements the same plugin API.

With this new system, it's even possible to create entirely private providers. This is useful if you want to use MQL tools with your own APIs but prefer not to leak their implementation or SDK to the outside world. Others can still write code against a private provider given its schema, even if they don't have access to the provider itself.

You can find examples of our built-in providers in GitHub in the cnquery providers folder.

There are also utilities to help scaffold a new provider:

go install apps/provider-scaffold/provider-scaffold.go
provider-scaffold \
  --path providers/my-name \
  --provider-id my-name \
  --provider-name "My Name" \
  --go-package github.com/user/cnquery-provider-my-name

‍

Any provider plugin you install becomes available to both cnquery and cnspec (and any other MQL runtime). This avoids duplication and bloated files on the system. If it works for one runtime, it works for all of them.

Modularization

These new plugins are incredibly easy to use. We wanted to make sure users didn't have to learn anything new after updating to v9.

Providers install and update on the fly and on their own. If you scan your local system, the OS provider is used. If you scan AWS, the AWS provider automatically installs. If you scan GitHub, the GitHub provider automatically installs. If you scan Okta… well, you get the idea.

This change allowed us to keep our main binaries small (90% smaller than v8) and reduce their security footprint dramatically because only the required capabilities are now installed.

Whenever you use an installed provider, it checks if there are updates available. We know most users have difficulty keeping software up to date. To keep things running smoothly, we've changed the default behavior to be conducive of better security practices and reduce the overhead on the user. And don't worry; there are safeguards to prevent it from checking too often.

Most importantly, you stay in control. You can choose to disable the auto-install and auto-update feature entirely. This is useful for containers and serverless environments. For those environments, we recommend installing cnquery or cnspec with the subset of providers that you expect to use. Their versions are then pinned and you can disable auto-updates using the CLI (--auto-update=false) or the config file (auto_update: false). This ensures that the containers don't try to install new or updated providers on every run.

All providers (apart from core, at the time of writing), are now optional. This means that if you don't want to use the OS provider, you don't have to install it. If you think your cnspec will only scan Azure or Google Cloud, you only install those providers. From a security perspective, this is awesome because the OS provider (which interacts with the local system) is not available if you disable auto-updates.

Caching and recordings

The last change we introduced in v9 tackles caching at its core. Before this release, we were only able to cache and share calls that were made on the OS. This limitation is now gone.

With the release of v9, we now have comprehensive caching and recording capabilities available for all existing, custom, and future providers. This not only allows us to steadily reduce the number of calls made to (expensive) APIs, but it also helps us in debugging and diving into the collected data.

We'll continue to extend the caching layer in coming releases. With the introduction of the recording feature, we can now use any data plane for our collection and playback.

Currently, we have support for caching/recording to local JSON files. These can be used with any of our core commands: scan, run, or shell. This means that you can scan a target and collect all the data that is used in one file. Or you can open an interactive shell and capture every resource and field collected. Both of these capabilities are helpful for diffs, drifts, and debugging purposes (because the person who uses your recording doesn't need access to the target system).

We soon will extend this feature to work with additional data backends. If you have a registered client, you will be able to use Mondoo as a transparent data plane and massively reduce the number of expensive (and throttling) API calls. Stay tuned: We are excited to announce this new capability in the near future!