Skip to main content
Version: Next

Your First Bundle - Basic

This tutorial will walk you through the process of creating a simple Hoppr bundle. During this tutorial, if you'd like to understand more about how Hoppr works, check out Hoppr Processing 101. This tutorial will not require a credentials file.

tip

Interested in walking through this tutorial without installing Hoppr? We have a Gitpod environment setup with Hoppr installed.

Prep​

To begin, create a working directory somewhere convenient:

mkdir test-hoppr
cd test-hoppr

We will use this directory to store our input and write Hoppr output files.

Create an SBOM​

We use the CycloneDX BOM format to specify our Software Bill of Materials (SBOM). Hoppr currently supports the json format for the SBOM. In practice, most SBOMs will be generated using a tool intended for that purpose. However, for this example, we will create our SBOM manually.

Our bundle will consist of the following items:

  • The ~/.bashrc file for the current user
  • The Hoppr Hippo logo downloaded from our documentation site
  • The most recent version of the hello-world docker image, downloaded from docker.io
  • The current Hoppr code base, downloaded from gitlab.com

Note that within the SBOM, artifact locations are given in terms of a PURL (Package URL). PURLs do not always include the full path to the source object. The repositories to search for each artifact will be specified in the manifest file (next section).

Our very simple SBOM is

{
    "bomFormat": "CycloneDX",
    "specVersion": "1.5",
    "version": 1,
    "components": [
        {
            "type": "file",
            "name": ".bashrc",
            "purl": "pkg:generic/~/.bashrc"
        },
        {
            "type": "library",
            "name": "HopprHippo-01.png",
            "purl": "pkg:generic/img/HopprHippo-01.png"
        },
        {
            "type": "library",
            "name": "hello-world",
            "version": "latest",
            "purl": "pkg:docker/library/hello-world@latest"
        },
        {
            "type": "library",
            "name": "hoppr",
            "purl": "pkg:git/hoppr/hoppr"
        }
     ]
}

Copy the above file and store it in your working directory as my-first-sbom.json.

Build a Manifest​

Next, we will need a manifest file. The manifest specifies the location of the SBOM (above) and lists the repositories in which Hoppr is to search for the artifacts defined there.

We'll assume that the above SBOM has been saved as a file named my-first-sbom.json in the same directory as we're placing the manifest file.

Although we are only listing one SBOM, multiple SBOMs may be included. In addition, other manifests may be specified recursively by using the include tag, although we will not be using that function in this example (hence the includes tag has an empty list for its value).

---
schemaVersion: v1
kind: manifest

metadata:
  name: "My First Manifest"
  version: 0.1.0
  description: "An introductory manifest"

sboms:
  - local: my-first-sbom.json

includes: []

repositories:
  generic:
  - url: "file:"
  - url: https://hoppr.dev/
  docker:
  - url: https://docker.io
  git:
  - url: https://gitlab.com

Copy the above file and store it in your working directory as my-first-manifest.yml.

Transfer​

Now that we have defined what artifacts we wish to work with and where they are to be found, we need to identify what we actually want to do with them. That is the purpose of the Transfer file.

The Transfer file defines a list of stages, which are executed in sequence. Within each stage, a number of Hoppr plugins are executed. Plugins within a stage are executed in parallel.

For our example, we will have two stages: the first, Collect, will collect the artifacts specified in the SBOM, and the second, Bundle, will bundle those artifacts into a tarball for transfer. Stage names are user-defined, and have no particular intrinsic meaning.

---
schemaVersion: v1
kind: Transfer

stages:
  Collect:
    plugins:
    - name: "hoppr.core_plugins.collect_docker_plugin"
    - name: "hoppr.core_plugins.collect_git_plugin"
    - name: "hoppr.core_plugins.collect_raw_plugin"
  Bundle:
    plugins:
    - name: "hoppr.core_plugins.bundle_tar"

For our Collect stage, we want to run three plugins -- one for each PURL type included in our example SBOM. Each of these plugins copies all artifacts of the appropriate type(s) to a temporary internal directory structure.

Our Bundle stage consists of a single plugin (hoppr.core_plugins.bundle_tar), which collects everything in the internal directory from the Collect stage and creates a tarball.

Other parameters may be added to the Transfer file, impacting how the entire process or individual stages run. Similarly, most plugins may be configured with a config parameter, but for our example, we will stick with the defaults for all of these.

Copy the above file and store it in your working directory as my-first-transfer.yml.

Run Hoppr​

It looks like we're now ready to run Hoppr and create our first bundle. Before we proceed, let's check that:

  1. Hoppr has been installed
  2. git (needed for the collect_git_plugin) has been installed
  3. skopeo (needed for the collect_docker_plugin) has been installed
  4. The input files described above have been created
$ cd test-hoppr

$ hopctl version
Hoppr Framework Version: 1.6.2
Python Version: 3.10.5

$ git --version
git version 2.31.1

$ skopeo --version
skopeo version 1.8.0

$ ls -l
total 5
-rwxrwxrwx. 1 vagrant vagrant      331 Nov 17 13:20 my-first-manifest.yml
-rwxrwxrwx. 1 vagrant vagrant      699 Nov 17 13:17 my-first-sbom.json
-rwxrwxrwx. 1 vagrant vagrant      296 Nov 17 13:18 my-first-transfer.yml

Now we're ready to execute the hopctl bundle command. Recall that the SBOM is not listed as a command-line parameter; it is specified in the manifest file.

hopctl bundle my-first-manifest.yml --transfer my-first-transfer.yml

If all goes well, you should see output like this:

Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
   Beginning method process_component
      CollectDockerPlugin SUCCESS for pkg:docker/library/hello-world@latest
      CollectGitPlugin SUCCESS for pkg:git /hoppr/hoppr.git
      CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
      CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
   Beginning method post_stage_process
      CollectDockerPlugin SUCCESS
      CollectGitPlugin SUCCESS
      CollectRawPlugin SUCCESS
========== Beginning Stage Bundle ==================================================
   Beginning method post_stage_process
      TarBundlePlugin SUCCESS

========== Results Summary ==========

Stage: Collect
   process_component
      4 jobs succeeded, 0 failed
   post_stage_process
      3 jobs succeeded, 0 failed

Stage: Bundle
   post_stage_process
      1 jobs succeeded, 0 failed

GRAND TOTAL: 8 jobs succeeded, 0 failed

The process created two new files in your directory:

  • bundle.tar.gz, which is the tarball containing all the requested artifacts.
  • hoppr_<date>_<time>.log, which includes log output helpful for debugging issues.
$ ls -l
total 13789
-rwxrwxrwx. 1 vagrant vagrant 14103142 Nov 17 13:31 bundle.tar.gz
-rwxrwxrwx. 1 vagrant vagrant     5026 Nov 17 13:31 hoppr_20221117-133142.log
-rwxrwxrwx. 1 vagrant vagrant      331 Nov 17 13:20 my-first-manifest.yml
-rwxrwxrwx. 1 vagrant vagrant      699 Nov 17 13:17 my-first-sbom.json
-rwxrwxrwx. 1 vagrant vagrant      296 Nov 17 13:18 my-first-transfer.yml

The tar file in this example may seem large, but that is driven primarily by the inclusion of the entire Hoppr git repository:

$ tar -tf bundle.tar.gz
./
./docker/
./docker/https%3A%2F%2Fdocker.io/
./docker/https%3A%2F%2Fdocker.io/library/
./docker/https%3A%2F%2Fdocker.io/library/hello-world_latest
./generic/
./generic/_metadata_/
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-manifest.yml
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-sbom.json
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-transfer.yml
./generic/_metadata_/_consolidated_bom.json
./generic/_metadata_/_delivered_bom.json
./generic/_metadata_/_run_data_
./generic/file%3A/
./generic/file%3A/~/
./generic/file%3A/~/.bashrc
./generic/https%3A%2F%2Fhoppr.dev/
./generic/https%3A%2F%2Fhoppr.dev/img/
./generic/https%3A%2F%2Fhoppr.dev/img/HopprHippo-01.png
./git/
./git/https%3A%2F%2Fgitlab.com/
./git/https%3A%2F%2Fgitlab.com/
./git/https%3A%2F%2Fgitlab.com/hoppr/
./git/https%3A%2F%2Fgitlab.com/hoppr/hoppr.git/
# Followed by all the files in the Hoppr git repository....

In addition to the requested artifacts, the bundle includes metadata (in the ./generic/_metadata_/ directory). This metadata includes:

  • The manifest file
  • The transfer file
  • All input SBOMs files
  • A consolidated SBOM (combining all input SBOMs into one) file
  • An SBOM file describing what was included in the bundle (which may differ from the consolidated SBOM if any filtering is performed by Hoppr)
    • For our example, the consolidated and delivered SBOMs are identical
  • A _run_data_ file with information about when and where the bundle was created
note

Hoppr upconverts to the latest CycloneDX specification version for the delivered SBOM.

Debugging Runtime Errors​

As we said, the above results are what we get when everything works correctly. But sometimes it doesn't. Here are a few common errors and what they mean:

Required Commands are Unavailable​

As stated on the installation page, many plugins make use of commands that are expected to be available on the system running Hoppr.

The following output shows the error that results when one of these tools is not found (in this example, skopeo, which is used by the collect_docker_plugin):

Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
   Beginning method process_component
      CollectDockerPlugin FAIL for pkg:docker/library/hello-world@latest: The following required commands are unavailable: skopeo.  Please install and try again.
      CollectGitPlugin SUCCESS for pkg:git/hoppr/hoppr.git
      CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
      CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
   Beginning method post_stage_process
      CollectDockerPlugin FAIL: The following required commands are unavailable: skopeo.  Please install and try again.
      CollectGitPlugin SUCCESS
      CollectRawPlugin SUCCESS
   Stage Collect failed, processing terminated: 1 'process_component' processes failed
1 'post_stage_process' processes failed

========== Results Summary ==========

Stage: Collect
   process_component
      3 jobs succeeded, 1 failed

      Failure Summary:
         CollectDockerPlugin: Component: pkg:docker/library/hello-world@latest: The following required commands are unavailable: skopeo.  Please install and try again.

   post_stage_process
      2 jobs succeeded, 1 failed

      Failure Summary:
         CollectDockerPlugin: The following required commands are unavailable: skopeo.  Please install and try again.


GRAND TOTAL: 5 jobs succeeded, 2 failed

Artifact not found​

When a requested artifact is not found, the resulting error may not be as clear as we would like. For example, I modified our SBOM to look for the docker image goodbye-world rather than hello-world:

Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
   Beginning method process_component
      CollectDockerPlugin FAIL for pkg:docker/library/goodbye-world@latest: Failure after 3 attempts, final message Skopeo failed to copy docker image to docker-archive:/tmp/tmpwl2rrutq/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
      CollectGitPlugin SUCCESS for pkg:git/hoppr/hoppr.git
      CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
      CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
   Beginning method post_stage_process
      CollectDockerPlugin SUCCESS
      CollectGitPlugin SUCCESS
      CollectRawPlugin SUCCESS
   Stage Collect failed, processing terminated: 1 'process_component' processes failed


========== Results Summary ==========

Stage: Collect
   process_component
      3 jobs succeeded, 1 failed

      Failure Summary:
         CollectDockerPlugin: Component: pkg:docker/library/goodbye-world@latest: Failure after 3 attempts, final message Skopeo failed to copy docker image to docker-archive:/tmp/tmpwl2rrutq/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1

   post_stage_process
      3 jobs succeeded, 0 failed

GRAND TOTAL: 6 jobs succeeded, 1 failed

From this, all we know is that skopeo failed, despite trying three times (Hoppr typically re-tries up to three times in the event of a failure).

To find out more, we need to look in the log. The relevant section of the log (from a hopctl run with the -v flag, to include DEBUG log messages) tells us:

[2022-11-17 20:08:19,219] - [CollectDockerPlugin--44413-2] - [INFO] - ---- Component: goodbye-world@latest --------------------------------------------------
[2022-11-17 20:08:19,221] - [CollectDockerPlugin--44413-2] - [INFO] - Repository: https://docker.io
[2022-11-17 20:08:19,221] - [CollectDockerPlugin--44413-2] - [INFO] -     Processing component [attempt 1 of 3]
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] -         Copying docker image:
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] -             source: docker://docker.io/library/goodbye-world:latest
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] -             destination: docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [DEBUG] -         Running command: 'skopeo copy docker://docker.io/library/goodbye-world:latest docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest'
[2022-11-17 20:08:20,081] - [CollectDockerPlugin--44413-2] - [DEBUG] -         skopeo command stdout content:

[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [ERROR] -         skopeo command failed with error:
    time="2022-11-17T20:08:20Z" level=fatal msg="initializing source docker://goodbye-world:latest: reading manifest latest in docker.io/library/goodbye-world: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"

[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [DEBUG] -         Skopeo failed to copy docker image to docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [INFO] -         Artifact collection failed, deleting file and retrying
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [WARNING] -     Method collect will be retried in 5 seconds
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [WARNING] -     Result message for attempt 1: Skopeo failed to copy docker image to docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1

Dig through that, and you see that the actual error message returned by skopeo was "initializing source docker://goodbye-world:latest: reading manifest latest in docker.io/library/goodbye-world: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required". Which (it turns out) is one of the messages skopeo gives when a requested image is not found.

Component processed 0 times​

I removed the collect_git_plugin from the Transfer file, and got this output:

Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
   Beginning method process_component
      CollectDockerPlugin SUCCESS for pkg:docker/library/hello-world@latest
      CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
      CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
      Stage Collect FAIL for pkg:git/hoppr/hoppr.git: Component processed 0 times, EXACTLY_ONCE coverage required
   Beginning method post_stage_process
      CollectDockerPlugin SUCCESS
      CollectRawPlugin SUCCESS
   Stage Collect failed, processing terminated: 1 'process_component' processes failed


========== Results Summary ==========

Stage: Collect
   process_component
      3 jobs succeeded, 1 failed

      Failure Summary:
         Stage Collect: Component: pkg:git/hoppr/hoppr.git: Component processed 0 times, EXACTLY_ONCE coverage required

   post_stage_process
      2 jobs succeeded, 0 failed

GRAND TOTAL: 5 jobs succeeded, 1 failed

What this is telling us is that there was a component in the SBOM that none of the included collector plugins was capable of collecting.

Hoppr stages include an attribute indicating how frequently SBOM components are to be processed. When all of the plugins in a given stage are "collectors", that value defaults to EXACTLY_ONCE.

Since the git collector was removed from the Transfer file, there was no collector available to collect the git component. It was not collected, resulting in this error.

Note that the Bundle stage did not run at all, because the Collect stage failed.