Your First Bundle - Basic
This tutorial will walk you through the process of creating a simple Hoppr bundle. During this tutorial, if you'd like to understand more about how Hoppr works, check out Hoppr Processing 101. This tutorial will not require a credentials file.
Interested in walking through this tutorial without installing Hoppr? We have a Gitpod environment setup with Hoppr installed.
Prep
To begin, create a working directory somewhere convenient:
mkdir test-hoppr
cd test-hoppr
We will use this directory to store our input and write Hoppr output files.
Create an SBOM
We use the CycloneDX BOM format to specify our Software Bill of Materials (SBOM). Hoppr currently supports the json
format for the SBOM. In practice, most SBOMs will be generated using a tool intended for that purpose. However, for this example, we will create our SBOM manually.
Our bundle will consist of the following items:
- The
~/.bashrc
file for the current user - The Hoppr Hippo logo downloaded from our documentation site
- The most recent version of the
hello-world
docker image, downloaded from docker.io - The current Hoppr code base, downloaded from gitlab.com
Note that within the SBOM, artifact locations are given in terms of a PURL (Package URL). PURLs do not always include the full path to the source object. The repositories to search for each artifact will be specified in the manifest file (next section).
Our very simple SBOM is
{
"bomFormat": "CycloneDX",
"specVersion": "1.4",
"version": 1,
"components": [
{
"type": "file",
"name": ".bashrc",
"purl": "pkg:generic/~/.bashrc"
},
{
"type": "library",
"name": "HopprHippo-01.png",
"purl": "pkg:generic/img/HopprHippo-01.png"
},
{
"type": "library",
"name": "hello-world",
"version": "latest",
"purl": "pkg:docker/library/hello-world@latest"
},
{
"type": "library",
"name": "hoppr",
"purl": "pkg:git/hoppr/hoppr.git"
}
]
}
Copy the above file and store it in your working directory as my-first-sbom.json
.
Build a Manifest
Next, we will need a manifest file. The manifest specifies the location of the SBOM (above) and lists the repositories in which Hoppr is to search for the artifacts defined there.
We'll assume that the above SBOM has been saved as a file named my-first-sbom.json
in the same directory as we're placing the manifest file.
Although we are only listing one SBOM, multiple SBOMs may be included. In addition, other manifests may be specified recursively by using the include
tag, although we will not be using that function in this example (hence the includes
tag has an empty list for its value).
---
schemaVersion: v1
kind: manifest
metadata:
name: "My First Manifest"
version: 0.1.0
description: "An introductory manifest"
sboms:
- local: my-first-sbom.json
includes: []
repositories:
generic:
- url: "file:"
- url: https://hoppr.dev/
docker:
- url: https://docker.io
git:
- url: https://gitlab.com
Copy the above file and store it in your working directory as my-first-manifest.yml
.
Transfer
Now that we have defined what artifacts we wish to work with and where they are to be found, we need to identify what we actually want to do with them. That is the purpose of the Transfer file.
The Transfer file defines a list of stages
, which are executed in sequence. Within each stage, a number of Hoppr plugins are executed. Plugins within a stage are executed in parallel.
For our example, we will have two stages: the first, Collect
, will collect the artifacts specified in the SBOM, and the second, Bundle
, will bundle those artifacts into a tarball for transfer. Stage names are user-defined, and have no particular intrinsic meaning.
---
schemaVersion: v1
kind: Transfer
stages:
Collect:
plugins:
- name: "hoppr.core_plugins.collect_docker_plugin"
- name: "hoppr.core_plugins.collect_git_plugin"
- name: "hoppr.core_plugins.collect_raw_plugin"
Bundle:
plugins:
- name: "hoppr.core_plugins.bundle_tar"
For our Collect
stage, we want to run three plugins -- one for each PURL type included in our example SBOM. Each of these plugins copies all artifacts of the appropriate type(s) to a temporary internal directory structure.
Our Bundle
stage consists of a single plugin (hoppr.core_plugins.bundle_tar
), which collects everything in the internal directory from the Collect
stage and creates a tarball.
Other parameters may be added to the Transfer
file, impacting how the entire process or individual stages run. Similarly, most plugins may be configured with a config
parameter, but for our example, we will stick with the defaults for all of these.
Copy the above file and store it in your working directory as my-first-transfer.yml
.
Run Hoppr
It looks like we're now ready to run Hoppr and create our first bundle. Before we proceed, let's check that:
- Hoppr has been installed
git
(needed for thecollect_git_plugin
) has been installedskopeo
(needed for thecollect_docker_plugin
) has been installed- The input files described above have been created
$ cd test-hoppr
$ hopctl version
Hoppr Framework Version: 1.6.2
Python Version: 3.10.5
$ git --version
git version 2.31.1
$ skopeo --version
skopeo version 1.8.0
$ ls -l
total 5
-rwxrwxrwx. 1 vagrant vagrant 331 Nov 17 13:20 my-first-manifest.yml
-rwxrwxrwx. 1 vagrant vagrant 699 Nov 17 13:17 my-first-sbom.json
-rwxrwxrwx. 1 vagrant vagrant 296 Nov 17 13:18 my-first-transfer.yml
Now we're ready to execute the hopctl bundle
command. Recall that the SBOM is not listed as a command-line parameter; it is specified in the manifest file.
hopctl bundle --transfer my-first-transfer.yml my-first-manifest.yml
If all goes well, you should see output like this:
Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
Beginning method process_component
CollectDockerPlugin SUCCESS for pkg:docker/library/hello-world@latest
CollectGitPlugin SUCCESS for pkg:git /hoppr/hoppr.git
CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
Beginning method post_stage_process
CollectDockerPlugin SUCCESS
CollectGitPlugin SUCCESS
CollectRawPlugin SUCCESS
========== Beginning Stage Bundle ==================================================
Beginning method post_stage_process
TarBundlePlugin SUCCESS
========== Results Summary ==========
Stage: Collect
process_component
4 jobs succeeded, 0 failed
post_stage_process
3 jobs succeeded, 0 failed
Stage: Bundle
post_stage_process
1 jobs succeeded, 0 failed
GRAND TOTAL: 8 jobs succeeded, 0 failed
The process created two new files in your directory:
bundle.tar.gz
, which is the tarball containing all the requested artifacts.hoppr_<date>_<time>.log
, which includes log output helpful for debugging issues.
$ ls -l
total 13789
-rwxrwxrwx. 1 vagrant vagrant 14103142 Nov 17 13:31 bundle.tar.gz
-rwxrwxrwx. 1 vagrant vagrant 5026 Nov 17 13:31 hoppr_20221117-133142.log
-rwxrwxrwx. 1 vagrant vagrant 331 Nov 17 13:20 my-first-manifest.yml
-rwxrwxrwx. 1 vagrant vagrant 699 Nov 17 13:17 my-first-sbom.json
-rwxrwxrwx. 1 vagrant vagrant 296 Nov 17 13:18 my-first-transfer.yml
The tar file in this example may seem large, but that is driven primarily by the inclusion of the entire Hoppr git repository:
$ tar -tf bundle.tar.gz
./
./docker/
./docker/https%3A%2F%2Fdocker.io/
./docker/https%3A%2F%2Fdocker.io/library/
./docker/https%3A%2F%2Fdocker.io/library/hello-world_latest
./generic/
./generic/_metadata_/
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-manifest.yml
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-sbom.json
./generic/_metadata_/%2Fvagrant%2Fhoppr-dev%2Fhoppr-docs%2Ftest-hoppr%2Fmy-first-transfer.yml
./generic/_metadata_/_consolidated_bom.json
./generic/_metadata_/_delivered_bom.json
./generic/_metadata_/_run_data_
./generic/file%3A/
./generic/file%3A/~/
./generic/file%3A/~/.bashrc
./generic/https%3A%2F%2Fhoppr.dev/
./generic/https%3A%2F%2Fhoppr.dev/img/
./generic/https%3A%2F%2Fhoppr.dev/img/HopprHippo-01.png
./git/
./git/https%3A%2F%2Fgitlab.com/
./git/https%3A%2F%2Fgitlab.com/
./git/https%3A%2F%2Fgitlab.com/hoppr/
./git/https%3A%2F%2Fgitlab.com/hoppr/hoppr.git/
# Followed by all the files in the Hoppr git repository....
In addition to the requested artifacts, the bundle includes metadata (in the ./generic/_metadata_/
directory). This metadata includes:
- The manifest file
- The transfer file
- All input SBOMs files
- A consolidated SBOM (combining all input SBOMs into one) file
- An SBOM file describing what was included in the bundle (which may differ from the consolidated SBOM if any filtering is performed by Hoppr)
- For our example, the consolidated and delivered SBOMs are identical
- A
_run_data_
file with information about when and where the bundle was created
Debugging Runtime Errors
As we said, the above results are what we get when everything works correctly. But sometimes it doesn't. Here are a few common errors and what they mean:
Required Commands are Unavailable
As stated on the installation page, many plugins make use of commands that are expected to be available on the system running Hoppr.
The following output shows the error that results when one of these tools is not found (in this example, skopeo
, which is used by the collect_docker_plugin
):
Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
Beginning method process_component
CollectDockerPlugin FAIL for pkg:docker/library/hello-world@latest: The following required commands are unavailable: skopeo. Please install and try again.
CollectGitPlugin SUCCESS for pkg:git/hoppr/hoppr.git
CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
Beginning method post_stage_process
CollectDockerPlugin FAIL: The following required commands are unavailable: skopeo. Please install and try again.
CollectGitPlugin SUCCESS
CollectRawPlugin SUCCESS
Stage Collect failed, processing terminated: 1 'process_component' processes failed
1 'post_stage_process' processes failed
========== Results Summary ==========
Stage: Collect
process_component
3 jobs succeeded, 1 failed
Failure Summary:
CollectDockerPlugin: Component: pkg:docker/library/hello-world@latest: The following required commands are unavailable: skopeo. Please install and try again.
post_stage_process
2 jobs succeeded, 1 failed
Failure Summary:
CollectDockerPlugin: The following required commands are unavailable: skopeo. Please install and try again.
GRAND TOTAL: 5 jobs succeeded, 2 failed
Artifact not found
When a requested artifact is not found, the resulting error may not be as clear as we would like. For example, I modified our SBOM to look for the docker image goodbye-world
rather than hello-world
:
Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
Beginning method process_component
CollectDockerPlugin FAIL for pkg:docker/library/goodbye-world@latest: Failure after 3 attempts, final message Skopeo failed to copy docker image to docker-archive:/tmp/tmpwl2rrutq/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
CollectGitPlugin SUCCESS for pkg:git/hoppr/hoppr.git
CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
Beginning method post_stage_process
CollectDockerPlugin SUCCESS
CollectGitPlugin SUCCESS
CollectRawPlugin SUCCESS
Stage Collect failed, processing terminated: 1 'process_component' processes failed
========== Results Summary ==========
Stage: Collect
process_component
3 jobs succeeded, 1 failed
Failure Summary:
CollectDockerPlugin: Component: pkg:docker/library/goodbye-world@latest: Failure after 3 attempts, final message Skopeo failed to copy docker image to docker-archive:/tmp/tmpwl2rrutq/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
post_stage_process
3 jobs succeeded, 0 failed
GRAND TOTAL: 6 jobs succeeded, 1 failed
From this, all we know is that skopeo
failed, despite trying three times (Hoppr typically re-tries up to three times in the event of a failure).
To find out more, we need to look in the log. The relevant section of the log (from a hopctl run with the -v
flag, to include DEBUG
log messages) tells us:
[2022-11-17 20:08:19,219] - [CollectDockerPlugin--44413-2] - [INFO] - ---- Component: goodbye-world@latest --------------------------------------------------
[2022-11-17 20:08:19,221] - [CollectDockerPlugin--44413-2] - [INFO] - Repository: https://docker.io
[2022-11-17 20:08:19,221] - [CollectDockerPlugin--44413-2] - [INFO] - Processing component [attempt 1 of 3]
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] - Copying docker image:
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] - source: docker://docker.io/library/goodbye-world:latest
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [INFO] - destination: docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest
[2022-11-17 20:08:19,222] - [CollectDockerPlugin--44413-2] - [DEBUG] - Running command: 'skopeo copy docker://docker.io/library/goodbye-world:latest docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest'
[2022-11-17 20:08:20,081] - [CollectDockerPlugin--44413-2] - [DEBUG] - skopeo command stdout content:
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [ERROR] - skopeo command failed with error:
time="2022-11-17T20:08:20Z" level=fatal msg="initializing source docker://goodbye-world:latest: reading manifest latest in docker.io/library/goodbye-world: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [DEBUG] - Skopeo failed to copy docker image to docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [INFO] - Artifact collection failed, deleting file and retrying
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [WARNING] - Method collect will be retried in 5 seconds
[2022-11-17 20:08:20,082] - [CollectDockerPlugin--44413-2] - [WARNING] - Result message for attempt 1: Skopeo failed to copy docker image to docker-archive:/tmp/tmp63pnwhg_/docker/https%3A%2F%2Fdocker.io/library/goodbye-world_latest, return_code=1
Dig through that, and you see that the actual error message returned by skopeo was "initializing source docker://goodbye-world:latest: reading manifest latest in docker.io/library/goodbye-world: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required". Which (it turns out) is one of the messages skopeo gives when a requested image is not found.
Component processed 0 times
I removed the collect_git_plugin
from the Transfer file, and got this output:
Beginning Hoppr Process execution, max_processes=2
Collecting Hoppr Metadata
========== Beginning Stage Collect ==================================================
Beginning method process_component
CollectDockerPlugin SUCCESS for pkg:docker/library/hello-world@latest
CollectRawPlugin SUCCESS for pkg:generic/~/.bashrc
CollectRawPlugin SUCCESS for pkg:generic/img/HopprHippo-01.png
Stage Collect FAIL for pkg:git/hoppr/hoppr.git: Component processed 0 times, EXACTLY_ONCE coverage required
Beginning method post_stage_process
CollectDockerPlugin SUCCESS
CollectRawPlugin SUCCESS
Stage Collect failed, processing terminated: 1 'process_component' processes failed
========== Results Summary ==========
Stage: Collect
process_component
3 jobs succeeded, 1 failed
Failure Summary:
Stage Collect: Component: pkg:git/hoppr/hoppr.git: Component processed 0 times, EXACTLY_ONCE coverage required
post_stage_process
2 jobs succeeded, 0 failed
GRAND TOTAL: 5 jobs succeeded, 1 failed
What this is telling us is that there was a component in the SBOM that none of the included collector plugins was capable of collecting.
Hoppr stages include an attribute indicating how frequently SBOM components are to be processed. When all of the plugins in a given stage are "collectors", that value defaults to EXACTLY_ONCE
.
Since the git collector was removed from the Transfer file, there was no collector available to collect the git component. It was not collected, resulting in this error.
Note that the Bundle stage did not run at all, because the Collect stage failed.