Quality Assessment of SBOM Generation Tools and Standards on Open Source Projects

With the increasing complexity of modern software composition and an ever-growing software supply chain, where numerous resources are sourced from open-source projects, the need to keep track of these resources has arisen. Following the Log4j incident [29], the American Government passed Executive Order (EO) 14028, mandating a SBOM for all software products sold to federal government agencies. Similarly, the European Union passed the Cyber Resiliance Act (CRA), which requires an SBOM for all digital products in the European market. For these reasons, machine-readable SBOM formats have emerged in recent years, implemented by a wide variety of projects that produce and consume such SBOMs.

This thesis investigates a collection of projects that generate SBOMs at various stages of the software development lifecycle. Each generator is applied to open-source projects to produce SBOMs. This study examines and compares the features provided by these SBOMs. Additionally, it assesses the completeness of the enumerated packages/components by analyzing the overlap among the SBOMs generated for each project.

The thesis highlights the distinctions between various tools and phases, highlighting potential bugs in the implementation of the tools investigated. It elucidates the variances in the SBOMs generated at different phases of the software development lifecycle. An analysis was conducted to identify which components of the CycloneDX and SPDX schemas were enriched with data during the SBOMs generation process. Furthermore, the research reveals that the tooling examined produces results of varying quality and depth. A metric was introduced to quantify the overlap among the diverse SBOMs, yielding mixed results. It is concluded that the quality and applicability of a produced SBOM can vary drastically depending on the use case. This variation is partly attributable to the different methodologies implemented by the investigated tools but also partly based on divergent results in the quality or depth of the generated SBOMs, where identifiers are produced in different ways or values are not sufficiently enriched.

The thesis aims to propose initial methods for validating the enrichment of a SBOMand assessing its completeness. This was done by testing the implemented generators on real-world projects.

Masterthesis.pdf

List of Sample Projects

The following is the sample list of all projects used in my master thesis.

Name	Container	Release	Source
adminer	here	here	here
adoptopenjdk	here
aerospike	here	here	here
almalinux	here
alpine	here		here
alt	here
amazoncorretto	here	here	here
amazonlinux	here
api-firewall	here	here	here
arangodb	here		here
archivebox	here	here	here
archlinux	here
babashka	here	here	here
backdrop	here	here	here
bash	here		here
biocontainers	here	here	here
bonita	here
btcpayserver	here	here	here
buildpack-deps	here		here
busybox	here		here
caddy	here	here	here
cassandra	here		here
celery	here	here	here
centos	here		here
chronograf	here	here	here
cirros	here	here	here
clamav	here		here
clearlinux	here
clojure	here		here
cloudprober	here	here	here
composer	here	here	here
consul	here	here	here
convertigo	here	here	here
coredns	here	here	here
couchbase	here
couchdb	here		here
crate	here	here	here
crossplane	here	here	here
crux	here	here	here
dart	here		here
debian	here
django	here		here
docker	here		here
domoticz	here	here	here
drupal	here		here
eclipse-mosquitto	here		here
eclipse-temurin	here
eggdrop	here	here	here
elasticsearch	here	here	here
elixir	here	here	here
emqx	here	here	here
erlang	here	here	here
esphome	here	here	here
euleros	here
express-gateway	here	here	here
farmos	here	here	here
fedora	here
flannel	here	here	here
flink	here		here
fluentd	here	here	here
fortio	here	here	here
freshrss	here	here	here
friendica	here	here	here
fsharp	here	here	here
gazebo	here	here	here
gcc	here		here
geonetwork	here	here	here
ghost	here	here	here
glassfish	here	here	here
golang	here		here
gradle	here	here	here
grafana	here	here	here
groovy	here		here
haproxy	here		here
haskell	here
haxe	here	here	here
hello-seattle	here
hello-world	here		here
hipache	here		here
hitch	here	here	here
hola-mundo	here
httpd	here		here
hylang	here	here	here
ibmjava	here	here	here
influxdb	here	here	here
iojs	here
irssi	here	here	here
jenkins	here	here	here
jetty	here	here	here
jobber	here	here	here
joomla	here
jruby	here	here	here
julia	here	here	here
jupyterhub	here		here
kaazing-gateway	here	here	here
kapacitor	here	here	here
keycloak	here	here	here
kibana	here	here	here
known	here	here	here
kong	here	here	here
libreddit	here	here	here
libretranslate	here	here	here
linkace	here	here	here
mageia	here
mariadb	here		here
matomo	here	here	here
mautic	here	here	here
maven	here	here	here
mediawiki	here		here
memcached	here		here
misskey	here	here	here
mitmproxy	here	here	here
mongo	here		here
mongo-express	here	here	here
monica	here	here	here
mono	here		here
mysql	here		here
nats	here	here	here
nats-streaming	here	here	here
neo4j	here	here	here
netdisco	here	here	here
neurodebian	here	here	here
nextcloud	here	here	here
nginx	here		here
node	here	here	here
nuxeo	here		here
odoo	here		here
okteto	here	here	here
open-liberty	here	here	here
openjdk	here		here
oraclelinux	here
orientdb	here	here	here
owncloud	here		here
passbolt	here	here	here
pegasus	here		here
percona	here	here	here
perl	here		here
pgrouting	here	here	here
photon	here	here	here
photoprism	here	here	here
php	here		here
php-zendserver	here		here
phpmyadmin	here	here	here
pihole	here	here	here
piwik	here	here	here
plone	here
portainer	here	here	here
postfixadmin	here	here	here
postgres	here		here
pypy	here
python	here		here
r-base	here
rabbitmq	here	here	here
rails	here	here	here
rakudo-star	here
rancher	here	here	here
rapidoid	here	here	here
rclone	here	here	here
redis	here	here	here
redmine	here		here
registry	here	here	here
rethinkdb	here	here	here
rocket.chat	here	here	here
rockylinux	here
ros	here		here
ruby	here	here	here
rust	here	here	here
sapmachine	here	here	here
satosa	here	here	here
scratch	here
searxng	here		here
sentry	here	here	here
shellspec	here	here	here
silverpeas	here		here
sl	here
solr	here		here
sonarqube	here	here	here
sonobuoy	here	here	here
sourcemage	here
spiped	here		here
sqlpad	here	here	here
steamcmd	here	here	here
storm	here		here
swarm	here	here	here
swift	here	here	here
swipl	here		here
teamspeak	here
telegraf	here	here	here
thrift	here	here	here
tomcat	here		here
tomee	here		here
traefik	here	here	here
ubuntu	here	here	here
ubuntu-debootstrap	here		here
ubuntu-upstart	here
unit	here		here
varnish	here
vault	here	here	here
websphere-liberty	here
wireguard-ui	here	here	here
wordpress	here		here
xwiki	here		here
yourls	here	here	here
znc	here		here
zookeeper	here		here