Troubleshooting Guide

# Troubleshooting Guide — ndaal SBOM Auditor

This guide helps operators and users find the root cause when the ndaal SBOM
Auditor does not behave as expected. It complements the Administrator Guide,
which holds the configuration details, and the User Guide, which explains the
normal workflow. Each problem lists the symptom, what to check, the likely
cause and the fix. Text in `code` is quoted exactly as the auditor prints it,
so you can search the log for it.

## First steps: collect evidence

**Version and platform.** Run `sbom-auditor --version`, or open **About** or
**System info** in the Info menu. **System info** (`/system-info`) also shows
the operating system, the architecture, the listen address and the uptime.
Compare the version with the changelog: several failures below are fixed in a
specific release.

**Health probes.** `-k` accepts the self-signed certificate:

```bash
curl -sk https://127.0.0.1:8680/healthz # answers: ok
curl -sk https://127.0.0.1:8680/readyz # answers: ready
```

`/readyz` answers `ready` whenever the process serves; it does not check the
database yet, so a green probe does not prove that the store is healthy.

**Logs.** The auditor logs to the console and appends to
`<log-dir>/sbom-auditor.log` (default `./ndaal/log`, resolved against the
working directory). `--log-format json` (`SBA_LOG_FORMAT=json`) switches both
to newline-delimited JSON. When the log file cannot be opened, the server keeps
running and warns `log file unavailable — logging to the console only`. The
default level is `info`; the standard `RUST_LOG` filter changes it. This adds
the auditor's own debug messages, such as `TLS handshake failed`,
`TLS handshake timed out` and `enrichment disabled in settings; skipped`:

```bash
RUST_LOG=info,sbom_auditor=debug sbom-auditor --port 8680
```

A healthy start-up logs, in this order: `logging to console + file`;
`auditor store ready` with the `db` path and whether it is `encrypted`;
`SBOM byte storage ready`; and `ndaal SBOM Auditor listening on https://…`,
ending in `self-signed certificate` or `operator-supplied certificate`.

**Data directory and run diagnostics.** Everything stateful lives in the data
directory (`--data-dir`, default `./data`): the SQLite database `auditor.db`
with its WAL files, and the stored SBOM bytes under `sboms/`. The run table on
an SBOM's detail page shows a skip reason or an exit code, but not the
diagnostic text of a failed run. Read that from the database in read-only
mode, with the id from the page address `/sbom/{id}`:

```bash
sqlite3 -readonly data/auditor.db \
"SELECT tool, status, exit_code, skip_reason, stderr_text
FROM analysis_runs WHERE sbom_id = 42 ORDER BY id;"
```

## Start-up and listening

### The server exits during start-up

- **Symptom:** the process stops at once with an `Error:` line and a non-zero
exit status.
- **Check:** the message; it names the cause.
- **Likely cause and fix:** see the table. A failure to create the data
directory or open the database also stops start-up; see
[SQLite store and persistence](#sqlite-store-and-persistence).

| Message contains | Cause | Fix |
| --- | --- | --- |
| `unknown option:`, `needs a value` | Mistyped flag, or a flag without its value | Compare with `sbom-auditor --help`. |
| `invalid port:`, `invalid SBA_PORT:` | Not a port in 1025 to 65535 (0 to 1024 are refused) | Choose a port from 1025 up; to serve on 443, put a reverse proxy in front. There is no silent default. |
| `refusing to bind non-loopback address` | `--bind` is not a loopback IP address | See [Non-loopback bind refused](#non-loopback-bind-refused). |
| `--tls-cert was given without --tls-key` | Only one half of the certificate pair | Pass both files (the reverse message exists too). |
| `is an RSA (PKCS#1) key` | RSA key supplied | Use an ECDSA (P-256 or P-384) or Ed25519 key. |
| `is group/other-accessible` | Key file readable by other users | `chmod 600` the key file. |
| `no CERTIFICATE block found`, `invalid or missing private key PEM` | Wrong or damaged PEM file | Check the files behind `--tls-cert` and `--tls-key`. |
| `must contain exactly 32 raw bytes or 64 hex characters` | Malformed storage encryption key | Supply a valid key file. |

### Nothing listens on the port

- **Symptom:** connections are refused; the log has no `listening on` line.
- **Check:** whether the process still runs, and who holds the port:
`lsof -nP -iTCP:8680 -sTCP:LISTEN`.
- **Likely cause and fix:**
- The process exited after `SBOM byte storage ready` with an address-in-use
error: another service holds the port. Stop it, or choose a free port with
`--port` (`SBA_PORT`) and keep `--allowed-host` in step.
- The process still runs: the search index is rebuilt from every stored SBOM
before the port is bound, so start-up time grows with the store. Wait, and
give health checks a matching grace period.

### Non-loopback bind refused

- **Symptom:** `refusing to bind non-loopback address '…': the auditor has no
authentication.`
- **Check:** the value of `--bind` or `SBA_BIND`.
- **Likely cause:** the auditor ships no authentication, so it binds loopback
only unless told otherwise. `--bind` takes a bare IP address; a host name
such as `localhost` is not recognised as loopback and is refused as well.
- **Fix:** bind `127.0.0.1`, or put an authenticating reverse proxy in front
and pass `--allow-non-loopback`, as "Exposing the auditor on a network" in
the Administrator Guide describes. The server then logs a warning starting
with `binding a NON-LOOPBACK address`.

## Reaching the UI in a browser

### The browser cannot connect

- **Symptom:** a connection error instead of a page.
- **Check:** the address starts with `https://` and uses the `--port` value
(default 8680); on the server, the `/healthz` probe answers.
- **Likely cause:** the auditor serves HTTPS only, with no plaintext listener
and no redirect, so `http://127.0.0.1:8680/` fails. It accepts TLS 1.3 only,
so older clients fail too. Port 8681 is reserved for a future HTTP/3
listener; nothing listens there yet.
- **Fix:** open `https://127.0.0.1:8680/` in a current browser.

### The browser warns about the certificate

- **Symptom:** the browser reports an untrusted certificate.
- **Check:** the `listening on` line ends in `self-signed certificate`.
- **Likely cause:** without `--tls-cert` and `--tls-key` the auditor presents a
self-signed ECDSA certificate, valid for 45 days and issued for `localhost`,
`127.0.0.1` and `::1` only. It is regenerated at every start-up, so the
warning can return after a restart; any other host name adds a mismatch.
- **Fix:** accept it for local use, or present a CA-issued certificate as
"Operator certificates" in the Administrator Guide describes.

### Requests fail with `403` and `host not allowed`

- **Symptom:** every request, health probes included, answers `403` with the
body `host not allowed`, typically behind a proxy, a load balancer or on a
LAN address.
- **Check:** the `Host` header (HTTP/2: `:authority`) the request carries, for
example with `curl -skv`.
- **Likely cause:** the anti-DNS-rebinding guard accepts only the bind address
with its port, `127.0.0.1`, `localhost` or `[::1]` on the bound port, and the
`--allowed-host` entries. The comparison is an exact string match, so
`auditor.example` and `auditor.example:443` are different entries.
- **Fix:** list every authority the clients and the proxy actually send, for
example `--allowed-host auditor.example,auditor.example:443`
(`SBA_ALLOWED_HOSTS`).

### Form submits fail with `cross-origin request rejected`

- **Symptom:** an upload, analyze, import, export or settings submit answers
`403` with `cross-origin request rejected`.
- **Check:** the `Origin`, `Sec-Fetch-Site` and `Host` headers of the request.
- **Likely cause:** every state-changing `POST` must be same-origin
(`Sec-Fetch-Site` of `same-origin` or `none`, else an `https://` `Origin`
equal to the request's host and port). A proxy that rewrites `Host`, or a
script that sends a foreign `Origin`, breaks this.
- **Fix:** let the proxy forward the original `Host` (allowed with
`--allowed-host`); in scripts, send no `Origin` or the auditor's own. A `403`
with an empty body and no security headers comes from a WAF in front (see
"Status codes you will actually hit" in the Developer Guide).

## Uploads and imports

### An upload is rejected

- **Symptom:** submitting a file does not lead to the SBOM's detail page.
- **Check:** the HTTP status and the message.
- **Likely cause and fix:** see the table, and size the limit against the RAM
(see [Behaviour under heavy load](#behaviour-under-heavy-load)).

| Status and message | Cause | Fix |
| --- | --- | --- |
| `413`, `request body too large` | Larger than `upload_max_bytes` (per request, default 10 GiB), or the transfer broke off | Raise the limit on **Settings**, split the input, or retry. |
| `408`, `Request Timeout` | Upload plus ingest took longer than 10 minutes | Use a faster path, or copy the file to the import folder and import it. |
| `400`, `the uploaded file is empty` | Zero-byte file | Upload the real file. |
| `400`, `no file part …`, `multipart boundary missing …` | A script sent a malformed form | Send `multipart/form-data` with one part named `file`. |
| `422`, `the file is not a recognised SBOM format` | Not CycloneDX, SPDX or syft JSON; a scanner's vulnerability report is not an SBOM | Upload the SBOM itself. |
| `500`, `storing the upload failed` | Database or disk write failed; the log shows `upload ingest failed` | Check free space and write access to the data directory. |

### The settings page will not save

- **Symptom:** **Save settings** does nothing and the browser marks a field,
or a scripted save answers `422` with a message naming the field.
- **Check:** the marked field and the message. Every rejection names its field
and, for numbers, the accepted range, for example
`upload_max_bytes: … is outside 1024..=10737418240`,
`analyzer_timeout_secs: … is outside 10..=86400`,
`vulnlookup_source_1: must be an https:// URL` or
`csaf_publisher_name: must not be empty`.
- **Likely cause:** a value outside the accepted range, or an empty required
field. In 1.6.26 to 1.6.28 the form accepted at most 1 GiB for the upload
limit while the built-in default was 10 GiB, so a store whose
settings were never saved could not be saved as shown; 1.6.29 accepts up to
10 GiB (10737418240 bytes).
- **Fix:** correct the named field and save again. On 1.6.26 to 1.6.28, enter
at most 1073741824 for **Maximum upload size (bytes)**, or upgrade.

### An import ingests fewer files than expected

- **Symptom:** **Scan import folder** reports fewer SBOMs than the folder
holds, or only an error.
- **Check:** the **Import result**: the **Imported**, **Deduplicated** and
**Skipped** counts, with one reason per skipped file under **Details**.
- **Likely cause and fix:**
- Only files ending in `.json`, `.xml`, `.spdx`, `.cdx` or `.sbom` (any
case) are considered; other files are ignored without a reason.
- `invalid import directory:` — the path is empty, longer than 4096 bytes,
holds a NUL byte, starts with `~` or contains a `..` component.
- `cannot open import directory` — the folder is missing or unreadable for
the service account; a relative path resolves against the working
directory.
- `file exceeds the N-byte import limit` — larger than `upload_max_bytes`.
- `unrecognised SBOM format` — as for uploads.
- **Deduplicated** files already exist with identical bytes; not an error.

## Analysis

### An analyzer shows `skip`

- **Symptom:** a run ends in `skip`; the **Detail** column gives the reason.
- **Check:** the **Tools** page (`/tools`) probes every tool from the server
process. **Binary** shows the name or path the auditor resolves; **Present**
reads `Not found` when the probe fails.
- **Likely cause and fix:**
- `<tool>: not found on PATH` — the server process could not start the
tool: it is not on the service's `PATH`, not executable, or its version
probe gave no answer within 15 seconds. Install it for the service
account, or set `SBA_TOOL_<NAME>_BIN` (for example `SBA_TOOL_GRYPE_BIN`)
to an absolute path and restart; the variables come from the process
environment.
- A reason ending in `syft is not installed to convert` — bomber needs
CycloneDX 1.6 or older and no scanner reads SPDX 3.0 directly, so the
input must be converted with `syft convert`. Install syft.

### An analyzer shows `fail`

- **Symptom:** a run ends in `fail`, with only an exit code or nothing in
**Detail**.
- **Check:** `stderr_text` from the `sqlite3` query in
[First steps](#first-steps-collect-evidence), and the server log.
- **Likely cause and fix:**
- `spawn failed: …` — the operating system refused to start the tool.
- `could not materialise the SBOM input: …` — the stored SBOM could not be
read or copied to a scratch directory under the system temporary
directory. With storage encryption, `sealed value could not be decrypted`
means a wrong key; see [Storage encryption](#storage-encryption-errors).
- grype stops working once its database is older than its maximum age
(5 days). The auditor runs `grype db update` once per process, before the
first grype scan, so an offline host or a long-running server can still
hold a stale database; the log then shows `grype DB warm-up timed out` or
`grype DB warm-up failed to spawn`. Run `grype db update` as the service
account, or restart the auditor.
- trivy scans offline with database updates switched off, so it needs a
database populated beforehand for the service account.
- osv-scanner queries the online OSV database and needs network access.
- Output larger than 32 MiB is cut off, no longer parses as JSON and counts
as `fail`.

### An analyzer shows `timeout`

- **Symptom:** a run ends in `timeout`.
- **Check:** **Analyzer timeout (seconds)** on **Settings**.
- **Likely cause:** the tool ran longer than `analyzer_timeout_secs` (default
300) and was stopped. The same limit bounds the `syft convert` step and the
grype database update.
- **Fix:** raise the timeout (10 to 86400 seconds) for very large SBOMs.

### Runs stay `pending` or `running`

- **Symptom:** the run table keeps polling every 2 seconds and never finishes.
- **Check:** the log line `analysis triggered` with its `queued` count, and
whether the server restarted during the analysis.
- **Likely cause and fix:**
- At most `analyzer_max_concurrency` tools (default 2, range 1 to 8) run at
once across all analyses; the rest wait as `pending`. A change to this
setting takes effect after a restart.
- A restart interrupted the analysis, and its runs are never finished.
Start the analysis again; the page shows the newest run of each tool.
- `queued=0` means no analyzer is enabled on **Settings**, so nothing runs
and no enrichment follows.

## Vulnerability enrichment

### Findings show `Not enriched`

- **Symptom:** the KEV column reads `Not enriched` for some or all findings.
- **Check:** enrichment starts only after every run of the analysis has ended,
so wait for the last run. Then search the log, and reproduce the request on
the server host as the service account, because the lookups come from the
server process, not from the browser:

```bash
grep -E 'post-analysis enrichment|enrichment lookup failed|no source is configured' \
ndaal/log/sbom-auditor.log
curl -sS --tlsv1.3 --tls-max 1.3 -o /dev/null -w '%{http_code}\n' \
https://vulnlookup.ndaal.eu:443/api/v1/vulnerability/CVE-2021-44228/enrichment
```

For `curl`, `200` means the path works, `000` that the host cannot reach the
source, and a certificate error that the CA bundle is missing or unreadable.
- **Likely cause and fix:** read the counts of
`post-analysis enrichment finished`, then the table.
- `candidates` — distinct CVE ids in the SBOM's findings; `0` means no
finding has a CVE id, so there is nothing to look up.
- `fresh_skipped` — answered from cached entries younger than 24 hours.
- `looked_up` and `stored` — answered by a source, and written to the cache.
The difference are CVEs no source knows (answered `404`); they stay
`Not enriched` and are asked again next time.
- `failed` — no source could answer, or the cache write failed; each is
logged as `enrichment lookup failed` with an `error` field.

| Log message | Cause | Fix |
| --- | --- | --- |
| No enrichment line at all | Enrichment is off (logged at debug level only) | Switch it on under **Vulnerability enrichment** on **Settings**. |
| `enrichment is enabled but no source is configured` | All three source URLs are blank | Enter at least one source. |
| `post-analysis enrichment failed; findings stay un-enriched` with `no system root certificates available` | No readable CA bundle on the host | Install it; the next pass picks it up without a restart. |
| `enrichment request to … timed out` | No answer within 15 seconds (5 to connect) | Check routing and firewalls from the server host. |
| `enrichment source … answered <code>` | An error status; redirects are not followed | Point the source at the vulnlookup host and its HTTPS port. |
| `enrichment request error: …` | DNS, connection or TLS failure, including an untrusted certificate | Run the `curl` check; fix resolution, reachability or trust. |
| `enrichment parse error: …` | Something other than vulnlookup answered on that port | Correct the source URL or TCP port. |

Three configuration details explain most wrong-source problems:

- Only the scheme and host of a source URL are used. A path in the URL is
dropped and a port written into it is ignored: the port comes from the
**TCP port** field (blank means 443). The URL must start with `https://`.
- The sources are tried in order, and when none answers, the log reports the
error of the last one. Source 3 defaults to `https://127.0.0.1/` on port
8680, the auditor's own default port, so when every source fails the logged
error names that loopback address, not source 1. Test source 1 with the
`curl` check, and blank source 3 when no vulnlookup runs on this host.
- A CVE that was found is not asked again for 24 hours. A failed lookup is not
cached, so after fixing the cause, analyse the SBOM again to retry.

On macOS, releases before 1.6.28 could crash with `SIGSEGV` in
`SecTrustSettingsCopyTrustSettings` when two analyses finished together, or
leave findings `Not enriched`; upgrade to 1.6.28 or later.

## Search and the index rebuild

### The dashboard search finds nothing

- **Symptom:** a search on the home page returns no SBOMs where you expect
one.
- **Check:** the query, and the start-up log for search messages.
- **Likely cause and fix:**
- The search matches whole words: `log4j` matches, `log4` does not. Use the
**Components** page for part of a component name or an exact package URL.
- A query longer than 1024 bytes returns nothing.
- `search rebuild failed; search is empty until re-ingest` means the
in-memory index could not be built at start-up. The dashboard keeps
working without it; restart the auditor to rebuild the index, or
re-ingest the SBOMs.

### The Components page says the search is incomplete

- **Symptom:** **Components** shows "This search is incomplete".
- **Check:** the number of SBOMs it reports as not yet indexed.
- **Likely cause:** those SBOMs have no component index yet, so they cannot
appear in results, and "not found" would wrongly read as "not affected".
- **Fix:** click **Rebuild index** and trust results only once the warning is
gone. Releases before 1.6.25 kept the warning for SBOMs without components
after every rebuild; upgrade if it never clears.

## Exports

- **Symptom:** **Export** returns to the form with an error, a download link
answers `404`, or the export directory fills faster than expected.
- **Check:** the message on the export page.
- **Likely cause and fix:**
- `Select at least one format to export.` — tick at least one format.
- `… already exists in the export directory` — tick
**Overwrite existing files**, or remove the old files.
- `export directory error: …` — `export_dir` breaks the same path rules as
the import directory.
- `write failed: …` — the export directory is not writable or the disk is
full. **Start download** streams the same selection to the browser and
writes nothing on the server, so it works without a writable directory.
- `no such export file` (`404` from `GET /export/download`) — only files
that exist in the export directory and carry an export or sidecar
extension are served. Export again, or use **Start download**.
- Many files: each artifact gets up to five checksum sidecars, and with
`csaf_split_per_finding` on, both CSAF versions add one advisory per
finding; "How many files one export writes" in the Administrator Guide
gives the arithmetic.

## SQLite store and persistence

### Stored SBOMs are missing after a restart

- **Symptom:** the dashboard is empty although SBOMs were stored before.
- **Check:** the `db` path in the `auditor store ready` line.
- **Likely cause:** the defaults `./data` and `./ndaal/log` are relative to the
working directory. Started elsewhere, or with another `--data-dir` or
`--db`, the auditor creates a new, empty store there.
- **Fix:** pass absolute `--data-dir` and `--log-dir` paths, as a service unit
should.

### The database cannot be opened

- **Symptom:** start-up stops after `logging to console + file`, before
`auditor store ready`.
- **Check:** ownership, permissions and free space of the data directory.
- **Likely cause:** the service account cannot create the data directory or
open `auditor.db`, or the disk is full. An older database is migrated
forward automatically, so this is not an upgrade step you missed.
- **Fix:** correct the directory's ownership and permissions, and back it up
before upgrading (see "Backup and recovery" in the Administrator Guide). Run
one auditor process per database: the store is a single SQLite connection in
WAL mode that waits at most 5 seconds for a lock.

### Storage encryption errors

- **Symptom:** start-up stops with `encryption key error:`, or analyses fail
and the log shows `stored SBOM is sealed but could not be decrypted`.
- **Check:** the file `--encryption-key-file` points to.
- **Likely cause:** a missing or malformed key file stops start-up. A valid but
different key opens the store, but every sealed SBOM and tool output then
fails to decrypt; the auditor fails closed instead of reporting an empty,
clean SBOM.
- **Fix:** restart with the original key file. Treat a decryption failure as a
key mismatch or tampering to investigate, never as a clean result.

## Self-update

- **Symptom:** `--check-update` or `--self-update` does not install a release.
- **Check:** the message it prints.
- **Likely cause and fix:**
- `sbom-auditor <version> is up to date.` — nothing newer is published; a
build ahead of the latest release is never downgraded.
- `Could not reach the update host; leaving the current version unchanged.`
— gitlab.com is not reachable over TLS 1.3 from this host.
- `--self-update is disabled by policy` — `--no-self-update` or
`SBA_NO_SELF_UPDATE` is set; update through your package manager.
- `no SHA-256 for <asset> in the release SHA256SUMS` — no published binary
matches this build's target. Releases cover `x86_64-unknown-linux-musl`,
`aarch64-unknown-linux-musl`, `x86_64-apple-darwin`,
`aarch64-apple-darwin`, `x86_64-pc-windows-gnu` and
`aarch64-pc-windows-gnullvm`; reinstall with `cargo install sbom-auditor`
or `cargo binstall sbom-auditor` instead.
- `no system root certificates available to verify the update host` — the
host has no readable CA bundle.

The update replaces the binary in place and exits; a running server keeps the
old code until you restart it.

## Behaviour under heavy load

- **Symptom:** under many concurrent users or uploads, connections drop,
requests time out or analyses queue up.
- **Check:** the log for the messages in the table.
- **Likely cause:** fixed limits make load degrade into refusals instead of
exhausting memory.
- **Fix:** lower `upload_max_bytes` to fit the RAM, because every in-flight
upload is buffered in full. Repeated `accept failed` warnings point to
file-descriptor exhaustion: raise the service's open-files limit. Raise the
analyzer concurrency only as far as CPU and memory allow, then restart.

| Limit | Value | What you see |
| --- | --- | --- |
| Open connections | 512 | Further connections are closed; `connection limit reached — shedding connection` |
| TLS handshake | 10 seconds | Connection dropped; `TLS handshake timed out` at debug level |
| Request head (HTTP/1.1) | 30 seconds | Connection closed |
| Whole request, body included | 120 seconds; `POST /upload` 10 minutes | `408` `Request Timeout` |
| Analyzer tools at once | `analyzer_max_concurrency` (1 to 8) | Runs wait as `pending` |
| Enrichment lookups at once | Same setting, at most 16 | Enrichment takes longer |

## Language and theme

- **The language does not stick.** The language menu sets the `sba_lang`
cookie (`Secure`, `HttpOnly`, kept for one year), which wins over the
browser's `Accept-Language`, which wins over English. Releases before 1.4.19
missed the cookie when an HTTP/2 browser split its cookies across several
headers; upgrade. A string missing from a translation falls back to English
on its own; the generated reports (SARIF, CSAF, PDF) and the HTTP API are
English-only by design.
- **The theme resets to light.** The theme is stored only in the browser,
under the `localStorage` key `sba-theme`. When the browser blocks storage
(private windows, strict privacy settings), each page load falls back to the
light theme; allow storage for the auditor's address.

## Known open issues

The 1.6.28 changelog lists two: no finding is enriched on the public instance
sbom.ndaal.eu (a cause on that host; local instances enrich the same SBOM), and
some CVEs are matched against the operating-system release rather than a
package, which usually does not apply (see "Step 3 — review findings" in the
User Guide).

## Reporting a problem

Open an issue in the project repository at
<https://gitlab.com/vPierre/ndaal_public_sbom_auditor> and include:

- The version (`sbom-auditor --version`) and the platform from **System info**.
- How you installed it (`cargo install`, `cargo binstall` or a source build)
and the start command with its flags and `SBA_*` variables.
- The relevant log lines with timestamps (at debug level if reproducible); for
analysis, the **Tools** page and the `sqlite3` run rows; for enrichment, the
`post-analysis enrichment finished` line and the server-side `curl` result.
- The steps to reproduce, what you expected and what happened instead.

Logs and SBOMs can reveal host names, file names and your component inventory;
review them before you share them. Do not report a suspected security
vulnerability in a public issue: send it to <security@ndaal.eu> as described in
`SECURITY.md` and `Vulnerability_Disclosure_Policy.md`.