Flatpak External Data Checker
This is a tool for checking for outdated or broken links of external data in Flatpak manifests.
Motivation
Flatpak apps are built using external data (git checkouts, tarballs,
simple files, etc.). A very specific case of this is the use of extra
data, which works as a way to download third party binaries at
installation time.
Of course, the links pointing to external data can become obsolete, so
it is very important to account for and correct such issues.
This is especially critical for the extra data, in which a broken link
impedes the installation of the application.
This project offers ways to easily check or monitor the state of such
links, as well as the suggestion of new versions.
It works by extracting all the external data of a Flatpak manifest and
giving it to a collection of checkers, which will set up the right
state and, possibly, new versions of each external data.
Use
The simplest use of this tool is by calling:
flatpak-external-data-checker MANIFEST_FILE
it should display messages about any broken or outdated external data.
Installation
This tool itself is available in flatpak format from Flathub. Install with
flatpak install --from https://dl.flathub.org/repo/appstream/org.flathub.flatpak-external-data-checker.flatpakref
And run with
flatpak run org.flathub.flatpak-external-data-checker MANIFEST_FILE
Running in a container
flatpak-external-data-checker is also avaiable as an
OCI image
from GitHub Container Registry.
You can use the run-in-container.sh
helper script to set up needed CLI
options for you and run the image using podman
:
~/src/endlessm/flatpak-external-data-checker/run-in-container.sh \
[ARGS …] \
~/src/flathub/com.example.App/com.example.App.json
Automatically submitting PRs
When run with the --update
flag, this tool can commit any necessary changes
to Git and open a GitHub pull request. In order to do this, it requires
a GitHub access token,
specified in the GITHUB_TOKEN
environment variable.
Automatically merging PRs
The tool will also automatically merge previously opened pull request for
unavailable (BROKEN
) sources if the change has successfully passed CI checks
and the token has sufficient privileges.
Automatically merging all submitted PRs, not just unavailable sources,
from the tool can be forcefully enabled by setting
automerge-flathubbot-prs
to true
in flathub.json
,
or it can be completely disabled by setting automerge-flathubbot-prs
to false
.
Changes to Flatpak manifests
For simple checks to see if a URL is broken, no changes are needed. However,
you can add additional metadata to the manifest to allow the checker to
discover new versions.
Some of the following checkers are able to determine upstream version number,
and automatically add it to releases list in metainfo. To specify which source
is the app upstream source, set property is-main-source
to true
in the
checker metadata for that source.
URL checker
If the upstream vendor has an URL that redirects to the latest version of the
application, you can add something like the following to check and update the URL for
the latest version:
"x-checker-data": {
"type": "rotating-url",
"url": "http://example.com/last-version",
"pattern": "http://example.com/foo-v([0-9.]+).tar.gz"
}
The version number for the latest version can be detected in two ways:
- If the filename ends with
.AppImage
, the version number is extracted
from the AppImage. (It is run in abwrap
sandbox.) - Otherwise, if
"pattern"
is specified in"x-checker-data"
, the given
regular expression is matched against the full
URL for the latest version, and the first match group is taken to be the
version. (This follows the convention used by
debian/watch
files.)
HTML checker
Both the version number and the download URL will be gathered from a static
HTML page which contains this information:
"x-checker-data": {
"type": "html",
"url": "https://www.example.com/download.html",
"version-pattern": "The latest version is ([\\d\\.-]+)",
"url-pattern": "(https://www.example.com/pub/foo/v([\\d\\.-]+)/foo.tar.gz)"
}
The HTML checker also supports building the download URL using
the retrieved version string, its components according to the Python
LooseVersion
class and semantic versioning fields:
"x-checker-data": {
"type": "html",
"url": "https://www.example.com/download.html",
"version-pattern": "The latest version is ([\\d\\.-]*)",
"url-template": "https://www.example.com/$version/v$version.tar.gz"
}
"x-checker-data": {
"type": "html",
"url": "https://www.example.com/download.html",
"version-pattern": "The latest version is ([\\d\\.-]*)",
"url-template": "https://www.example.com/$major.$minor/v$version.tar.gz"
}
If the placeholder is immediately followed by an underscore, you need to add braces:
"x-checker-data": {
"type": "html",
"url": "https://www.example.com/download.html",
"version-pattern": "The latest version is ([\\d\\.-]*)",
"url-template": "https://www.example.com/$version0.$version1/v${version0}_${version1}_version2.tar.gz"
}
If the HTML page contains multiple versions with download links,
set single pattern containing two nested match groups for both url and version:
"x-checker-data": {
"type": "html",
"url": "https://sourceforge.net/projects/qrupdate/rss",
"pattern": "<link>(https://sourceforge.net/.+/qrupdate-([\\d\\.]+\\d).tar.gz)/download</link>"
}
To disable sorting and get first matched version/url, set sort-matches
to false
.
Git checker
To check for latest git tag in corresponding git source repo, add checker
metadata with type git
and set tag-pattern
to a regular expression with
exactly one match group (the pattern will be used to extract version from tag):
"x-checker-data": {
"type": "git",
"tag-pattern": "^v([\\d.]+)$"
}
By default tags are sorted based on version number extracted from tag.
To disable sorting and keep order from git ls-remote
, set sort-tags
to false
.
If the project follows semver specification, you can set
version-scheme
property to semantic
in order to use semantic version scheme for sorting.
In this case, make sure that tag-pattern
extracts only valid semver strings.
JSON checker
The JSON checker allows using jq to query
JSON data with arbitrary schema to get version and download url.
To use the JSONChecker, specify JSON data URL, version query and url query
(you can use $version
variable got from the version query in url query):
{
"type": "json",
"url": "https://api.github.com/repos/stedolan/jq/releases/latest",
"version-query": ".tag_name | sub(\"^jq-\"; \"\")",
"url-query": ".assets[] | select(.name==\"jq-\" + $version + \".tar.gz\") | .browser_download_url"
}
for git type sources, specify tag query and, optionaly, commit and version queries:
{
"type": "json",
"url": "https://api.github.com/repos/stedolan/jq/releases/latest",
"tag-query": ".tag_name",
"version-query": "$tag | sub(\"^jq-\"; \"\")",
"timestamp-query": ".published_at"
}
timestamp-query
is optional, but if provided - must return a string with timestamp in ISO format.
See the jq manual for complete information about writing queries.
Debian repo checker
For the DebianRepoChecker, which deals only with deb packages, it
can read the following metadata (add it to manifest element it refers
to, e.g. where "type": "extra-data" is declared):
"x-checker-data": {
"type": "debian-repo",
"package-name": "YOUR_PACKAGE_NAME",
"root": "ROOT_URL_TO_THE_DEBIAN_REPO",
"dist": "DEBIAN_DIST",
"component": "DEBIAN_COMPONENT"
}
Anitya (release-monitoring) checker
Anitya is an upstream release monitoring
project by Fedora. It supports multiple backends for retrieving version information
from different services, including GitHub, GitLab, Sourceforge, etc.
To use the AnityaChecker, specify numeric project ID on release-monitoring.org
and add a template for source download URL. This checker supports the same placeholders
as the HTMLChecker:
"x-checker-data": {
"type": "anitya",
"project-id": 6377,
"stable-only": false,
"url-template": "https://github.com/flatpak/flatpak/releases/download/$version/flatpak-$version.tar.xz"
}
Set stable-only
to true
to retrieve latest stable version (as recognized by Anitya).
For git type sources, instead of url-template
, set tag-template
to derive git tag from version.
GNOME checker
Check for latest source tarball for a GNOME project.
"x-checker-data": {
"type": "gnome",
"name": "pygobject",
"versions": {
"<": "3.38.0"
},
"stable-only": true
}
Set stable-only
to false
to check for pre-releases, too.
You can also set version constraints in versions
property (optional).
It should contain key-value pairs where key is the comparison operator
(one of <
, >
, <=
, >=
, ==
, !=
), and the value is the version to compare with.
So, {"<": "3.38.0", "!=": "3.37.1"}
means "any version less than 3.38.0 except 3.37.1".
All constraints must match simultaneously, i.e. if one doesn't match - version is rejected.
PyPI checker
Check for Python package updates on PyPI.
"x-checker-data": {
"type": "pypi",
"name": "Pillow"
}
By default it will check for source package (sdist
package type).
To check for binary package instead, set packagetype
to bdist_wheel
(only noarch wheels are supported currently).
JetBrains checker
Special checker that will check for available updates
for JetBrains products:
"x-checker-data": {
"type": "jetbrains",
"code": "PRODUCT-CODE",
"release-type": "release or eap (defaults to release)"
}
Snapcraft checker
Special checker that will check for available updates
for Snapcraft packages:
"x-checker-data": {
"type": "snapcraft",
"name": "PACKAGE-NAME",
"channel": "stable, beta, or any other tag the project uses"
}
Rust checker
Special checker that will check for available updates
for Rust:
"x-checker-data": {
"type": "rust",
"package": "package name, for example: rust",
"channel": "nightly, stable or beta",
"target": "target triple, for example: x86_64-unknown-linux-gnu"
}
Chromium checker
Special checker that will check for available updates to the
Chromium tarballs, as well as the toolchain
binaries or sources used to build it.
"x-checker-data": {
"type": "chromium",
"component": "chromium, llvm-prebuilt, or llvm-git; defaults to chromium"
}
The following components are supported:
chromium
: updates the Chromium tarball itself, used on URL-based sources
(e.g.type: archive
).llvm-prebuilt
: updates a tarball pointing to the official LLVM prebuilt
toolchain archive matching the latest Chromium version. Used on URL-based
sources.llvm-git
: updates atype: git
source for its commit to point to the LLVM
sources for the toolchain used by the latest Chromium version.