From 6e80675573a63332a2625935e305f62cd7af497d Mon Sep 17 00:00:00 2001 From: nicole mazzuca Date: Thu, 6 May 2021 12:13:29 -0700 Subject: [PATCH] [docs registries] Add reference documentation (#17672) * using registries reference documentation * start work on "creating registries" document also fix minor issue in "using registries" doc * Robert's comments, continue working * finish creating registries docs * add links to example registries * aupopa cr --- .gitignore | 1 + docs/README.md | 2 + docs/maintainers/registries.md | 357 +++++++++++++++++++++++++++++++++ docs/users/registries.md | 135 +++++++++++++ 4 files changed, 495 insertions(+) create mode 100644 docs/maintainers/registries.md create mode 100644 docs/users/registries.md diff --git a/.gitignore b/.gitignore index 15b8c8b6dd..6aff3dc65d 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ *.user *.userosscache *.sln.docstates +/vcpkg-configuration.json # fuzzing sync_dir* diff --git a/docs/README.md b/docs/README.md index a0a53b1cc0..95b26f903a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -23,6 +23,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too - [Usage with Android](users/android.md) - [Usage with Mingw-w64](users/mingw.md) - [Host Dependencies](users/host-dependencies.md) +- [Using Registries](users/registries.md) ### Maintainer Help @@ -31,6 +32,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too - [Portfile functions](maintainers/portfile-functions.md) - [Common CMake definitions](maintainers/vcpkg_common_definitions.md) - [Maintainer Guidelines](maintainers/maintainer-guide.md) +- [Creating Registries](maintainers/registries.md) ### [Vcpkg-Tool](https://github.com/microsoft/vcpkg-tool) Maintainer Help diff --git a/docs/maintainers/registries.md b/docs/maintainers/registries.md new file mode 100644 index 0000000000..bef296ea36 --- /dev/null +++ b/docs/maintainers/registries.md @@ -0,0 +1,357 @@ +# Creating Registries + +**The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/maintainers/registries.md).** + +There are two parts to using registries; this documents the creation side of +the relationship. In order to learn more about using registries that others +have created, please read [this documentation](../users/registries.md). +## Table of Contents + +- [Creating Registries](#creating-registries) + - [Table of Contents](#table-of-contents) + - [Overview](#overview) + - [Git Registries](#git-registries) + - [Adding a New Version](#adding-a-new-version) + - [Filesystem Registries](#filesystem-registries) + - [Adding a New Version](#adding-a-new-version-1) + +## Overview + +Registries are collections of ports and their versions. There are two major +choices of implementation for registries, if you want to create your own - +git registries, and filesystem registries. + +Git registries are simple git repositories, and can be shared publicly or +privately via normal mechanisms for git repositories. The vcpkg repository at +, for example, is a git registry. + +Filesystem registries are designed as more of a testing ground. Given that they +literally live on your filesystem, the only way to share them is via shared +directories. However, filesystem registries can be useful as a way to represent +registries held in non-git version control systems, assuming one has some way +to get the registry onto the disk. + +Note that we expect the set of registry types to grow over time; if you would +like support for registries built in your favorite public version control +system, don't hesitate to open a PR. + +The basic structure of a registry is: + +- The set of versions that are considered "latest" at certain times in history, + known as the "baseline". +- The set of all the versions of all the ports, and where to find each of + these in the registry. + +### Git Registries + +As you're following along with this documentation, it may be helpful to have +a working example to refer to. We've written one and put it here: +. + +All git registries must have a `versions/baseline.json` file. This file +contains the set of "latest versions" at a certain commit. It is laid out as +a top-level object containing only the `"default"` field. This field should +contain an object mapping port names to the version which is currently the +latest. + +Here's an example of a valid baseline.json: + +```json +{ + "default": { + "kitten": { + "baseline": "2.6.2", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 2 + } + } +} +``` + +The `versions` directory contains all the information about which versions of +which packages are contained in the registry, along with where those versions +are stored. The rest of the registry just acts as a backing store, as far as +vcpkg is concerned: only things inside the `versions` directory will be used +to direct how your registry is seen by vcpkg. + +Each port in a registry should exist in the versions directory as +`-/.json`; in other words, the +information about the `kitten` port would be located in +`versions/k-/kitten.json`. This should be a top-level object with only a +single field: `"versions"`. This field should contain an array of version +objects: + +- The version of the port in question; should be exactly the same as the + `vcpkg.json` file, including the version fields and `"port-version"`. +- The `"git-tree"` field, which is a git tree; in other words, what you get + when you write `git rev-parse COMMIT-ID:path/to/port`. + +Note that the version fields for ports with `CONTROL` files, is +`"version-string"`; we do not recommend using `CONTROL` files in new +registries, however. + +_WARNING_: One very important part of registries is that versions should +_never_ be changed. Updating to a later ref should never remove or change an +existing version. It must always be safe to update a registry. + +Here's an example of a valid version database for a `kitten` port with one +version: + +```json +{ + "versions": [ + { + "version": "2.6.2", + "port-version": 0, + "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e" + } + ] +} +``` + +In general, it's not important where you place port directories. However, the +idiom in vcpkg is to follow what the built in vcpkg registry does: your +`kitten` port should be placed in `ports/kitten`. + +_WARNING_: One other thing to keep in mind is that when you update a registry, +all previous versions should also be accessible. Since your user will set their +baseline to a commit ID, that commit ID must always exist, and be accessible +from your HEAD commit, which is what is actually fetched. This means that your +HEAD commit should be a child of all previous HEAD commits. + +#### Adding a New Version + +There is some git trickery involved in creating a new version of a port. The +first thing to do is make some changes, update the `"port-version"` and regular +version field as you need to, and then test with `overlay-ports`: +`vcpkg install kitten --overlay-ports=ports/kitten`. + +Once you've finished your testing, you'll need to make sure that the directory +as it is is under git's purview. You'll do this by creating a temporary commit: + +```pwsh +> git add ports/kitten +> git commit -m 'temporary commit' +``` + +Then, get the git tree ID of the directory: + +```pwsh +> git rev-parse HEAD:ports/kitten +73ad3c823ef701c37421b450a34271d6beaf7b07 +``` + +Then, you can add this version to the versions database. At the top of your +`versions/k-/kitten.json`, you can add (assuming you're adding version +`2.6.3#0`): + +```json +{ + "versions": [ + { + "version": "2.6.3", + "port-version": 0, + "git-tree": "73ad3c823ef701c37421b450a34271d6beaf7b07" + }, + { + "version": "2.6.2", + "port-version": 0, + "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e" + } + ] +} +``` + +then, you'll want to modify your `versions/baseline.json` with your new version +as well: + +```json +{ + "default": { + "kitten": { + "baseline": "2.6.3", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 2 + } + } +} +``` + +and amend your current commit: + +```pwsh +> git commit --amend +``` + +then share away! + +### Filesystem Registries + +As you're following along with this documentation, it may be helpful to have +a working example to refer to. We've written one and put it here: +. + +All filesystem registries must have a `versions/baseline.json` file. This file +contains the set of "latest versions" for a certain version of the registry. +It is laid out as a top-level object containing a map from version name to +"baseline objects", which map port names to the version which is considered +"latest" for that version of the registry. + +Filesystem registries need to decide on a versioning scheme. Unlike git +registries, which have the implicit versioning scheme of refs, filesystem +registries can't rely on the version control system here. One possible option +is to do a daily release, and have your "versions" be dates. + +_WARNING_: A baseline must always refer to the same set of versions. If you +want to add new versions, you need to create a new version of the registry in +the `baseline.json` file. + +Here's an example of a valid `baseline.json`, for a registry that has decided +upon dates for their versions: + +```json +{ + "2021-04-16": { + "kitten": { + "baseline": "2.6.2", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 2 + } + }, + "2021-04-15": { + "kitten": { + "baseline": "2.6.2", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 1 + } + } +} +``` + +The `versions` directory contains all the information about which versions of +which packages are contained in the registry, along with where those versions +are stored. The rest of the registry just acts as a backing store, as far as +vcpkg is concerned: only things inside the `versions` directory will be used +to direct how your registry is seen by vcpkg. + +Each port in a registry should exist in the versions directory as +`-/.json`; in other words, the +information about the `kitten` port would be located in +`versions/k-/kitten.json`. This should be a top-level object with only a +single field: `"versions"`. This field should contain an array of version +objects: + +- The version of the port in question; should be exactly the same as the + `vcpkg.json` file, including the version fields and `"port-version"`. +- The `"path"` field: a relative directory, rooted at the base of the registry + (in other words, the directory where `versions` is located), to the port + directory. It should look something like `"$/path/to/port/dir`" + +Note that the version fields for ports with `CONTROL` files, is +`"version-string"`; we do not recommend using `CONTROL` files in new +registries, however. + +In general, it's not important where you place port directories. However, the +idiom in vcpkg is to follow somewhat closely to what the built in vcpkg +registry does: your `kitten` port at version `x.y.z` should be placed in +`ports/kitten/x.y.z`, with port versions appended as you see fit (although +since `#` is not a good character to use for file names, perhaps use `_`). + +_WARNING_: One very important part of registries is that versions should +_never_ be changed. One should never remove or change an existing version. +Your changes to your registry shouldn't change behavior to downstream users. + +Here's an example of a valid version database for a `kitten` port with one +version: + +```json +{ + "versions": [ + { + "version": "2.6.2", + "port-version": 0, + "git-tree": "$/ports/kitten/2.6.2_0" + } + ] +} +``` + +#### Adding a New Version + +Unlike git registries, adding a new version to a filesystem registry mostly +involves a lot of copying. The first thing to do is to copy the latest +version of your port into a new version directory, update the version and +`"port-version"` fields as you need to, and then test with `overlay-ports`: +`vcpkg install kitten --overlay-ports=ports/kitten/new-version`. + +Once you've finished your testing, you can add this new version to the top of +your `versions/k-/kitten.json`: + +```json +{ + "versions": [ + { + "version": "2.6.3", + "port-version": 0, + "git-tree": "$/ports/kitten/2.6.3_0" + }, + { + "version": "2.6.2", + "port-version": 0, + "git-tree": "$/ports/kitten/2.6.2_0" + } + ] +} +``` + +then, you'll want to modify your `versions/baseline.json` with your new version +as well (remember not to modify existing baselines): + +```json +{ + "2021-04-17": { + "kitten": { + "baseline": "2.6.3", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 2 + } + }, + "2021-04-16": { + "kitten": { + "baseline": "2.6.2", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 2 + } + }, + "2021-04-15": { + "kitten": { + "baseline": "2.6.2", + "port-version": 0 + }, + "port-b": { + "baseline": "19.00", + "port-version": 1 + } + } +} +``` + +and you're done! \ No newline at end of file diff --git a/docs/users/registries.md b/docs/users/registries.md new file mode 100644 index 0000000000..53db7cb1f4 --- /dev/null +++ b/docs/users/registries.md @@ -0,0 +1,135 @@ +# Using Registries + +**The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/users/registries.md).** + +There are two parts to using registries; this documents the use side of the +relationship. In order to learn more about creating registries for others to +use, please read [this documentation](../maintainers/registries.md). + +## Table of Contents + +- [Using Registries](#using-registries) + - [Table of Contents](#table-of-contents) + - [`vcpkg-configuration.json`](#vcpkg-configurationjson) + - [Registry Objects](#registry-objects) + - [Registry Objects: `"kind"`](#registry-objects-kind) + - [Registry Objects: `"baseline"`](#registry-objects-baseline) + - [Registry Objects: `"repository"`](#registry-objects-repository) + - [Registry Objects: `"path"`](#registry-objects-path) + - [Configuration: `"default-registry"`](#configuration-default-registry) + - [Configuration: `"registries"`](#configuration-registries) + - [Example Configuration File](#example-configuration-file) + - [Package Name Resolution](#package-name-resolution) + - [Versioning Support](#versioning-support) + +## `vcpkg-configuration.json` + +From a high level perspective, everything that a project needs to define +about registries is contained in the vcpkg configuration file. In classic +mode, the configuration file lies in the vcpkg root; for manifest mode, +the file must exist next to the project's `vcpkg.json` file. +This file is named `vcpkg-configuration.json`, and it's a simple top-level +object file. + +### Registry Objects + +Registries are defined in JSON as objects. They must contain at least the +`"kind"` and `"baseline"` fields, and additionally the different kinds of +registry will have their own way of defining where the registry can be found: + +- git registries require the `"repository"` field +- filesystem registries require the `"path"` field +- built-in registries do not require a field, since there is only one + built-in registry. + +#### Registry Objects: `"kind"` + +The `"kind"` field must be a string: + +- For git registries: `"git"` +- For filesystem registries: `"filesystem"` +- For the builtin registry: `"builtin"` + +#### Registry Objects: `"baseline"` + +The `"baseline"` field must be a string. For git registries and for the +built-in registry, it should be a 40-character commit ID. +For filesystem registries, it can be any string that the registry defines. + +#### Registry Objects: `"repository"` + +This should be a string, of any repository format that git understands: + +- `"https://github.com/microsoft/vcpkg"` +- `"git@github.com:microsoft/vcpkg"` +- `"/dev/vcpkg-registry"` + +#### Registry Objects: `"path"` + +This should be a path; it can be either absolute or relative; relative paths +will be based at the directory the `vcpkg-configuration.json` lives in. + +### Configuration: `"default-registry"` + +The `"default-registry"` field should be a registry object. It defines +the registry that is used for all packages that are not claimed by any +package registries. It may also be `null`, in which case no packages that +are not claimed by package registries may be installed. + +### Configuration: `"registries"` + +The `"registries"` field should be an array of registry objects, each of +which additionally contain a `"packages"` field, which should be an array of +package names. These define the package registries, which are used for +the specific packages named by the `"packages"` field. + +The `"packages"` fields of all the package registries must be disjoint. + +### Example Configuration File + +Let's assume that you have mirrored at +: this will be your default registry. +Additionally, you want to use North Wind Trader's registry for their +beison and beicode libraries. The following `vcpkg-configuration.json` +will work: + +```json +{ + "default-registry": { + "kind": "git", + "repository": "https://git.example.com/vcpkg", + "baseline": "eefee7408133f3a0fef711ef9c6a3677b7e06fd7" + }, + "registries": [ + { + "kind": "git", + "repository": "https://github.com/northwindtraders/vcpkg-registry", + "baseline": "dacf4de488094a384ca2c202b923ccc097956e0c", + "packages": [ "beicode", "beison" ] + } + ] +} +``` + +## Package Name Resolution + +The way package name resolution works in vcpkg is fairly distinct from many +package managers. It is very carefully designed to _never_ implicitly choose +the registry that a package is fetched from. Just from +`vcpkg-configuration.json`, one can tell exactly from which registry a +package definition will be fetched from. + +The name resolution algorithm is as follows: + +- If there is a package registry that claims the package name, + use that registry; otherwise +- If there is a default registry defined, use that registry; otherwise +- If the default registry is set to `null`, error out; otherwise +- use the built-in registry. + +### Versioning Support + +Versioning with custom registries works exactly as it does in the built-in +registry. You can read more about that in the [versioning documentation]. + +[versioning documentation]: versioning.md \ No newline at end of file