Yum proxy repository does not keep cache (3.12.0-01)

Hello,

I’ve created two repositories:

  1. Yum hosted - “hosted”
  2. Yum proxy (to hosted) - “proxy”

My case is to keep the packages on “proxy” repositories regardless of whether they are available on “hosted” as described in documentation:

A proxy repository is a repository that is linked to a remote repository. Any request for a component is verified against the local content of the proxy repository. If no local component is found, the request is forwarded to the remote repository. The component is then retrieved and stored locally in the repository manager, which acts as a cache. Any future requests for the same component are fulfilled from the local storage, eliminating network bandwidth and time overhead when retrieving the component from the remote repository again. In addition, when components are downloaded to the NXRM3 proxy, a copy remains there indefinitely, unless removed by an NXRM administrator. This is extremely useful in the event a component becomes unavailable in the remote repository, and provides more control over the components needed to build your applications.

So the test cases are:

  1. Upload a package to the “hosted” repo
  2. Install package on the system from “proxy” repo
  3. Remove package from the system
  4. Remove package from the “hosted” repo
  5. Install package on the system from the “proxy” repo after metadata refreshes

So the 5th failed. The “proxy” repository contains the needed package, but the metadata does not.

Is it the right behavior for the yum proxy repositories for now?

Also, I’ve checked thatsome sort of this is planned, but not released:

…and in a future iteration of this feature we will be adding “keep n versions of packages”.

Or my configuration of a “proxy” repo is broken:

Maximum Metadata Age = 10 (for tests purposes)
Cache responses for content not present in the proxied repository = true
How long to cache the fact that a file was not found in the repository (in minutes) = 20 (for tests purposes)

Proxy repositories also proxy the metadata of the remote, rewriting it locally. If the remote metadata is no longer lists the package as available, it will not be available through the proxy repository.

We are considering covering your use case, but it isn’t as straightforward as it might first appear. See here for more information:


Regards,

Rich

Hi @rseddon!

I’m also very interested in being able to serve cached packages even in case of remote metadata updates. I have quite a lot of negative experience when updating remote metadata for the corresponding proxy repository is just not enough. For example, maintainers of those repositories upload a new version of a package and remove an old one quite often leading to deployment failures, unreproducible infrastructures, etc.

Don’t you think that snapshot repositories like in nexus-repository-apt plugin could solve the issue in a pretty simple and straitforward way?
Apt plugin allows to snapshot/freeze a current repository by just copying its metadata into a separate folder and then this snapshot can be referenced to as an ordinary repository.

Here is how it looks like:

… and here is how these snapshot repositories can be referenced to

apt_repos:
  - repo: deb https://{{ apt_archive_machine }}/repository/ubuntu-16-cache/snapshots/20180627_1/ xenial main restricted universe multiverse
  - repo: deb https://{{ apt_archive_machine }}/repository/ubuntu-16-updates-cache/snapshots/20180627_1/ xenial-updates main restricted universe multiverse
  - repo: deb https://{{ apt_archive_machine }}/repository/ubuntu-16-security-cache/snapshots/20180627_1/ xenial-security main restricted universe multiverse
  - repo: deb https://{{ apt_archive_machine }}/repository/ubuntu-16-backports-cache/snapshots/20180627_1/ xenial-backports main restricted universe multiverse
  - repo: deb https://{{ apt_archive_machine }}/repository/ubuntu-16-e2fsprogs-cache/snapshots/20180628_1/ xenial main
  - repo: deb [arch=amd64] https://{{ apt_archive_machine }}/repository/ubuntu-16-ext-release-local/snapshots/20180628_1/ xenial main

In case of apt after all this manipulations everything is stable and reproducible.

Hi @szhemzhitsky,

Thanks for this suggestion, it is actually a route we are investigating to solve this problem. It’s a good solution because it avoids performance problems of having to merge the metadata with what is stored locally on every update (and therefore update SHAs). The only downside of this is it puts the onus on the user to periodically create a snapshot of the metadata.

I’m curious, if we did implement this solution for YUM how would you and your organization approach the triggering of a new snapshot?

Hi @jstephens,

I’m curious, if we did implement this solution for YUM how would you and your organization approach the triggering of a new snapshot?

Currently (in case of apt) we are triggering a new snapshot for external repos manually, usually in case of security updates available there.
Then the corresponding environment (we use ansible to manage them) is switched onto the new snapshot repository and updated from there (e.g. apt-get update && apt-get upgrade in case of apt).
In that case all the servers of the updated environment have the same versions of the packages and all the necessary security updates. If it’s necessary to completely reproduce an environment it’s necessary just to replay the scripts using the same snapshot repo.