Archives of last quarterly package builds?

[grarpamp]

Packages are delivered via a single quarterly label here

https://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/

which corresponds to the latest quarterly branch label here

https://svnweb.freebsd.org/ports/branches/?sortby=date#dirlist


However, similar to how the tags here

https://svnweb.freebsd.org/ports/tags/?sortby=date#dirlist

are archived here

https://pkg.freebsd.org/FreeBSD:11:amd64/
as these
https://pkg.freebsd.org/FreeBSD:11:amd64/release_[n]


The last "ie: final" builds of each quarterly branch before they
roll over should also be moved off into their own archived
quarterly directories as

https://pkg.freebsd.org/FreeBSD:11:amd64/yyyyQ[n]

For example /quarterly/ should be repointed from 2018Q2
to 2018Q3, leaving 2018Q2 as a live "pkg" accessible archive.


Eventually all such archives could be moved to historical
archive server under typical release support expiry periods.


This would also serve critical purpose as an independant
original remote repository for validating local package / file
signatures against compromise, corruption, loss.


For example, does the last 2018Q2 (or older ones) still exist
anywhere for users to reference and use?

[Julian Elischer]

Post by grarpamp
Packages are delivered via a single quarterly label here
https://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/
which corresponds to the latest quarterly branch label here
https://svnweb.freebsd.org/ports/branches/?sortby=date#dirlist
However, similar to how the tags here
https://svnweb.freebsd.org/ports/tags/?sortby=date#dirlist
are archived here
https://pkg.freebsd.org/FreeBSD:11:amd64/
as these
https://pkg.freebsd.org/FreeBSD:11:amd64/release_[n]
The last "ie: final" builds of each quarterly branch before they
roll over should also be moved off into their own archived
quarterly directories as

I've asked for this but the answer is "no we don't do that..  and have
no plans to".
Which is a putty as it means you need to make your own snapshots if
you want to have any reproducability.
It no linger matters to me as we now roll all our own packages from
source (we have private OS changes
that make this a requirement), but it was a sore point for many years.

Post by grarpamp
https://pkg.freebsd.org/FreeBSD:11:amd64/yyyyQ[n]
For example /quarterly/ should be repointed from 2018Q2
to 2018Q3, leaving 2018Q2 as a live "pkg" accessible archive.
Eventually all such archives could be moved to historical
archive server under typical release support expiry periods.
This would also serve critical purpose as an independant
original remote repository for validating local package / file
signatures against compromise, corruption, loss.
For example, does the last 2018Q2 (or older ones) still exist
anywhere for users to reference and use?

no.

Post by grarpamp
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-ports

[grarpamp]

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale? Or is another model of pkg build,
distribution, and archiving coming?

It seems no more would be needed than
- an update to release / handbook / mirror info noting their status
as "final, to be removed [to archives] on date + timeframe", say 1 year.
- simple sysadmin on pkg / web side as part of each quarter activity.
- some storage space.
- obviously they are the final builds of the branch, thus frozen.

Anything else / prereqs missing to doing that?

In addition to the earlier reasons...
'packages /latest' trails 'repo HEAD' so it's a fairly linear turnover.
Yet /quarterly gets a massive bump when the branches swap out
from underneath it.
So one could also see where enterprise and other pkg users might
be expecting a similar progression in /quarterly, and to manually
cutforward, not automatic large whack at once. Those production
bumps can hurt. So they might choose to track repo_conf /yyyyQn
and trial the new quarter before moving to it.

It just seems that the final builds on those quarterly branches
should be left online for a while, instead of just ...poof...GONE.

ie: with pkg, this should work for a year or so...
/.../repo_conf: url: "pkg+https://pkg.FreeBSD.org/${ABI}/2018Q2"

Some labels could also be added for use in pkg's repo_conf...
/last_quarter - simpler than dealing with the date, alternately: /prev_quarter
/this_quarter - same as today's /quarterly
/head - unlikely due to build / mirror times and other factors
/yyyyQn - expose these for manual tracking and cutforward,
and the validation purposes below

[bcc for thread ref]

Post by Julian Elischer

Post by grarpamp
Packages are delivered via a single quarterly label here
https://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/
which corresponds to the latest quarterly branch label here
https://svnweb.freebsd.org/ports/branches/?sortby=date#dirlist
However, similar to how the tags here
https://svnweb.freebsd.org/ports/tags/?sortby=date#dirlist
are archived here
https://pkg.freebsd.org/FreeBSD:11:amd64/
as these
https://pkg.freebsd.org/FreeBSD:11:amd64/release_[n]
The last "ie: final" builds of each quarterly branch before they
roll over should also be moved off into their own archived
quarterly directories as

I've asked for this but the answer is "no we don't do that.. and have no
plans to".
Which is a putty as it means you need to make your own snapshots if you want
to have any reproducability.
It no linger matters to me as we now roll all our own packages from source
(we have private OS changes
that make this a requirement), but it was a sore point for many years.

Post by grarpamp
https://pkg.freebsd.org/FreeBSD:11:amd64/yyyyQ[n]
For example /quarterly/ should be repointed from 2018Q2
to 2018Q3, leaving 2018Q2 as a live "pkg" accessible archive.
Eventually all such archives could be moved to historical
archive server under typical release support expiry periods.
This would also serve critical purpose as an independant
original remote repository for validating local package / file
signatures against compromise, corruption, loss.
For example, does the last 2018Q2 (or older ones) still exist
anywhere for users to reference and use?

no.

[Mark Linimon]

Post by grarpamp

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale? Or is another model of pkg build,
distribution, and archiving coming?

This was discussed in a long thread last June:

https://lists.freebsd.org/pipermail/freebsd-ports/2017-June/109126.html

Short answer: we don't have enough resources.

mcl

[grarpamp]

Post by Mark Linimon
https://lists.freebsd.org/pipermail/freebsd-ports/2017-June/109126.html
Short answer: we don't have enough resources.

The OP there titles and suggests opening more development
branches, which is of course a much bigger subject and
resources, not really relavant to this thread here.


To be clear, as in this subject, "Archive last builds"...
(Builds: the binary packages,
Archive: to store, after the rest of the ports src
and builds have moved on)

No development resources are consumed, that is banned here,
see "frozen" in this thread. The only real expense in this thread
is a few minutes of sysadmin time once a quarter, basically
symlinks, and some HTML doc.

Will continue reading that thread just in case anyone there
had talked about simply keeping the final [quarterly] builds
around for a while.

In fact, since the packages have the hash in the filename
and thus don't collide, every one over the life of the branch
could be kept in physical distribution, even though only the
current set would appear in the pkg metadata files for use
by pkg.

[Mark Linimon]

And many 10s of GB which we would be forcing all the mirrors to carry
(and remember, *N archs *n OSVERSIONS).

This has been cited as a stopper in the past. I don't think I have
a reference to those threads handy.

(e.g., this is not the first time this has been brought up.)

mcl

[Kurt Jaeger]

Hi!

Post by grarpamp

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale?

I don't know, but as the setup of your own package builder box
is simple enough -- wouldn't that be an alternative for you ?

[grarpamp]

Post by Kurt Jaeger
as the setup of your own package builder box
is simple enough -- wouldn't that be an alternative for you ?

Simple and even done. Yet being local, that wouldn't cover others out
there who might have found or thought of similar or additional reasons
to suggest the project might maintain such static archives for a while.

Tis all.

[Julian Elischer]

Post by Kurt Jaeger
Hi!

Post by grarpamp

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale?

I don't know, but as the setup of your own package builder box
is simple enough -- wouldn't that be an alternative for you ?

If the answer is always "don't use the quarterly packages but create
your own, then why have them at all?"

Think about it.. Every three months we spend time adding bug fixes and
stability fixes to a branch.
then just as it's getting good..    we delete all the pkgs.  It makes
no sense at all.

[Kurt Jaeger]

Hi!

Post by Julian Elischer

Post by Kurt Jaeger

Post by grarpamp

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale?

I don't know, but as the setup of your own package builder box
is simple enough -- wouldn't that be an alternative for you ?

If the answer is always "don't use the quarterly packages but create
your own, then why have them at all?"

The idea is: use the quarterlies, and if the next quarter comes,
upgrade to that quarterly. The quarterlies are a way to test
if we can provide some 'more stable tree' than HEAD for the ports.

It's not perfect, and we all learn the use cases and the issues etc.

I don't have the overview over all the posts on that issues, so:
is there a text that describes alternative approaches ? Something
where implementation can be discussed ?

[Rainer Duffner]

Post by Kurt Jaeger
The idea is: use the quarterlies, and if the next quarter comes,
upgrade to that quarterly. The quarterlies are a way to test
if we can provide some 'more stable tree' than HEAD for the ports.
It's not perfect, and we all learn the use cases and the issues etc.
is there a text that describes alternative approaches ? Something
where implementation can be discussed ?

The problem is that different people have different foci.

I think it’s assumed that one hosts and maintains his (or her) own copy of the ports-tree and maintains it according to one’s own focus-points.

E.g.: if I was to maintain my own fork of the ports-tree, I’d lay the emphasis on a number of ports that greatly concern me (apache, php, nginx, varnish, python and some of its base-ports, plugins for nagios and some other stuff I’ve forgotten). I’d basically follow upstream with those very closely.
The rest, I’d let dormant most of the time, unless a security-vulnerability made an update inevitable.

But I’m really not in a position to do that, so I use the quarterly cuts. They are a good compromise.

Sometimes, I copy over a port from HEAD to my quarterly checkout because I really want to have the update in. But that has become rare, actually.


Different people have different requirements.
I think if you need very high stability, you’ll likely end up using something else (CentOS+ Software Collections - or Ubuntu, if you’re really desperate...)

Certainly, someone from the foundation or some other company has done the math on what it would take (man-power and financials) to maintain certain subsets of the ports for longer than three months.
Or everything.

It will, however, be almost impossible to get it right for everybody.

[Julian Elischer]

Post by Rainer Duffner

Post by Kurt Jaeger
The idea is: use the quarterlies, and if the next quarter comes,
upgrade to that quarterly. The quarterlies are a way to test
if we can provide some 'more stable tree' than HEAD for the ports.
It's not perfect, and we all learn the use cases and the issues etc.
is there a text that describes alternative approaches ? Something
where implementation can be discussed ?

My issue is that just as it starts to get the bugs wrinkled out of it,
it's deleted.

I gave up.  and now we mirror the ports tree in house and do a build
of all the
packages at a given revision on the head branch,
and then OCCASIONALLY we slide a single package forward (or back) if
we were
unfortunate in our snapshot and caught it with a bug/problem.

Post by Rainer Duffner
The problem is that different people have different foci.
I think it’s assumed that one hosts and maintains his (or her) own
copy of the ports-tree and maintains it according to one’s own
focus-points.
E.g.: if I was to maintain my own fork of the ports-tree, I’d lay
the emphasis on a number of ports that greatly concern me (apache,
php, nginx, varnish, python and some of its base-ports, plugins for
nagios and some other stuff I’ve forgotten). I’d basically follow
upstream with those very closely.
The rest, I’d let dormant most of the time, unless a
security-vulnerability made an update inevitable.
But I’m really not in a position to do that, so I use the quarterly
cuts. They are a good compromise.
Sometimes, I copy over a port from HEAD to my quarterly checkout
because I really want to have the update in. But that has become
rare, actually.
Different people have different requirements.
I think if you need very high stability, you’ll likely end up using
something else (CentOS+ Software Collections - or Ubuntu, if you’re
really desperate...)
Certainly, someone from the foundation or some other company has
done the math on what it would take (man-power and financials) to
maintain certain subsets of the ports for longer than three months.
Or everything.
It will, however, be almost impossible to get it right for everybody.

[Ronald Klop]

Van: grarpamp <***@gmail.com>
Datum: donderdag, 2 augustus 2018 22:21
Aan: freebsd-***@freebsd.org
CC: freebsd-***@freebsd.org
Onderwerp: Re: Archives of last quarterly package builds?

Post by grarpamp

Post by Julian Elischer
I've asked for this but the answer is
"no we don't do that.. and have no plans to".

What is the rationale? Or is another model of pkg build,
distribution, and archiving coming?
It seems no more would be needed than
- an update to release / handbook / mirror info noting their status
as "final, to be removed [to archives] on date + timeframe", say 1 year.
- simple sysadmin on pkg / web side as part of each quarter activity.
- some storage space.
- obviously they are the final builds of the branch, thus frozen.
Anything else / prereqs missing to doing that?

I'm not involved in any pkg building, so don't take me too seriously.
Keeping archives also means keeping packages with security vulnerabilities. Why would somebody put that on the internet?

Regards,
Ronald.

[grarpamp]

Post by Mark Linimon
And many 10s of GB which we would be forcing all the mirrors to carry
(and remember, *N archs *n OSVERSIONS).
This has been cited as a stopper in the past.

There's enough slack to pull down at least one
new quarter, how deep that slack goes hasn't
been chimed in on but this was copied out
to ***@.

However on the plus side, one could easily imagine
a program where mirrors could receive HW / donations
from the foundation... 4TiB is only $100.

They're static reference copies, already transferred,
and only maintained for a year or so, so usage,
thus bandwidth should be low. For the previous
quarters, squelching the web index in dir '/All'
exposing them only to pkg from pkg metadata,
and to rsync module... is further possible.

And the last final pkg build "pkg /latest" for
each major version num "... 8 9 10 11 12 13 ..."
should probably also be kept on archive rsync
server for years like the iso's and distfiles.

Post by Mark Linimon
Keeping archives also means keeping packages with security vulnerabilities.
Why would somebody put that on the internet?

Why would FreeBSD or any other OS put all their
old vulnerable release ISO's, and keep old vulnerable
commits in their repos, on the internet. Lots of reasons,
many noted in this thread.

[Rainer Duffner]

Post by grarpamp
For example /quarterly/ should be repointed from 2018Q2
to 2018Q3, leaving 2018Q2 as a live "pkg" accessible archive.

I think the only real answer is to host your own repo.

I’m not sure how people who use FreeBSD for anything more than their personal server can do this without their own repo (unless their selection of packages is very limited).

Similarly, you’ve got to have mirror of centos+epel or the ubuntu repo if you use it for anything more than dabbling.

[Brooks Davis]

Post by Rainer Duffner

Post by grarpamp
For example /quarterly/ should be repointed from 2018Q2
to 2018Q3, leaving 2018Q2 as a live "pkg" accessible archive.

I think the only real answer is to host your own repo.
I???m not sure how people who use FreeBSD for anything more than their personal server can do this without their own repo (unless their selection of packages is very limited).

It's worth noting that if you don't need custom builds, you can use pkg
to maintain a local repo with only the packages you need in it.

-- Brooks