[Bug 232350] ports-mgmt/pkg: periodic pkg-checksum and pkg-backup interfere with 'overnight' port builds

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Mark Linimon <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Flags|maintainer-feedback?(kuriya |
|***@FreeBSD.org) |
Assignee|***@FreeBSD.org |***@FreeBSD.org
Summary|periodic pkg-checksum and |ports-mgmt/pkg: periodic
|pkg-backup interfere with |pkg-checksum and pkg-backup
|'overnight' port builds |interfere with 'overnight'
| |port builds

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Brad Davis <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #1 from Brad Davis <***@FreeBSD.org> ---
This is a race condition for sure, but in this situation I am not sure how we
could reliably tell if people where building ports easily.

I would just recommend two options:

1) The preferred option, build in poudriere so that the builds are isolated
from the rest of the system.
2) As part of the build command you are running, use sysrc to disable these
jobs and at the end of the command use sysrc to reenable them.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

***@utanet.at changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@utanet.at

--- Comment #2 from ***@utanet.at ---
It is the question if it is a problem of pkg. Or it is possible to handle it in
ther periodic scripts.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Ian Lepore <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #3 from Ian Lepore <***@FreeBSD.org> ---
IMO, the fix for this should strive to reduce failures due to concurrent access
by the nightly periodic jobs to nearly zero, then ensure that if there is a
failure, it is in the periodic jobs, not in the ports build process.

A way to accomplish that would be to change the behavior of the periodic jobs
to be something like:

set a retry counter and retry limit
do
copy all files needed by the periodic job to a dir in /tmp
run the statistics/validation/whatever on the temp files
while validation is not-successful and retry count < limit
clean up temp files

With that logic it's possible for the validation to fail if the copy happened
to grab a file that was being updated at the instant of the copy, but a retry
loop will reduce the chances of that happening again to almost nothing. Even if
it does fail, what fails is the lower-priorty periodic work, not the expensive
and more-important ports building work.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #4 from ***@utanet.at ---
See also bug #227952.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Alex Kozlov <***@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@FreeBSD.org

--- Comment #5 from Alex Kozlov <***@FreeBSD.org> ---
IMO pkg locking should be fixed. There is no need to keep lock for all duration
of pkg-check run. The poudriere suggestion is good, but it is more of a
workaround.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #6 from Ian Lepore <***@FreeBSD.org> ---
(In reply to Alex Kozlov from comment #5)

"Should be fixed" is easy to say, but it's meaningless. Provide details,
what's your proposal? It doesn't matter if the lock can be held for a shorter
duration, the locking conflicts will still occur, it will just be less likely.
But failing to handle a less-likely error is still a failure.

How specifically do you propose to handle the case where nightly periodic jobs
are running at the same time as package building and/or installation and there
is lock contention? Will there be retries? A limit to how many times or how
long to keep retrying? A way to configure those limits?

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #7 from ***@utanet.at ---
If have set:
/usr/local/etc/pkg.conf

LOCK_WAIT = 100;
LOCK_RETRIES = 100;


makes it better, but does not solve it.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #8 from Alex Kozlov <***@FreeBSD.org> ---
The race window will shrink to negligible values. Also there is no need to
build and *install* packages at the same time. Even if original poster doesn't
want to use poudriere, which I strongly recommend, he can build packages
overnight and install them later, manually.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #9 from Bob Frazier <***@mrp3.com> ---
the thing is, if a port has a zillion run/build dependencies, and you simply
type 'make' for the port, then the dependent packages will all be installed as
part of the process. In the case of build dependencies, it would not be
possible to build a package until the build dependencies have been installed.
And the process does not appear to differentiate between build dependencies and
run dependencies when you use 'make' (or 'make package' for that matter).

So it's back to 'Catch 22' on the build-dependency packages, if building
packages "without installing" is the only option.

Why not just eliminate the problem by making a copy of the relevant files,
rather than locking them at all, during the periodic jobs, and working with the
copies?

Ian's suggestion of copying them to /tmp first makes the most sense.

The alternative would be to shut off the periodic jobs somehow while ports are
building, even if that means creating a /var/run/ports-are-building file (or
similar) and locking it for the duration of the process.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Post by b***@freebsd.org
the thing is, if a port has a zillion run/build dependencies, and you
simply type 'make' for the port, then the dependent packages will all
be installed as part of the process.

Ah yes, if you use make then while you can pre-build build dependency,
there're still problems with other types of dependencies.
I guess there are only two choices: poudriere or disabling pkg-* periodic
scripts. You can run them from cron at another time.

Post by b***@freebsd.org
Why not just eliminate the problem by making a copy of the relevant
files, rather than locking them at all, during the periodic jobs,
and working with the copies?

Well, if you work on copy of pkgdb, then you check against and backup
potentially stalled version of db.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #11 from Ian Lepore <***@FreeBSD.org> ---
(In reply to Alex Kozlov from comment #10)

It is not an error or a problem to be working on a snapshot of the database
taken at the beginning of the periodic run. This is a standard axiom of fixing
races with lockless (or lock minimization) techniques... it doesn't matter
whether you capture the snapshot before or after some arbitrary action, it only
matters that you capture and operate on a consistant snapshot. If updates are
happening to the live database while you run the validation on the snapshot,
those updates will be validated in the run the next day, and that situation is
exactly identical to the situation in which running with exclusionary locks
prevented the update from happening until after the validation run completes.

These are not new problems, and the techniques for solving them are not new
either. I first ran into these problems and the snapshot-based solutions to
them in the 1970s.

The main downside to such techniques is that it requires copying the data to be
validated, and sometimes the size of that data makes that too expensive of an
operation. That could be the case here, I have no idea how much data would
have to be copied in this instance.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #12 from Alex Kozlov <***@FreeBSD.org> ---
The pkg-checksum reads almost entire $PREFIX, so if you don't want for files
disappear/change checksum during validation, you need to copy $(pkg info -la)
files to /tmp.
It's easier with pkgdb, just a few megabytes, but if e.g. pkg-audit works on
stalled pkgdb, there is possibility that you install vulnerable port and will
know about it only after next periodic run. Granted, chances for this is not
very high, but IMO it's much easier and safer to move pkg-* periodic scripts to
another time or build ports in poudriere.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #13 from Ian Lepore <***@FreeBSD.org> ---
(In reply to Alex Kozlov from comment #12)

but if e.g. pkg-audit works on stalled pkgdb, there is possibility that you
install vulnerable port and will know about it only after next periodic run

You seem to have missed the most important point of my comment: You CANNOT
prevent that from happening with ANY technique or algorithm. If the port
build/install completes before validation begins, it gets validated tonight.
If validation begins first and locks out the installation of the port while
validation is running, it gets validated tomorrow night. That's the exact same
situation as validating against a snapshot.

"Use poudriere" may (or often may not) be good advice for any given user and
situation, but IT IS NOT A SOLUTION.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

You CANNOT prevent that from happening with ANY technique or algorithm.

You cannot, but it's not mean that you have to increase duration of
vulnerability window.

"Use poudriere" may (or often may not) be good advice for any given user and
situation, but IT IS NOT A SOLUTION.

The solution is to improve db locking in at least pkg-check, all other
suggestions are workarounds. And IMO poudriere is better one that your idea of
copying $PREFIX to /tmp.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

have to increase duration of vulnerability window

The duration of the vuln window is always 24 hours.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #16 from ***@utanet.at ---
The simpelst solution is
put in /etc/periodic.conf
security_status_pkgchecksum_enable=YES|security_status_pkgchecksum_enable=NO

and at the end of your update-script
pkg check -s

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #17 from ***@utanet.at ---
paste error:
shoudl be
security_status_pkgchecksum_enable=NO

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #18 from Alex Kozlov <***@FreeBSD.org> ---
Sigh, I guess easiest workaround would be:
security_status_pkgaudit_enable=NO
security_status_pkgchecksum_enable=NO in periodic.conf
then
${PREFIX}/etc/periodic/security/410.pkg-audit in the beginning of port building
and
${PREFIX}/etc/periodic/security/460.pkg-checksum at the end
Oh, and LOCK_WAIT=100 in pkg.conf for pkg-backup and all.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #19 from ***@utanet.at ---
Additional it needs a check if pkg check is already running:

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #20 from ***@utanet.at ---
No, not needed if it is set to NO.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

--- Comment #21 from Bob Frazier <***@mrp3.com> ---
having entries in pkg.conf is useful, but can they be automatically
added/removed by the ports build system? Having to manually fiddle with this
over-complicates the ports build process.

And, it seems to me that building/installing poudriere on an embedded device
(like Raspberry Pi) is probably not a good solution for simply building from
ports on the same device. In my opinion, that "breaks ports".


If I had unlimited bandwidth, unlimited CPU, unlimited RAM, and unlimited disk
access speed/space, and unlimited time to read all of the documentation and get
familiar with how to set it up correctly, then certainly 'install poudriere'
would be a simple and adequate solution. But for older and embedded hardware,
and generally busy people, not so much. I'd rather just go to the ports
directory and type 'make install', thanks, and have the system work as
intended.


Another possibility I suggested earlier is for pkg to have "a file someplace",
in /var/run, in /tmp, doesn't really matter where all that much, but one that's
locked in a way that allows for simultaneous locking by other processes, and
prevents the periodic process from locking the entire pkg database.

non-exclusive locks seem to be one of the best ways to make this work, with an
exclusive lock on the same file from periodic processes. It's a method I've
used before for concurrency of this kind.

In a makefile context it may be possible to start a daemon process that tracks
the PID of the parent process as part of the make environment. When the parent
process stops (complete or error), the daemon would exit. The daemon would
then keep the non-exclusive lock on the file. If the 'pkg' utility were
modified to do the locking, it could be something like 'pkg --lock' to do a
non-exclusive lock (and run the daemon), and 'pkg --xlock' to do an exclusive
one, with this command being called for the appropriate context [the difference
would be for building 'pkg' itself, which would have to just work without doing
this step]. The 'pkg' utility would return an error code if the lock fails,
and whatever process would check for this and act appropriately.

[b***@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232350

Tatsuki Makino <***@hotmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@hotmail.com

--- Comment #22 from Tatsuki Makino <***@hotmail.com> ---
Below is a little hentai way for portmaster user.

1. Add the following to /etc/periodic.conf SHELL SCRIPT.

if ! lockf -s -t 0 /var/run/portmaster true ; then
security_status_pkgchecksum_enable="NO"
fi

2. Run portmaster like "lockf /var/run/portmaster portmaster ...".