hydra-notify: make the prometheus endpoint configurable, default-off

[?]

Aug 18, 2021, 7:09 PM

4H6FVIWGOKFWAUB6TLR7XIWHHSSHCJOVCWRIYCTNQOQDJHGIQIAQC

Dependencies

[2] FUOOHCAA Init a docs section for monitoring, document queue-runner-status and the prometheus metrics
[3] Q2E4J3PH Update docs
[4] I7DFJWL6 hydra-notify: initial scratch take of prometheus events
[5] 32KJOERM Turn hydra-notify into a daemon
[6] 6WRGCITD Enable declarative projects.
[7] IE2PRAQU hydra-queue-runner: Send build notifications
[8] 3AKZKWCR RunCommand: Test
[9] NTEDD7T4 Provide a plugin hook for when build steps finish
[10] P4SME2BC Abstract over postgres' LISTEN/NOTIFY
[11] QEXFA5Y2 hydra-notify: use Hydra::Event
[12] T3OHZDYP hydra-notify: move BuildFinished processing to an Event
[*] 2GK5DOU7 * Downloading closures.
[*] 5DXGOH6Q statsd: add a chance to set hostname and port in hydra.conf
[*] I6QMKSIS Move getBaseUrl

Change contents

edit in doc/manual/src/configuration.md at line 80

[3.2366]

[3.2366]

```

hydra-notify's Prometheus service
---------------------------------

hydra-notify supports running a Prometheus webserver for metrics. The
exporter does not run unless a listen address and port are specified
in the hydra configuration file, as below:

```conf
<hydra_notify>
<prometheus>
listen_address = 127.0.0.1
port = 9199
</prometheus>
</hydra_notify>
edit in doc/manual/src/monitoring/README.md at line 15

[2.392]

[2.392]

## Notification Daemon

The `hydra-notify` process can expose Prometheus metrics for plugin execution. See
[hydra-notify's Prometheus service](../configuration.md#hydra-notifys-prometheus-service)
for details on enabling and configuring the exporter.
edit in doc/manual/src/monitoring/README.md at line 22

[2.393]

The notification exporter exposes metrics on a per-plugin, per-event-type basis: execution
durations, frequency, successes, and failures.
edit in src/lib/Hydra/Helper/Nix.pm at line 72

[15.633]

[15.633]

sub getHydraNotifyPrometheusConfig {
my ($config) = @_;
my $cfg = $config->{hydra_notify};

if (!defined($cfg) || ref $cfg ne "HASH") {
return undef;
}
edit in src/lib/Hydra/Helper/Nix.pm at line 81

[15.634]

[15.634]

my $cfg = $cfg->{prometheus};
if (!defined($cfg) || ref $cfg ne "HASH") {
return undef;
}
edit in src/lib/Hydra/Helper/Nix.pm at line 86

[15.635]

[16.174]

if (defined($cfg->{"listen_address"}) && defined($cfg->{"port"})) {
return {
"listen_address" => $cfg->{'listen_address'},
"port" => $cfg->{'port'},
};
}

return undef;
}
edit in src/script/hydra-notify at line 16

[4.167]→[4.167:437](∅→∅),[5.18]→[5.3128:3129](∅→∅),[4.437]→[5.3128:3129](∅→∅),[5.1456]→[5.3128:3129](∅→∅),[5.5666]→[5.3128:3129](∅→∅),[5.3128]→[5.3128:3129](∅→∅),[5.3129]→[4.438:473](∅→∅)

my $prom = Prometheus::Tiny::Shared->new;

my $fork_manager = Parallel::ForkManager->new(1 );
$fork_manager->start_child("metrics_exporter", sub {
my $server = HTTP::Server::PSGI->new(
host => "127.0.0.1",
port => 9091,
timeout => 5,
);

$server->run($prom->psgi);
});
edit in src/script/hydra-notify at line 20

[5.3186]

[5.19]

my $config = getHydraConfig();

my $prom = Prometheus::Tiny::Shared->new;

my $promCfg = Hydra::Helper::Nix::getHydraNotifyPrometheusConfig($config);
if (defined($promCfg)) {
my $fork_manager = Parallel::ForkManager->new(1);
$fork_manager->start_child("metrics_exporter", sub {
my $server = HTTP::Server::PSGI->new(
host => $promCfg->{"listen_address"},
port => $promCfg->{"port"},
timeout => 1,
);

$server->run($prom->psgi);
});
}
edit in src/script/hydra-notify at line 45

[5.3187]→[5.3187:3218](∅→∅)

my $config = getHydraConfig();
file addition: hydra-notify.t (----------)

[15.986]

use strict;
use Setup;

my %ctx = test_init(hydra_config => q|
<hydra_notify>
<prometheus>
listen_address = 127.0.0.1
port = 9199
</prometheus>
</hydra_notify>
|);

require Hydra::Helper::Nix;
use Test2::V0;

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig(Hydra::Helper::Nix::getHydraConfig()), {
'listen_address' => "127.0.0.1",
'port' => 9199
}, "Reading specific configuration from the hydra.conf works");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => ":)"
}), undef, "Invalid (hydra_notify is a string) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => []
}), undef, "Invalid (hydra_notify is a list) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {}
}), undef, "Invalid (hydra_notify is an empty hash) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => ":)"
}
}), undef, "Invalid (hydra_notify.prometheus is a string) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => {}
}
}), undef, "Invalid (hydra_notify.prometheus is an empty hash) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => {
"listen_address" => "0.0.0.0"
}
}
}), undef, "Invalid (hydra_notify.prometheus.port is missing) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => {
"port" => 1234
}
}
}), undef, "Invalid (hydra_notify.prometheus.listen_address is missing) configuration options are undef");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => {
"listen_address" => "127.0.0.1",
"port" => 1234
}
}
}), {
"listen_address" => "127.0.0.1",
"port" => 1234
}, "Fully specified hydra_notify.prometheus config is valid and returned");

is(Hydra::Helper::Nix::getHydraNotifyPrometheusConfig({
"hydra_notify" => {
"prometheus" => {
"listen_address" => "127.0.0.1",
"port" => 1234,
"extra_keys" => "meh",
}
}
}), {
"listen_address" => "127.0.0.1",
"port" => 1234
}, "extra configuration in hydra_notify.prometheus is not returned");

done_testing;