Normalize nixexpr{input,path} from builds to jobsetevals.

[?]

Jan 22, 2021, 12:14 PM

MB5INTIQWM5VXEEWFNX46SWVNTORXWZGKZDOYZEORQ5T5ZM7XI2QC

Dependencies

[2] UROSQNYP reproduce.tt: Add --print-flags option
[3] BQHDSG6O Merge remote-tracking branch 'origin/master' into flake
[4] SVDYZSZC Schema: add errorMsg, errorTime to JobsetEvals
[5] QUMWPGCU Add flake configuration to the web interface
[6] VCG6KNUX Remove the "clone build" feature
[7] DH3KNBAV Merge remote-tracking branch 'upstream/who-broke-builds' into upstream-master
[8] TQKGQ5R3
[9] D3DIBMOK * For products that are directories (like manuals), allow a default
[10] 3PNG7NIB Remove trailing whitespace
[11] BD3GRK4B * Get rid of "positive failures" and separate log phases.
[12] 26PYNDXV Remove the errorMsg column from the Builds table
[13] Y6AHH4TH Remove the logfile and logSize columns from the database
[14] KSBB33RE Add a dashboard
[15] 4CELXP7P Remove the longDescription field
[16] KPZNJ33U * In views, support selecting a job that doesn't depend on the
[17] ITJWNQEZ Update Hydra schema, otherwise hydra-notify will not work.
[18] NZXX6PLD Remove the Jobs table
[19] J5UVLXOK * Start of a basic Catalyst web interface.
[20] JM3DPYOM generated schema with new dbix class schema loader, grrrrrr
[21] K22TMPH5 Make the info tables less compressed
[22] G2T4WAHI Store the inputs of each evaluation in the database
[23] YAPITGB3 * Boolean inputs.
[24] V4RNHJNR * Add a link to each project's homepage. Suggested by karltk.
[25] LCKWLQW3 * In Sqlite "release" is now a keyword, so use "release_" instead.
[26] N74SRHS6 Remove the "releases" feature
[27] 3E6IP3R3 * Add the name of the jobset to ReleaseSetJobs, otherwise we can't
[28] HZWUT4YN Allow users to reproduce builds on their own systems
[29] SW7STLQ7 Cache flake-based jobset evaluations
[30] SS4TZXNU Distinguish between permanent evaluation errors and transient input errors
[31] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
[32] 6WRGCITD Enable declarative projects.
[33] SIDK2E7V Jobs: add a nullable jobset_id foreign key to Jobsets.
[34] PCKLFRT5 Support push notification of repository changes
[35] 3G7OYUSE reproduce.tt: Check for the presence of commands like nix-build
[36] JIJDYWPY Remove the Build menu from the top bar
[37] 444WMU3Q TODO
[38] YTIDBFGU Drop unused "disabled" columns
[39] EYNG4EL4 * Regenerate the bindings from a clean sqlite database.
[40] 7ECJWNVX Cleanup Project model
[41] 53IMJNBB Add isChannel column and meta attribute.
[42] W5OAZWPD Drop the errorMsg column in the Jobs table
[43] ICRGXEPI Builds.jobset_id: make not-null
[44] L2E6EVE2 * Merged the Build and Job tables.
[45] K3HODXGH Check all inputs for blame but only email selected inputs
[46] TWVSALRL * Allow the maximum number of concurrent builds per platform to be
[47] KRQGRRN7 Use extra-binary-caches
[48] MSIHMO45 Tweak build steps
[49] X27GNHDV * Basic job info in the database.
[50] ZVTSOVHN * Support Subversion checkouts.
[51] P3M6UFMP Merge remote-tracking branch 'origin/master' into flake
[52] MWXDVTS3 Copy the flake migration from the flake branch
[53] 37R34XJO * Negative caching: don't perform a build if a dependency already
[54] YU6CND7C Remove support for views
[55] 2GUAKGTB Fix indentation of build.tt
[56] SB2V735V Keep track of the database schema version
[57] W4G5MZZS hydra-evaluator improvements
[58] SJLEZFC4 check getHydraPath in stead of Envvar HYDRA_DBI directly
[59] X5W5PGGM Builds: populate Builds.jobset_id in hydra-eval-jobset
[60] PGSSRA7C Add an input type "nix" for passing arbitrary Nix expressions
[61] WE5Q2NVI Allow build to be bumped to the front of the queue via the web interface
[62] IK53RV4V
[63] LZVO64YG Merge in the first bits of the API work
[64] PMNWRTGJ Add multiple output support
[65] FTPCV25M Store aggregate members in the database
[66] 6HWIUOSU Avoid fetching Projects/Jobsets just to get the name column
[67] AZCCANUB Missing part of aaacf9eda36f5cdcc77cec20d6f49fbb6f925105
[68] KQAQ4FIF Update Schema classes
[69] GEADFVZ5 hydra-queue-runner: Improved scheduling
[70] QNDXPVCI * Store the jobset's nixExprPath and nixExprInput fields in a build to
[71] T5BIOVJE Add support for tracking custom metrics
[72] BHZXGT2H * Channels: provide an index page that lists all the packages in the
[73] CMU3YKOU * Store the release name.
[74] TLZ2SPBR
[75] VJHIHMEH * Store the meta.longDescription and meta.license attributes in the
[76] S6OISBQ3 * Mark the "current" builds in a jobset, i.e. those corresponding to
[77] YTZOC7C5 * Editing of jobset inputs.
[78] CCIORPJ5 When manually scheduling an eval, force re-instantiation of store derivations
[79] JATC3WQY Make actions dropdown easier to find by making the dropdown title bold
[80] TPNHTE5V Remove obsolete Builds columns and provide accurate "Running builds"
[81] S66BOMVU * Added authentication.
[82] QLOLZHRX Allow a per-jobset check interval
[83] TL7KZWSU reproduce.tt: Use color
[84] RU7AQO7U * Role-based access control. Only admins can create projects. Only
[85] KN3VYE5P * Cleaned up the foreign key constraints.
[86] AHTEIK7G * Added a maintainers field to the Builds table.
[87] 3LEGFJFP "Reproduce" action: Support flakes
[88] H7CNGK4O * Log evaluation errors etc. in the DB.
[89] 4S5JF5JP Use latest DBIx::Class::Schema::Loader
[90] BXHG3HYL When renaming a jobset, add a redirect from the old name
[91] IGR322YP sql: Generate models from postgresql
[92] SM5PVIIJ reproduce.tt: Use -I instead of $NIX_PATH
[93] CLJQCY2X * Store info about all the build actions and allow them to be
[94] PCD3ZH6Z Partially revert 1c20cfdf2403feb78cef515faf15c04d5c9f17bd
[95] ZI535LI6 * hydra: 'new' UI for project/jobset/job/build
[96] JTHWA6AM Rename aggregate members to constituents
[97] R5D7DZPE
[98] 7X2VKCAM Reorder
[99] SJPEC375 Builds: add a nullable jobset_id foreign key to Jobsets.
[100] ECBA3GQO * Make the schema class names match the case of the SQL table names.
[101] VU2OLHD2 Move most of AddBuilds to hydra-eval-jobset
[102] 3TZYSSGX built.tt: Handle the case where the original build is not known
[103] KOTB7BKV
[104] SHBLLAVH * More global substitution.
[105] 6BLUKEQ2 * Caching of "path" inputs, and fake a revision number for those.
[106] B72GLND4
[107] FHF6IZJQ * Basic release management: releases are now dynamically computed as
[108] 3ZCEPLNO
[109] S5PV6IIM * Represent jobs explicitly in the DB.
[110] GNIEG2GC * Disambiguate jobs by jobset name. I.e. jobs with the same name in
[111] FZR7ET5D Jobsets: add a SERIAL, unique, non-null id column
[*] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
[*] N22GPKYT * Put info about logs / build products in the DB.
[*] RFE6T5LG * Store jobset evaluations in the database explicitly. This includes
[*] D5QIOJGP * Move everything up one directory.

Change contents

edit in src/lib/Hydra/Schema/Builds.pm at line 135

[5.8581]→[5.8581:8598](∅→∅),[5.8612]→[5.8612:8634](∅→∅),[5.8634]→[5.7989:8009](∅→∅),[5.8009]→[5.8675:8692](∅→∅),[5.8675]→[5.8675:8692](∅→∅),[5.8706]→[5.8706:8726](∅→∅),[5.8726]→[5.4218:4239](∅→∅)

is_nullable: 1

=head2 nixexprinput

data_type: 'text'
is_nullable: 1

=head2 nixexprpath

data_type: 'text'
edit in src/lib/Hydra/Schema/Builds.pm at line 239

[5.9030]→[5.102:120](∅→∅),[5.949]→[5.102:120](∅→∅),[5.120]→[5.9031:9076](∅→∅),[5.9076]→[5.223:240](∅→∅),[5.223]→[5.223:240](∅→∅),[5.240]→[5.4614:4659](∅→∅)

"nixexprinput",
{ data_type => "text", is_nullable => 1 },
"nixexprpath",
{ data_type => "text", is_nullable => 1 },
replacement in src/lib/Hydra/Schema/Builds.pm at line 531

[5.3378]→[5.2522:2664](∅→∅)

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2020-05-27 17:40:41
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:RIKKFfcKXFWIUeM8ma++iw

[5.3378]

[5.4481]

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2021-01-22 07:11:57
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Df5N0EByYJqoSUqA0dld/A
edit in src/lib/Hydra/Schema/JobsetEvals.pm at line 91

[5.16449]

[5.512]

=head2 nixexprinput

data_type: 'text'
is_nullable: 1

=head2 nixexprpath

data_type: 'text'
is_nullable: 1
edit in src/lib/Hydra/Schema/JobsetEvals.pm at line 145

[5.18063]

[5.632]

"nixexprinput",
{ data_type => "text", is_nullable => 1 },
"nixexprpath",
{ data_type => "text", is_nullable => 1 },
replacement in src/lib/Hydra/Schema/JobsetEvals.pm at line 232

[5.18354]→[4.238:380](∅→∅)

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2021-01-21 11:13:38
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:zDBtAFc4HiFUcL/TpkuCcg

[5.18354]

[5.1674]

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2021-01-22 07:11:57
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:hdu+0WWo2363dVvImMKxdA
replacement in src/lib/Hydra/Schema/Jobsets.pm at line 378

[5.22628]→[5.2808:2950](∅→∅)

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2020-05-27 17:40:41
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:aDW78MCelU/ma953aTcHvA

[5.22628]

[5.1062]

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2021-01-22 07:11:57
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:6P1qlC5oVSPRSgRBp6nmrw
replacement in src/lib/Hydra/Schema/Projects.pm at line 261

[5.7308]→[5.2951:3093](∅→∅)

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2020-05-27 17:40:41
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:iBGJjFWiI9Wy9zwT7xGOEA

[5.7308]

[5.171]

# Created by DBIx::Class::Schema::Loader v0.07049 @ 2021-01-22 07:11:57
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Ff5gJejFu+02b0lInobOoQ
replacement in src/root/build.tt at line 123

[5.226]→[5.0:48](∅→∅)

[% IF build.nixexprinput || eval.flake %]

[5.226]

[5.260]

[% IF eval.nixexprinput || eval.flake %]
replacement in src/root/build.tt at line 349

[5.192]→[5.7702:7736](∅→∅),[5.609]→[5.7702:7736](∅→∅),[5.7702]→[5.7702:7736](∅→∅)

[% IF build.nixexprinput %]

[5.609]

[5.7736]

[% IF eval.nixexprinput %]
replacement in src/root/build.tt at line 352

[5.7784]→[5.7784:7910](∅→∅)

<td>file <tt>[% HTML.escape(build.nixexprpath) %]</tt> in input <tt>[% HTML.escape(build.nixexprinput) %]</tt></td>

[5.7784]

[5.7910]

<td>file <tt>[% HTML.escape(eval.nixexprpath) %]</tt> in input <tt>[% HTML.escape(eval.nixexprinput) %]</tt></td>
replacement in src/root/reproduce.tt at line 172

[5.2953]→[5.2953:2996](∅→∅)

[% IF input.name == build.nixexprinput +%]

[5.2953]

[5.2996]

[% IF input.name == eval.nixexprinput +%]
replacement in src/root/reproduce.tt at line 200

[5.3499]→[2.178:255](∅→∅)

args+=("$nixExprInputDir/[% build.nixexprpath %]" -A '[% build.job.name %]')

[5.3499]

[2.255]

args+=("$nixExprInputDir/[% eval.nixexprpath %]" -A '[% build.job.name %]')
edit in src/script/hydra-eval-jobset at line 463

[5.16199]→[5.16199:16301](∅→∅)

, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
edit in src/script/hydra-eval-jobset at line 725

[5.993]

[115.6334]

, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
edit in src/sql/hydra.sql at line 165

[5.594]→[5.594:664](∅→∅),[5.664]→[5.0:69](∅→∅),[5.69]→[5.0:96](∅→∅),[5.96]→[3.539:539](∅→∅),[5.96]→[5.743:791](∅→∅),[3.539]→[5.743:791](∅→∅),[5.1269]→[5.743:791](∅→∅),[5.743]→[5.743:791](∅→∅),[5.9348]→[5.9348:9349](∅→∅)

-- Copy of the nixExprInput/nixExprPath fields of the jobset that
-- instantiated this build. Needed if we want to reproduce this
-- build. FIXME: this should be stored in JobsetEvals, storing it
-- here is denormal.

nixExprInput text,
nixExprPath text,
edit in src/sql/hydra.sql at line 462

[5.1048]

[115.7667]

nixExprInput text, -- name of the jobsetInput containing the Nix or Guix expression
nixExprPath text, -- relative path of the Nix or Guix expression
file addition: upgrade-71.sql (----------)

[116.3004]

ALTER TABLE JobsetEvals
ADD COLUMN nixExprInput text,
ADD COLUMN nixExprPath text;

-- This migration took 4.5 hours on a server
-- with 5400RPM drives, against a copy of hydra's
-- production dataset. It might take a significantly
-- less amount of time there, and not justify a
-- batched migration.
UPDATE jobsetevals
SET (nixexprinput, nixexprpath) = (
SELECT builds.nixexprinput, builds.nixexprpath
FROM builds
LEFT JOIN jobsetevalmembers
ON jobsetevalmembers.build = builds.id
WHERE jobsetevalmembers.eval = jobsetevals.id
LIMIT 1
)
WHERE jobsetevals.id in (
SELECT jobsetevalsprime.id
FROM jobsetevals as jobsetevalsprime
WHERE jobsetevalsprime.nixexprinput IS NULL
-- AND jobsetevalsprime.id > ? --------- These are in case of a batched migration
ORDER BY jobsetevalsprime.id ASC -- /
-- LIMIT ? -- ----------------------
);

ALTER TABLE builds
DROP COLUMN nixexprinput,
DROP COLUMN nixexprpath;