akkartik/sum-grid.love - Change EZHO4TSWIYYUE73S6XQWIEF3HA3H7MKCNJOT27NTWTVSPVS2SL5QC

new file-system format for freewheeling apps

No more version history, now we have just the contents of the current version.
Editing a definition no longer changes the order in which definitions load.

This should make repos easier to browse, and more amenable to modify. You don't need driver.love anymore. And a stable order eliminates some gotchas. For example:

using driver.love, define Foo = 3 in a definition define Bar = Foo + 1 edit and redefine Foo = 4

Before this commit, you'd get an error when you restart the app. Definitions used to be loaded in version order, and editing a definition would move it to the end of the load order, potentially after definitions using it. I mostly avoided this by keeping top-level definitions independent. It's fine to refer to any definition inside a function body, we only need to be careful with initializers for global variables which run immediately while loading.

After this commit you can still end up in a weird state if you modify a definition that other later definitions use. In the above example, you will now see Foo = 4 and Bar = 4. But when you restart, Foo = 4 and Bar = 5. But that's no more confusing than Emacs's C-x C-e. It's still a good idea to keep top-level definitions order-independent. It's just confusing in a similar way to existing tools if you fail to do so. And your tools won't tend to break as badly.

Why did I ever do my weird version history thing? I think it's my deep aversion to risking losing any data entered. (Even though the app currently will seem to lose data in those situations. You'd need to leave your tools to find the data.) Now I rely on driver.love's undo to avoid data loss, but once you shut it down you're stuck with what you have on disk. Or in git.

I also wasn't aware for a long time of any primitives for deleting files. This might have colored my choices a lot.

Created by Kartik K. Agaram on April 16, 2023

EZHO4TSWIYYUE73S6XQWIEF3HA3H7MKCNJOT27NTWTVSPVS2SL5QC

Dependencies

In channels

main

Change contents

File deletion: tools
BF:BFD[11.2] → [9.3:19]
BF:BFD[9.19] → [9.2:2]

File deletion: stitch-live.lua

BF:BFD[9.2] → [9.794:833]

BF:BFD[9.833] → [9.21:21]

B:BD[9.21] → [9.22:793]

json = require 'json'
function main(args)
  local infile = io.open(args[1])
  local manifest_s = infile:read('*a')
  infile:close()
  local manifest = json.decode(manifest_s)
  local core_filenames = {}
  for k,v in pairs(manifest) do
    if not starts_with(k, 'fw_') then
      table.insert(core_filenames, k)
    end
  end
  table.sort(core_filenames)
  for _,core in ipairs(core_filenames) do
    local filename = ('%04d'):format(manifest[core])..'-'..core
    local f = io.open(filename)
    if f then
      print(f:read('*a'))
      print('')
    end
  end
end
function starts_with(s, prefix)
  if #s < #prefix then
    return false
  end
  for i=1,#prefix do
    if s:sub(i,i) ~= prefix:sub(i,i) then
      return false
    end
  end
  return true
end
main(arg)

File deletion: 0001-fwmanifest, 0001-manifest
BF:BFD[11.2] → [5.69:108]
BF:BFD[5.108] → [12.593:593]
BFD:BFD[11.2] → [12.615:652]
BF:BFD[12.652] → [12.593:593]
B:BD[12.593] → [13.540:563]
```
{"fw_parent":0,"on":1}
```

File deletion: representation.md

BF:BFD[11.2] → [14.1782:1823]

BF:BFD[14.1823] → [14.7:7]

B:BD[14.7] → [14.8:988]

∅:D[7.86] → [14.1065:1422]

B:BD[14.1065] → [14.1065:1422]

∅:D[7.129] → [14.1462:1781]

B:BD[14.1462] → [14.1462:1781]

B:BD[14.1781] → [8.7:689]

B:BD[14.1422] → [7.87:129]

B:BD[14.988] → [7.7:86]

# The on-disk representation of freewheeling apps
When you start up a freewheeling app, you'll see a directory printed out in
the parent terminal (always launch it from a terminal window):
```
new edits will go to /home/...
```
When editing such an app using the driver (see [README.md](README.md)), new
definitions will go into this directory. Let's call it `$SAVE_DIR` in the rest
of this doc.
It is always safe to move such definitions into this repo. (We'll call it `.`
in the rest of this doc.) You'll want to do this if you're sharing them with
others, and it's also helpful if the driver crashes on your app. Moving
definitions will never, ever change app behavior.
```sh
$ mv -i $SAVE_DIR/[0-9]* .  # should never clobber any existing files
$ mv $SAVE_DIR/head .  # expected to clobber the existing file
```
Try looking inside the `head` file with a text editor. It'll contain a number,
the current version of the _manifest_ for this app. For example:
```
478
```
If you moved the files you should see such a file in `.`. If you open this
file, you'll see a JSON table from definition names to version ids. For
example:
```
{ "a": 273, "b": 478}
```
This means the current definition of `a` is in `0273-a` and of `b` in
`0478-b`.
Poking around these files gets repetitive, so there's a tool to streamline
things:
```
```
`stitch-live.lua` takes a manifest file as its argument, and prints out all
the definitions that make up the app at that version.
To compare two versions of the app, use `stitch-live.lua` to copy the
definitions in each into a separate file, and use a file comparison tool (e.g.
`diff`) to compare the two files.
# Scenarios considered in designing this representation
* Capture history of changes.
  - Though it is perhaps too fine-grained and noisy.
* Merge changes from non-live forks to live ones.
  - New files in repo can't hide changes in save dir, because filenames are
    always disjoint between the two.
  - This doesn't apply yet to live updates. Two forks of a single live app
    will likely have unnecessary merge conflicts.
* No special tools required to publish changes to others.
  - Just move files from save dir to repo.
# Scenarios I _would_ like to take into consideration in the future
* Cleaner commits; it's clear what changed.
* merge changes between live forks.
lua tools/stitch-live.lua 0478-fwmanifest
This means the current state of the app is in a file called `0478-fwmanifest`.

File deletion: head
BF:BFD[11.2] → [15.10819:10847]
BF:BFD[15.10847] → [15.10815:10815]
B:BD[15.10815] → [12.60:62]
```
1
```

Replacement in live.lua at line 4 [15.722]

B:BD[15.888] → [16.8:152]

--    from the numeric prefix in file 'head', obtain a manifest
--    load all files (which must start with a numeric prefix) from the manifest

[15.888]

[15.1032]

--    load files in order

Replacement in live.lua at line 10 [15.722]

B:BD[15.1170] → [15.1170:1238]

B:BD[15.1238] → [16.153:225]

∅:D[16.225] → [15.1306:1330]

B:BD[15.1306] → [15.1306:1330]

B:BD[15.1330] → [16.226:330]

--  on incoming messages to a specific file, however, the app must:
--    save the message's value to a new, smallest unused numeric prefix
--    execute the value
--      if there's an error, go back to the previous value of the same
--      definition if one exists

[15.1170]

[15.1330]

--  on incoming messages to a specific file, the app must:
--    determine the definition name from the first word
--    execute the value, returning any errors
--    look up the filename for the definition or define a new filename for it
--    save the message's value to the filename

Replacement in live.lua at line 29 [15.722]

B:BD[15.1673] → [10.12:56]

-- === on startup, load the version at head

[15.1673]

[15.1723]

-- === on startup, load all files with numeric prefix

Replacement in live.lua at line 35 [15.722]

B:BD[15.1775] → [17.8:221]

  Live.head = 0
  Live.next_version = 1
  Live.history = {}  -- array of filename roots corresponding to each numeric prefix
  Live.manifest = {}  -- mapping from roots to numeric prefixes as of version Live.head

[15.1775]

[15.1988]

  Live.filenames_to_load = {}  -- filenames in order of numeric prefix
  Live.filename = {}  -- map from definition name to filename (including numeric prefix)
  Live.final_prefix = 0

Insertion in live.lua at line 39 [15.722]
[15.2015]
[17.222]
```
  -- some hysteresis
```

Replacement in live.lua at line 48 [15.722]

B:BD[15.2158] → [15.2158:2411]

B:BD[15.2411] → [17.248:330]

∅:D[17.330] → [15.2493:2544]

B:BD[15.2493] → [15.2493:2544]

B:BD[15.2544] → [17.331:481]

∅:D[17.481] → [15.2694:2867]

B:BD[15.2694] → [15.2694:2867]

B:BD[15.2867] → [2.8:78]

∅:D[2.78] → [15.2897:3655]

B:BD[15.2897] → [15.2897:3655]

B:BD[15.3655] → [6.9:46]

∅:D[6.46] → [15.3690:3845]

B:BD[15.3690] → [15.3690:3845]

  local files = {}
  live.append_files_with_numeric_prefix('', files)
  table.sort(files)
  live.check_integrity(files)
  live.append_files_with_numeric_prefix(love.filesystem.getSaveDirectory(), files)
  table.sort(files)
  live.check_integrity(files)
  Live.history = live.load_history(files)
  Live.next_version = #Live.history + 1
  local head_string = love.filesystem.read('head')
  Live.head = tonumber(head_string)
  if Live.head > 0 then
    Live.manifest = json.decode(love.filesystem.read(live.versioned_manifest(Live.head)))
  end
  live.load_everything_in_manifest()
end
function live.append_files_with_numeric_prefix(dir, files)
  for _,file in ipairs(love.filesystem.getDirectoryItems(dir)) do
    if file:match('^%d') then
      table.insert(files, file)
    end
  end
end
function live.check_integrity(files)
  local manifest_found, file_found = false, false
  local expected_index = 1
  for _,file in ipairs(files) do
    for numeric_prefix, root in file:gmatch('(%d+)-(.+)') do
      -- only runs once
      local index = tonumber(numeric_prefix)
      -- skip files without numeric prefixes
      if index ~= nil then
        if index < expected_index then
          print(index, expected_index)
        end
        assert(index >= expected_index)
        if index > expected_index then
          assert(index == expected_index+1)
          assert(manifest_found and file_found)
          expected_index = index
          manifest_found, file_found = false, false
        end
        assert(index == expected_index)
        if root == 'fwmanifest' then
          assert(not manifest_found)
          manifest_found = true
        else
          assert(not file_found)
          file_found = true
        end

[15.2158]

[15.3845]

  -- if necessary, copy files from repo to save dir
  if io.open(love.filesystem.getSaveDirectory()..'/0000-freewheeling-start') == nil then
    print('copying all definitions from repo to save dir')
    for _,filename in ipairs(love.filesystem.getDirectoryItems('')) do
      for numeric_prefix, root in filename:gmatch('(%d+)-(.+)') do
        -- only runs once
        local buf = love.filesystem.read(filename)
        print('copying', filename)
        love.filesystem.write(filename, buf)

Replacement in live.lua at line 60 [15.722]

B:BD[15.3869] → [15.3869:4022]

end
function live.load_history(files)
  local result = {}
  for _,file in ipairs(files) do
    for numeric_prefix, root in file:gmatch('(%d+)-(.+)') do

[15.3869]

[15.4022]

  -- load files from save dir
  for _,filename in ipairs(love.filesystem.getDirectoryItems('')) do
    for numeric_prefix, root in filename:gmatch('(%d+)-(.+)') do

Replacement in live.lua at line 64 [15.722]

B:BD[15.4046] → [15.4046:4132]

B:BD[15.4132] → [6.47:84]

∅:D[6.84] → [15.4167:4253]

B:BD[15.4167] → [15.4167:4253]

      local index = tonumber(numeric_prefix)
      -- skip
      if index ~= nil then
        if root ~= 'fwmanifest' then
          assert(index == #result+1)
          table.insert(result, root)
        end

[15.4046]

[15.4253]

      if tonumber(numeric_prefix) > 0 then  -- skip 0000
        Live.filename[root] = filename
        table.insert(Live.filenames_to_load, filename)
        Live.final_prefix = math.max(Live.final_prefix, tonumber(numeric_prefix))

Replacement in live.lua at line 71 [15.722]
B:BD[15.4277] → [15.4277:4293]
```
  return result
```
[15.4277]
[15.4293]
```
  table.sort(Live.filenames_to_load)
  live.load_all(files)
```

Replacement in live.lua at line 75 [15.722]

B:BD[15.4298] → [15.4298:4342]

B:BD[15.4342] → [18.8:35]

B:BD[18.35] → [17.482:519]

B:BD[17.519] → [13.202:383]

∅:D[13.383] → [15.4405:4496]

B:BD[15.4405] → [15.4405:4496]

B:BD[15.4496] → [18.36:80]

∅:D[18.80] → [15.4598:4612]

B:BD[15.4598] → [15.4598:4612]

B:BD[15.4612] → [18.81:154]

function live.load_everything_in_manifest()
  local files_to_load = {}
  for k,v in pairs(Live.manifest) do
    -- Most keys in the manifest are definitions. If we need to store any
    -- metadata we'll do it in keys starting with a specific prefix.
    if not starts_with(k, 'fw_') then
      local root, index = k, v
      local filename = live.versioned_filename(index, root)
      table.insert(files_to_load, filename)
    end
  end
  table.sort(files_to_load)
  for _,filename in ipairs(files_to_load) do

[15.4298]

[18.154]

function live.load_all()
  for _,filename in ipairs(Live.filenames_to_load) do
--?     print('loading', filename)

Replacement in live.lua at line 81 [15.722]

B:BD[4.47] → [4.47:147]

    if status == nil then
      error(('error loading %s from manifest: %s'):format(filename, err))

[4.47]

[4.147]

    if not status then
      return err

Deletion in live.lua at line 86 [15.722]
B:BD[15.4616] → [13.384:406]
B:BD[13.406] → [3.8:23]
```
PARENT = 'fw_parent'
APP = 'fw_app'
```

Deletion in live.lua at line 87 [15.722]

B:BD[15.4617] → [15.4617:4708]

function live.versioned_filename(index, root)
  return ('%04d-%s'):format(index, root)
end

Replacement in live.lua at line 88 [15.722]
B:BD[15.4709] → [15.4709:4749]
B:BD[15.4749] → [5.9:52]
∅:D[5.52] → [15.4790:4794]
B:BD[15.4790] → [15.4790:4794]
```
function live.versioned_manifest(index)
  return ('%04d-fwmanifest'):format(index)
end
```
[15.4709]
[15.4794]
```
APP = 'fw_app'
```

Replacement in live.lua at line 158 [15.722]

B:BD[15.6312] → [3.24:138]

∅:D[3.138] → [16.711:763]

B:BD[15.6312] → [16.711:763]

    Live.manifest[APP] = love.filesystem.getIdentity()  -- doesn't need to be persisted, but no harm if it does..
    live.send_to_driver(json.encode(Live.manifest))

[15.6312]

[15.6354]

    Live.filename[APP] = love.filesystem.getIdentity()
    live.send_to_driver(json.encode(Live.filename))

Replacement in live.lua at line 161 [15.722]

B:BD[15.6384] → [19.12:309]

∅:D[19.309] → [15.6574:6619]

∅:D[17.765] → [15.6574:6619]

B:BD[15.6574] → [15.6574:6619]

B:BD[15.6619] → [19.310:358]

∅:D[19.358] → [20.11:35]

B:BD[17.806] → [20.11:35]

    local definition_name = buf:match('^%S+%s+(%S+)')
    Live.manifest[definition_name] = nil
    live.eval(definition_name..' = nil')  -- ignore errors which will likely be from keywords like `function = nil`
    local next_filename = live.versioned_filename(Live.next_version, definition_name)
    love.filesystem.write(next_filename, '')
    table.insert(Live.history, definition_name)
    live.roll_forward()

[15.6384]

[15.6977]

    local definition_name = buf:match('^%s*%S+%s+(%S+)')
    if Live.filename[definition_name] then
      local index = table.find(Live.filenames_to_load, Live.filename[definition_name])
      table.remove(Live.filenames_to_load, index)
      live.eval(definition_name..' = nil')  -- ignore errors which will likely be from keywords like `function = nil`
      love.filesystem.remove(Live.filename[definition_name])
      Live.filename[definition_name] = nil
    end

Replacement in live.lua at line 170 [15.722]

B:BD[15.7004] → [19.359:413]

    local definition_name = buf:match('^%S+%s+(%S+)')

[15.7004]

[19.413]

    local definition_name = buf:match('^%s*%S+%s+(%S+)')

Deletion in live.lua at line 200 [15.722]
∅:D[21.132] → [19.832:918]
B:BD[22.1415] → [19.832:918]
∅:D[19.918] → [15.7228:7274]
∅:D[17.1203] → [15.7228:7274]
B:BD[15.7228] → [15.7228:7274]
B:BD[15.7274] → [19.919:1022]
∅:D[19.1022] → [20.36:60]
B:BD[17.1291] → [20.36:60]
```
    local next_filename = live.versioned_filename(Live.next_version, definition_name)
    love.filesystem.write(next_filename, buf)
    table.insert(Live.history, definition_name)
    Live.manifest[definition_name] = Live.next_version
    live.roll_forward()
```
Deletion in live.lua at line 202 [15.722]
B:BD[15.7741] → [20.61:84]
```
      live.roll_back()
```

Insertion in live.lua at line 206 [15.722]

[23.30]

[24.9]

    -- eval succeeded without errors; persist the definition
    local filename = Live.filename[definition_name]
    if filename == nil then
      Live.final_prefix = Live.final_prefix+1
      filename = ('%04d-%s'):format(Live.final_prefix, definition_name)
      table.insert(Live.filenames_to_load, filename)
      Live.filename[definition_name] = filename
    end
    love.filesystem.write(filename, buf)

Deletion in live.lua at line 220 [15.722]

B:BD[15.8038] → [15.8038:8043]

B:BD[15.8043] → [20.85:839]

end
-- update Live.Head and record the new Live.Manifest (which caller has already modified)
function live.roll_forward()
  Live.manifest[PARENT] = Live.head
  local manifest_filename = live.versioned_manifest(Live.next_version)
  love.filesystem.write(manifest_filename, json.encode(Live.manifest))
  Live.head = Live.next_version
  love.filesystem.write('head', tostring(Live.head))
  Live.next_version = Live.next_version + 1
end
-- update app.Head and reload app.Manifest appropriately
function live.roll_back()
  Live.head = Live.manifest[PARENT]
  love.filesystem.write('head', tostring(Live.head))
  local previous_manifest_filename = live.versioned_manifest(Live.head)
  Live.manifest = json.decode(love.filesystem.read(previous_manifest_filename))

Replacement in live.lua at line 227 [15.722]

B:BD[15.8075] → [17.1810:1924]

  if Live.manifest[name] then
    return love.filesystem.read(live.versioned_filename(Live.manifest[name], name))

[15.8075]

[15.8189]

  if Live.filename[name] then
    return love.filesystem.read(Live.filename[name])

Deletion in README.md at line 8 [25.6]

B:BD[14.813454] → [14.813454:813548]


[More information about the on-disk representation of freewheeling apps.](representation.md)

Insertion in README.md at line 21 [25.6]
[14.813669]
[26.222]
```
## Hacking
```

Insertion in README.md at line 29 [25.6]

[14.813908]

[26.504]


To publish your changes:
  * delete all files with a numeric prefix from the repo, and then
  * move all files with a numeric prefix from the save dir to the repo

File addition: 0000-freewheeling-start (----------)

[11.2]

This file contains no definition, but is used as a marker in the save dir to
indicate all definitions have been copied from the repo to the save dir.