akkartik/template-live - Change QFFTXR7EODYP3OPAO5ZJ4UDLKPBYBU2ESWHP3ABH3BEBYBIONCWAC

new file-system format for freewheeling apps

No more version history, now we have just the contents of the current version.
Editing a definition no longer changes the order in which definitions load.

This should make repos easier to browse, and more amenable to modify. You don't need driver.love anymore. And a stable order eliminates some gotchas. For example:

using driver.love, define Foo = 3 in a definition define Bar = Foo + 1 edit and redefine Foo = 4

Before this commit, you'd get an error when you restart the app. Definitions used to be loaded in version order, and editing a definition would move it to the end of the load order, potentially after definitions using it. I mostly avoided this by keeping top-level definitions independent. It's fine to refer to any definition inside a function body, we only need to be careful with initializers for global variables which run immediately while loading.

After this commit you can still end up in a weird state if you modify a definition that other later definitions use. In the above example, you will now see Foo = 4 and Bar = 4. But when you restart, Foo = 4 and Bar = 5. But that's no more confusing than Emacs's C-x C-e. It's still a good idea to keep top-level definitions order-independent. It's just confusing in a similar way to existing tools if you fail to do so. And your tools won't tend to break as badly.

Why did I ever do my weird version history thing? I think it's my deep aversion to risking losing any data entered. (Even though the app currently will seem to lose data in those situations. You'd need to leave your tools to find the data.) Now I rely on driver.love's undo to avoid data loss, but once you shut it down you're stuck with what you have on disk. Or in git.

I also wasn't aware for a long time of any primitives for deleting files. This might have colored my choices a lot.

Created by Kartik K. Agaram on April 16, 2023

QFFTXR7EODYP3OPAO5ZJ4UDLKPBYBU2ESWHP3ABH3BEBYBIONCWAC

Dependencies

In channels

main

Change contents

File deletion: tools
BF:BFD[14.2] → [9.2:18]
BF:BFD[9.18] → [9.1:1]

File deletion: stitch-live.lua

BF:BFD[9.1] → [9.793:832]

BF:BFD[9.832] → [9.20:20]

B:BD[9.20] → [9.21:792]

json = require 'json'
function main(args)
  local infile = io.open(args[1])
  local manifest_s = infile:read('*a')
  infile:close()
  local manifest = json.decode(manifest_s)
  local core_filenames = {}
  for k,v in pairs(manifest) do
    if not starts_with(k, 'fw_') then
      table.insert(core_filenames, k)
    end
  end
  table.sort(core_filenames)
  for _,core in ipairs(core_filenames) do
    local filename = ('%04d'):format(manifest[core])..'-'..core
    local f = io.open(filename)
    if f then
      print(f:read('*a'))
      print('')
    end
  end
end
function starts_with(s, prefix)
  if #s < #prefix then
    return false
  end
  for i=1,#prefix do
    if s:sub(i,i) ~= prefix:sub(i,i) then
      return false
    end
  end
  return true
end
main(arg)

File deletion: 0001-fwmanifest
BF:BFD[14.2] → [9.1761:1800]
BF:BFD[9.1800] → [9.1736:1736]
B:BD[9.1736] → [9.1737:1760]
```
{"fw_parent":0,"on":1}
```

File deletion: representation.md

BF:BFD[14.2] → [15.1780:1821]

BF:BFD[15.1821] → [15.1:1]

B:BD[15.1] → [15.2:1779]

B:BD[15.1779] → [6.1:683]

# The on-disk representation of freewheeling apps
When you start up a freewheeling app, you'll see a directory printed out in
the parent terminal (always launch it from a terminal window):
```
new edits will go to /home/...
```
When editing such an app using the driver (see [README.md](README.md)), new
definitions will go into this directory. Let's call it `$SAVE_DIR` in the rest
of this doc.
It is always safe to move such definitions into this repo. (We'll call it `.`
in the rest of this doc.) You'll want to do this if you're sharing them with
others, and it's also helpful if the driver crashes on your app. Moving
definitions will never, ever change app behavior.
```sh
$ mv -i $SAVE_DIR/[0-9]* .  # should never clobber any existing files
$ mv $SAVE_DIR/head .  # expected to clobber the existing file
```
Try looking inside the `head` file with a text editor. It'll contain a number,
the current version of the _manifest_ for this app. For example:
```
478
```
This means the current state of the app is in a file called `0478-fwmanifest`.
If you moved the files you should see such a file in `.`. If you open this
file, you'll see a JSON table from definition names to version ids. For
example:
```
{ "a": 273, "b": 478}
```
This means the current definition of `a` is in `0273-a` and of `b` in
`0478-b`.
Poking around these files gets repetitive, so there's a tool to streamline
things:
```
lua tools/stitch-live.lua 0478-fwmanifest
```
`stitch-live.lua` takes a manifest file as its argument, and prints out all
the definitions that make up the app at that version.
To compare two versions of the app, use `stitch-live.lua` to copy the
definitions in each into a separate file, and use a file comparison tool (e.g.
`diff`) to compare the two files.
# Scenarios considered in designing this representation
* Capture history of changes.
  - Though it is perhaps too fine-grained and noisy.
* Merge changes from non-live forks to live ones.
  - New files in repo can't hide changes in save dir, because filenames are
    always disjoint between the two.
  - This doesn't apply yet to live updates. Two forks of a single live app
    will likely have unnecessary merge conflicts.
* No special tools required to publish changes to others.
  - Just move files from save dir to repo.
# Scenarios I _would_ like to take into consideration in the future
* Cleaner commits; it's clear what changed.
* merge changes between live forks.

File deletion: head
BF:BFD[14.2] → [14.21798:21826]
BF:BFD[14.21826] → [14.21794:21794]
B:BD[14.21794] → [9.837:839]
```
1
```

Replacement in main.lua at line 4 [14.4]

B:BD[14.170] → [7.2:146]

--    from the numeric prefix in file 'head', obtain a manifest
--    load all files (which must start with a numeric prefix) from the manifest

[14.170]

[14.314]

--    load files in order

Replacement in main.lua at line 10 [14.4]

B:BD[14.452] → [14.452:520]

B:BD[14.520] → [8.2:74]

∅:D[8.74] → [14.588:612]

B:BD[14.588] → [14.588:612]

B:BD[14.612] → [8.75:179]

--  on incoming messages to a specific file, however, the app must:
--    save the message's value to a new, smallest unused numeric prefix
--    execute the value
--      if there's an error, go back to the previous value of the same
--      definition if one exists

[14.452]

[14.612]

--  on incoming messages to a specific file, the app must:
--    determine the definition name from the first word
--    execute the value, returning any errors
--    look up the filename for the definition or define a new filename for it
--    save the message's value to the filename

Replacement in main.lua at line 30 [14.4]

B:BD[14.951] → [14.951:1001]

-- ========= on startup, load the version at head

[14.951]

[14.1001]

-- === on startup, load all files with numeric prefix

Deletion in main.lua at line 33 [14.4]

B:BD[14.1040] → [14.1040:1269]

  -- version control
  app.Head = 0
  app.Next_version = 1
  app.History = {}  -- array of filename roots corresponding to each numeric prefix
  app.Manifest = {}  -- mapping from roots to numeric prefixes as of version app.Head

Insertion in main.lua at line 34 [14.4]

[16.45]

[14.1269]


  app.Filenames_to_load = {}  -- filenames in order of numeric prefix
  app.Filename = {}  -- map from definition name to filename (including numeric prefix)
  app.Final_prefix = 0

Replacement in main.lua at line 84 [14.4]

B:BD[14.1601] → [14.1601:3082]

B:BD[14.3082] → [3.1:38]

∅:D[3.38] → [14.3117:3272]

B:BD[14.3117] → [14.3117:3272]

  local files = {}
  app.append_files_with_numeric_prefix('', files)
  table.sort(files)
  app.check_integrity(files)
  app.append_files_with_numeric_prefix(love.filesystem.getSaveDirectory(), files)
  table.sort(files)
  app.check_integrity(files)
  app.History = app.load_history(files)
  app.Next_version = #app.History + 1
  local head_string = love.filesystem.read('head')
  app.Head = tonumber(head_string)
  if app.Head > 0 then
    app.Manifest = json.decode(love.filesystem.read(app.versioned_manifest(app.Head)))
  end
  app.load_everything_in_manifest()
end
function app.append_files_with_numeric_prefix(dir, files)
  for _,file in ipairs(love.filesystem.getDirectoryItems(dir)) do
    table.insert(files, file)
  end
end
function app.check_integrity(files)
  local manifest_found, file_found = false, false
  local expected_index = 1
  for _,file in ipairs(files) do
    for numeric_prefix, root in file:gmatch('(%d+)-(.+)') do
      -- only runs once
      local index = tonumber(numeric_prefix)
      -- skip files without numeric prefixes
      if index ~= nil then
        if index < expected_index then
          print(index, expected_index)
        end
        assert(index >= expected_index)
        if index > expected_index then
          assert(index == expected_index+1)
          assert(manifest_found and file_found)
          expected_index = index
          manifest_found, file_found = false, false
        end
        assert(index == expected_index)
        if root == 'fwmanifest' then
          assert(not manifest_found)
          manifest_found = true
        else
          assert(not file_found)
          file_found = true
        end

[14.1601]

[14.3272]

  -- if necessary, copy files from repo to save dir
  if io.open(love.filesystem.getSaveDirectory()..'/0000-freewheeling-start') == nil then
    print('copying all definitions from repo to save dir')
    for _,filename in ipairs(love.filesystem.getDirectoryItems('')) do
      for numeric_prefix, root in filename:gmatch('(%d+)-(.+)') do
        -- only runs once
        local buf = love.filesystem.read(filename)
        print('copying', filename)
        love.filesystem.write(filename, buf)

Replacement in main.lua at line 96 [14.4]

B:BD[14.3296] → [14.3296:3448]

end
function app.load_history(files)
  local result = {}
  for _,file in ipairs(files) do
    for numeric_prefix, root in file:gmatch('(%d+)-(.+)') do

[14.3296]

[14.3448]

  -- load files from save dir
  for _,filename in ipairs(love.filesystem.getDirectoryItems('')) do
    for numeric_prefix, root in filename:gmatch('(%d+)-(.+)') do

Replacement in main.lua at line 100 [14.4]

B:BD[14.3472] → [14.3472:3558]

B:BD[14.3558] → [3.39:76]

∅:D[3.76] → [14.3593:3679]

B:BD[14.3593] → [14.3593:3679]

      local index = tonumber(numeric_prefix)
      -- skip
      if index ~= nil then
        if root ~= 'fwmanifest' then
          assert(index == #result+1)
          table.insert(result, root)
        end

[14.3472]

[14.3679]

      if tonumber(numeric_prefix) > 0 then  -- skip 0000
        app.Filename[root] = filename
        table.insert(app.Filenames_to_load, filename)
        app.Final_prefix = math.max(app.Final_prefix, tonumber(numeric_prefix))

Replacement in main.lua at line 107 [14.4]
B:BD[14.3703] → [14.3703:3719]
```
  return result
```
[14.3703]
[14.3719]
```
  table.sort(app.Filenames_to_load)
  app.load_all(files)
```

Replacement in main.lua at line 111 [14.4]

B:BD[14.3724] → [17.4:56]

∅:D[17.56] → [14.3724:3767]

B:BD[14.3724] → [14.3724:3767]

B:BD[14.3767] → [18.1:28]

∅:D[18.28] → [14.3767:3803]

B:BD[14.3767] → [14.3767:3803]

B:BD[14.3803] → [19.1:39]

∅:D[19.39] → [14.3829:3919]

B:BD[14.3829] → [14.3829:3919]

B:BD[14.3919] → [18.29:73]

∅:D[18.73] → [14.4020:4034]

B:BD[14.4020] → [14.4020:4034]

B:BD[14.4034] → [18.74:147]

-- return any error encountered, or nil if all well
function app.load_everything_in_manifest()
  local files_to_load = {}
  for k,v in pairs(app.Manifest) do
    if not starts_with(k, 'fw_') then
      local root, index = k, v
      local filename = app.versioned_filename(index, root)
      table.insert(files_to_load, filename)
    end
  end
  table.sort(files_to_load)
  for _,filename in ipairs(files_to_load) do

[14.3724]

[18.147]

function app.load_all()
  for _,filename in ipairs(app.Filenames_to_load) do
--?     print('loading', filename)

Deletion in main.lua at line 123 [14.4]
B:BD[19.41] → [19.41:62]
```
PARENT = 'fw_parent'
```
Deletion in main.lua at line 124 [14.4]
∅:D[2.16] → [14.4038:4129]
∅:D[19.62] → [14.4038:4129]
B:BD[14.4038] → [14.4038:4129]
```
function app.versioned_filename(index, root)
  return ('%04d-%s'):format(index, root)
end
```
Deletion in main.lua at line 125 [14.4]
B:BD[14.4130] → [14.4130:4169]
B:BD[14.4169] → [3.77:120]
∅:D[3.120] → [14.4210:4215]
B:BD[14.4210] → [14.4210:4215]
```
function app.versioned_manifest(index)
  return ('%04d-fwmanifest'):format(index)
end
```

Replacement in main.lua at line 198 [14.4]

B:BD[14.5929] → [2.17:130]

∅:D[2.130] → [14.5929:5969]

B:BD[14.5929] → [14.5929:5969]

    app.Manifest[APP] = love.filesystem.getIdentity()  -- doesn't need to be persisted, but no harm if it does..
    app.send(json.encode(app.Manifest))

[14.5929]

[14.5969]

    app.Filename[APP] = love.filesystem.getIdentity()
    app.send(json.encode(app.Filename))

Replacement in main.lua at line 201 [14.4]

B:BD[14.5999] → [13.5:298]

∅:D[13.298] → [14.6185:6230]

B:BD[14.6185] → [14.6185:6230]

B:BD[14.6230] → [13.299:346]

∅:D[13.346] → [20.4:27]

B:BD[14.6447] → [20.4:27]

    local definition_name = buf:match('^%S+%s+(%S+)')
    app.Manifest[definition_name] = nil
    app.eval(definition_name..' = nil')  -- ignore errors which will likely be from keywords like `function = nil`
    local next_filename = app.versioned_filename(app.Next_version, definition_name)
    love.filesystem.write(next_filename, '')
    table.insert(app.History, definition_name)
    app.roll_forward()

[14.5999]

[14.6577]

    local definition_name = buf:match('^%s*%S+%s+(%S+)')
    if app.Filename[definition_name] then
      local index = table.find(app.Filenames_to_load, app.Filename[definition_name])
      print('index')
      table.remove(app.Filenames_to_load, index)
      for i,v in ipairs(app.Filenames_to_load) do
        print(i, v)
      end
      app.eval(definition_name..' = nil')  -- ignore errors which will likely be from keywords like `function = nil`
      print(_G[definition_name])
      love.filesystem.remove(app.Filename[definition_name])
      print('deleted')
      app.Filename[definition_name] = nil
    end

Replacement in main.lua at line 216 [14.4]

B:BD[14.6604] → [13.347:401]

    local definition_name = buf:match('^%S+%s+(%S+)')

[14.6604]

[13.401]

    local definition_name = buf:match('^%s*%S+%s+(%S+)')

Deletion in main.lua at line 239 [14.4]

B:BD[16.1386] → [13.801:885]

∅:D[13.885] → [14.6824:6870]

B:BD[14.6824] → [14.6824:6870]

B:BD[14.6870] → [13.886:986]

∅:D[13.986] → [20.28:51]

B:BD[14.7132] → [20.28:51]

    local next_filename = app.versioned_filename(app.Next_version, definition_name)
    love.filesystem.write(next_filename, buf)
    table.insert(app.History, definition_name)
    app.Manifest[definition_name] = app.Next_version
    app.roll_forward()

Deletion in main.lua at line 241 [14.4]
B:BD[14.7323] → [21.4:26]
```
      app.roll_back()
```

Insertion in main.lua at line 244 [14.4]

[10.17]

[14.7600]

    end
    -- eval succeeded without errors; persist the definition
    local filename = app.Filename[definition_name]
    if filename == nil then
      app.Final_prefix = app.Final_prefix+1
      filename = ('%04d-%s'):format(app.Final_prefix, definition_name)
      table.insert(app.Filenames_to_load, filename)
      app.Filename[definition_name] = filename

Insertion in main.lua at line 253 [14.4]
[14.7608]
[5.1]
```
    love.filesystem.write(filename, buf)
```

Replacement in main.lua at line 258 [14.4]

B:BD[21.32] → [11.4:91]

∅:D[11.91] → [20.52:80]

B:BD[21.32] → [20.52:80]

B:BD[20.80] → [11.92:265]

∅:D[11.265] → [22.4:86]

B:BD[20.80] → [22.4:86]

∅:D[22.86] → [20.116:163]

B:BD[20.116] → [20.116:163]

B:BD[20.163] → [11.266:323]

∅:D[20.163] → [21.32:57]

∅:D[11.323] → [21.32:57]

B:BD[21.32] → [21.32:57]

B:BD[21.57] → [22.87:173]

∅:D[23.122] → [21.143:292]

∅:D[22.173] → [21.143:292]

B:BD[21.143] → [21.143:292]

-- update app.Head and record the new app.Manifest (which caller has already modified)
function app.roll_forward()
  app.Manifest[PARENT] = app.Head
  local manifest_filename = app.versioned_manifest(app.Next_version)
  love.filesystem.write(manifest_filename, json.encode(app.Manifest))
  app.Head = app.Next_version
  love.filesystem.write('head', tostring(app.Head))
  app.Next_version = app.Next_version + 1
end
-- update app.Head and reload app.Manifest appropriately
function app.roll_back()
  app.Head = app.Manifest[PARENT]
  love.filesystem.write('head', tostring(app.Head))
  local previous_manifest_filename = app.versioned_manifest(app.Head)
  app.Manifest = json.decode(love.filesystem.read(previous_manifest_filename))

[21.32]

[14.7614]

function table.find(h, x)
  for k,v in pairs(h) do
    if v == x then
      return k
    end
  end

Replacement in main.lua at line 267 [14.4]

B:BD[14.7650] → [14.7650:7761]

  if app.Manifest[name] then
    return love.filesystem.read(app.versioned_filename(app.Manifest[name], name))

[14.7650]

[14.7761]

  if app.Filename[name] then
    return love.filesystem.read(app.Filename[name])

Deletion in README.md at line 8 [24.4663627]
∅:D[12.11343] → [24.4663929:4664023]
B:BD[24.4663929] → [24.4663929:4664023]
```
[More information about the on-disk representation of freewheeling apps.](representation.md)
```
Insertion in README.md at line 12 [24.4663627]
[24.4664159]
[24.4664159]
```
## Hacking
```

Insertion in README.md at line 22 [24.4663627]

[24.4664476]

To publish your changes:
  * delete all files with a numeric prefix from the repo, and then
  * move all files with a numeric prefix from the save dir to the repo

Deletion in README.md at line 33 [24.4663627]

B:BD[24.4664493] → [4.260:738]

* This app encourages a style of development that requires top-level
  definitions to be decoupled from each other. No live functions load until
  all definitions have been run. However top-level globals are initialized as
  they're loaded. This makes a definition like this a very bad idea, assuming
  `Foo` and `Bar` are top-level variables:
    Foo = {
      Bar
    }
  Don't assume that Bar will exist when Foo is defined. We don't currently
  have any checks for this.

File addition: 0000-freewheeling-start (----------)

[14.2]

This file contains no definition, but is used as a marker in the save dir to
indicate all definitions have been copied from the repo to the save dir.

new file-system format for freewheeling apps

Dependencies

In channels

Change contents

File deletion: tools

File deletion: stitch-live.lua

File deletion: 0001-fwmanifest

File deletion: representation.md

File deletion: head