Trust tensorflow's thread safety (it wasn't the cause of an earlier bug)

[?]
Jul 6, 2021, 10:48 AM
CWOSQTC4HIBTZMMASA34RVBLPT2FRCVGEOIYUZXISNKMDNILV3WQC

Dependencies

  • [2] TCMUFA6E Try some different hyperparameters
  • [3] 4IXWCEHM Gamma correction
  • [4] SJHJS463 Model trainer: main function to open sdl window and launch second thread for actual training
  • [5] E742MTJA Fix segfault (don't pass pointers between functions in different SDL versions), minor refactor
  • [6] 2ABZP2KN Model trainer: save map.png after training
  • [7] ALMG52BO model trainer: speed up by waiting 15 seconds between monitor renders
  • [8] EB3DTD43 Model trainer: use ground altitude instead of crust mass distribution
  • [9] ZRPV3GAJ Model trainer: Add function to make tensor of cartesian coordinates for drawing the map (equirectangular projection)
  • [10] 7ML3OFE7 Model trainer: initial train and visualize thread pair (total crust mass; todo: altitude instead)
  • [11] FL3C6ERZ Assume python is being run with -m switch
  • [12] ROQCAPZJ Begin function for showing the map (for now just opens SDL window)
  • [*] ZM2EMAZO Start doing python multi-module stuff properly

Change contents

  • edit in trainmodel/src/model.py at line 36
    [4.2054][4.2054:2238](),[4.2238][2.139:191](),[2.191][4.2284:2530](),[4.2284][4.2284:2530]()
    class Monitor(keras.callbacks.Callback):
    def __init__(self, target):
    self.lock = threading.Lock()
    self.target = target
    self.stable = clone_my_model(target)
    def on_train_batch_end(self, batch, logs=None):
    self.lock.acquire()
    self.stable.set_weights(self.target.get_weights())
    self.lock.release()
    def snapshot(self):
    self.lock.acquire()
    result = self.stable
    self.lock.release()
    return result
  • replacement in trainmodel/src/drawmap.py at line 12
    [4.54][4.535:553]()
    def run(monitor):
    [4.54]
    [4.98]
    def run(m):
  • replacement in trainmodel/src/drawmap.py at line 22
    [4.82][4.82:127]()
    outputs = monitor.snapshot()(inputs)
    [4.82]
    [3.0]
    outputs = m(inputs)
  • replacement in trainmodel/src/__main__.py at line 9
    [4.30][4.50:80]()
    def train(monitor, filename):
    [4.30]
    [4.51]
    def train(m, filename):
  • replacement in trainmodel/src/__main__.py at line 12
    [4.129][2.192:303]()
    monitor.target.fit(x=training_data[0], y=training_data[1], batch_size=100, epochs=5000, callbacks=monitor)
    [4.129]
    [4.128]
    m.fit(x=training_data[0], y=training_data[1], batch_size=100, epochs=5000)
  • replacement in trainmodel/src/__main__.py at line 15
    [4.110][4.110:175]()
    outputs = tf.reshape(monitor.target(inputs), (1024,2048, 1))
    [4.110]
    [3.101]
    outputs = tf.reshape(m(inputs), (1024,2048, 1))
  • replacement in trainmodel/src/__main__.py at line 28
    [4.390][4.390:508]()
    monitor = model.Monitor(m)
    bg_thread = threading.Thread(target=train, args=(monitor, sys.argv[1]), kwargs={})
    [4.390]
    [4.230]
    bg_thread = threading.Thread(target=train, args=(m, sys.argv[1]), kwargs={})
  • replacement in trainmodel/src/__main__.py at line 30
    [4.252][4.509:534]()
    drawmap.run(monitor)
    [4.252]
    [4.270]
    drawmap.run(m)