Documentation and cache performance update
[?]
Sep 22, 2021, 5:02 PM
7M2QIAYK7RM2YDZ3DZMYNJQH26H4676VBINAW3VSBMJQJE4RI4MQCDependencies
- [2]
AQMU4WCOtypo fix - [3]
FSXHAVRFUpdate for IntelliJ 2021.2.2 - [4]
Q35OTML2Remove use of coroutines, which was blocking IntelliJ UI in larger repositories Improvements for bigger repositories, now Dracon caches the changes that happened in a revision in a file, so everytime Dracon needs to query the changes of a revision, it loads directly from memory instead of doing a full-scan in Pijul repository. For tiny projects it is not a problem, but for medium ones it takes more than five minutes to scan the entire repository (and it was tested with a repo of only 700 records, however there was changes that had more than 60.000 lines). The cache file is saved in IntelliJ Data Path (project specific) and is compressed with gzip, so it will not use so much disk space (the cost worths the gains). - [5]
37OJKSWJImproved caching code a lot - [6]
4TGL4RKFFix concurrent pijul execution locking each other - [7]
QXUEMZ3BInitial CahngeProvider - [8]
MZYZIVHYFirst experimental build, it seems like it is breaking Git plugin, however, it still something =D - [9]
FRFFQV7VBasic show history support. - [10]
ISO7J5ZHMore caches, better and generic cache code. Now Dracon listen to file changes to drop cached data. Implemented caches: - File contents in specific revision (never dropped) - Pijul ls and Pijul diff results - File Revision and File changes by patch - some others.. Dracon is incredible fast now, but still will take some time for bigger repos. - [11]
VBL5BQH7Experimental - [12]
2N67RQZCAdd auto installation support and cache content of ContentRevision - [13]
Q7FXTHVUFirst record support, YEAAAH, RECOOORD - [14]
7L5LODGZParse changes from `pijul change` - [*]
FNNW5IEAAdded more plugin files to Pijul - [*]
GGYFPXNDInitial plugin - [*]
EAGIDXOLBuild 2 to only listen to changes in project under pijul
Change contents
- replacement in "src/main/kotlin/com/github/jonathanxd/dracon/revision/PijulVcsFileRevision.kt" at line 30
class PijulVcsFileRevision(val project: Project,class PijulVcsFileRevision constructor(val project: Project, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulRecordDetails.kt" at line 25
project, entry.id, emptyList(), entry.commitTime, rootFile, entry.subject, entry.author,project, entry.id, emptyList() /*entry.getPijulParents()*/, entry.commitTime, rootFile, entry.subject, entry.author, - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLogProvider.kt" at line 343
// Use empty parents since Pijul hierarchy is not linear// this leads to a problem shown in the [pijul-log-tree.png] file.val parents = emptyList<Hash>()//it.parents - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLogProvider.kt" at line 349
emptyList(),parents, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 134
val path: String,val rootPath: String,val path: Filename,val rootPath: FileDir, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 145
this.rootPath == "/" -> this.paththis.rootPath.endsWith("/") -> this.rootPath + this.paththis.rootPath.isEmpty() -> this.pathelse -> this.rootPath + "/" + this.paththis.rootPath.dir == "/" -> this.path.namethis.rootPath.dir.endsWith("/") -> (this.rootPath + this.path).fullPaththis.rootPath.dir.isEmpty() -> this.path.nameelse -> (this.rootPath + this.path).fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 159
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 168
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 179
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 188
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 198
val older: String,val new: String,val older: FilePath,val new: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 206
get() = this.newget() = this.new.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 216
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 226
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 236
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 246
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 256
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 264
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 274
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 283
get() = this.pathget() = this.path.fullPath - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 293
val path: String,val path: FilePath, - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 302
get() = this.pathget() = this.path.fullPath - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 406
val HUNK_ZOMBIE_PATTERN = Regex("(\\d+)\\. Resurrecting zombie lines in (\"[^\"]+\"):(\\d+) (\\d+\\.\\d+)\\n?") - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 407
val CHANGE_PATTERN = Regex("([-+]) (.*)\\n?") - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 409
// Corner case: :D 9.1256 -> 4.12200:12305/9, B:BD 4.12200 -> 4.12200:12305/4 - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 413
val FLAG_PATTERN = Regex("([BFD]+:[BFD]+ [^\\n]+)\\n?")val FLAG_PATTERN = Regex("(([BFD]+:[BFD]+ [^\\n]+)|(:[BFD]+ [^\\n]+))\\n?") - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 507
val fileOrPath = add.fileNameval rootPath = add.pathval fileOrPath = add.fileName.parseFilename()val rootPath = add.path.parseFileDir() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 535
val path = remove.fileNameval path = remove.fileName.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 563
val path = undel.fileNameval path = undel.fileName.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 591
val new = move.newval old = move.oldval new = move.new.parseFilePath()val old = move.old.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 618
val path = edit.fileval path = edit.file.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 651
val path = replacement.fileNameval path = replacement.fileName.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 685
val path = nameConflict.fileNameval path = nameConflict.fileName.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 715
val path = orderConflict.fileNameval path = orderConflict.fileName.parseFilePath() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 748
val path = zombieLines.fileNameval path = zombieLines.fileName.parseFilePath() - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/PijulLog.kt" at line 755
- edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 19
// Corner case: 120. File addition: "coroutine" in "src/main/kotlin/com/github/jonathanxd/dracon" +dx "binary" - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 21
Regex("(?<changeNumber>\\d+)\\. File addition: (\"(?<file>.*)(?:\" )+)in (\"(?<path>.*)(?:\" )+)(\"(?<encoding>[^\"]+)\")(?<meta> [^ ]+)?\\n?")Regex("(?<changeNumber>\\d+)\\. File addition: (\"(?<file>.*)(?:\" )+)in (\"(?<path>.*)(?:\" )+)(?<mode>[^ ]+)?( ?\"(?<encoding>[^\"]+)\")(?<meta> [^ ]+)?\\n?") - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 44
// Corner case: 9. Moved: "src/main/kotlin/com/github/jonathanxd/dracon/handler" "handler" +dx 7.107 - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 46
Regex("(?<changeNumber>\\d+)\\. Moved: (\"(?<old>.*)(?:\" )+)(\"(?<new>.*)(?:\" )+)(?<inode>\\d+\\.\\d+)([^ ]+ )?( \"(?<encoding>[^\"]+)\")?([^ ]+ )?(?<meta>.*)\\n?")Regex("(?<changeNumber>\\d+)\\. Moved: (\"(?<old>.*)(?:\" )+)(\"(?<new>.*)(?:\" )+)((?<mode>[^ ]+) )?(?<inode>\\d+\\.\\d+)([^ ]+ )?( \"(?<encoding>[^\"]+)\")?([^ ]+ )?(?<meta>.*)\\n?") - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 83
val mode: String? get() = matchResult.groups["mode"]?.value - edit in "src/main/kotlin/com/github/jonathanxd/dracon/log/Patterns.kt" at line 108
val mode: String? get() = matchResult.groups["mode"]?.value - file addition: Filename.kt[16.6847]
package com.github.jonathanxd.dracon.logimport com.github.jonathanxd.dracon.cmd.removeTrailingSlash@JvmInlinevalue class Filename(val name: String)@JvmInlinevalue class FileDir(val dir: String)@JvmInlinevalue class FilePath(val fullPath: String)//\"src/test/kotlin/com/github/jonathanxd/dracon/test/DraconTest.kt\"fun String.parseFilePath(): FilePath = FilePath(this.unescape())fun String.parseFilename(): Filename = Filename(this.unescape())fun String.parseFileDir(): FileDir = FileDir(this.unescape())operator fun FileDir.plus(name: Filename) =FilePath(this.dir.resolve(name.name))operator fun FileDir.plus(name: FilePath) =FilePath(this.dir.resolve(name.fullPath))operator fun FilePath.plus(name: FilePath) =FilePath(this.fullPath.resolve(name.fullPath))fun String.resolve(other: String) ="${this.removePathTrailingSlash()}/${other.removePathLeadingSlash()}"const val ESCAPED_STRING_PREFIX = """\""""fun String.unescape(): String =if (this.startsWith(ESCAPED_STRING_PREFIX) && this.endsWith(ESCAPED_STRING_PREFIX))this.substring(ESCAPED_STRING_PREFIX.length, this.length - ESCAPED_STRING_PREFIX.length)else thisprivate fun String.removePathTrailingSlash() =if (this.endsWith("/")) this.substring(0, this.lastIndex)else thisprivate fun String.removePathLeadingSlash() =if (this.startsWith("/")) this.substring(1, this.length)else this - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/PijulLogRevisionCache.kt" at line 29
override val cache = DataCache<String, PijulRevisionNumber>(project, "revision")override val cache = ActorDataCache<String, PijulRevisionNumber>(project, "revision") - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/PijulLogEntryChangeCache.kt" at line 29
override val cache = DataCache<String, PijulLogEntry>(project, "path_change")override val cache = ActorDataCache<String, PijulLogEntry>(project, "path_change") - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/PijulLogEntryCache.kt" at line 22
override val cache = DataCache<String, PijulLogEntry>(project, "change")override val cache = ActorDataCache<String, PijulLogEntry>(project, "change") - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/FileStatusCache.kt" at line 30
override val cache = DataCache<String, CacheablePijulFileStatus>(this.project, "file_status")override val cache = ActorDataCache<String, CacheablePijulFileStatus>(this.project, "file_status") - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/FileStatusCache.kt" at line 34
this.cache.lock()try {this.cache.withLock { - edit in "src/main/kotlin/com/github/jonathanxd/dracon/cache/FileStatusCache.kt" at line 54
} finally {this.cache.unlock() - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/DataCache.kt" at line 32
const val DATA_CACHE_VERSION: Long = 2const val DATA_CACHE_VERSION: Long = 3 - edit in "src/main/kotlin/com/github/jonathanxd/dracon/cache/DataCache.kt" at line 250
// Do async, accept MISS while locked// cache storage is very slow due to compression - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/DataCache.kt" at line 265
return if (version != version) {return if (version != this.version) { - replacement in "src/main/kotlin/com/github/jonathanxd/dracon/cache/CacheService.kt" at line 26
val cache: DataCache<K, V>val cache: ActorDataCache<K, V>/*** Executes an operation in [cache] using the **cache write lock**.** This does not guarantee absolute lock over the file caching mechanism.*/fun <R> withLock(f: () -> R): R =this.cache.withLock(f) - file addition: ActorDataCache.kt[4.8571]
package com.github.jonathanxd.dracon.cacheimport com.github.jonathanxd.dracon.coroutine.DRACON_SCHEDULERimport com.intellij.execution.process.mediator.util.blockingGetimport com.intellij.openapi.project.Projectimport com.intellij.openapi.project.getProjectDataPathimport kotlinx.coroutines.*import kotlinx.coroutines.channels.actorimport kotlinx.coroutines.channels.trySendBlockingimport java.io.ObjectInputStreamimport java.io.ObjectOutputStreamimport java.nio.file.Filesimport java.nio.file.Pathimport java.nio.file.StandardOpenOptionimport java.time.Instantimport java.util.concurrent.ConcurrentHashMapimport java.util.concurrent.Executorsimport java.util.concurrent.locks.ReentrantLockimport java.util.zip.GZIPInputStreamimport java.util.zip.GZIPOutputStreamimport kotlin.concurrent.withLock/*** Denotes the current version of data cache, must be updated when data cache format changes.*/const val ACTOR_DATA_CACHE_VERSION: Long = 3/*** Actor based data cache storage, with support to cache misses and falling back to default resolver,* this allows writes to on-file cached to lock without freezing IntelliJ IDEA (when request for data happens in UI-thread).*/class ActorDataCache<K, V>(val project: Project,val name: String) {private val cachePath = project.getProjectDataPath("com.github.jonathanxd.dracon")private val cacheFile = cachePath.resolve("$name.gz")private val manager = ActorDataCacheManager<K, CachedValue<V>>(this.cacheFile, DATA_CACHE_VERSION)private val lock = ReentrantLock()private val updateExecutor = Executors.newSingleThreadExecutor()private val inMemory = ConcurrentHashMap<K, CachedValue<V>>()init {this.manager.load {inMemory.putAll(it)}}/*** Executes*/fun <R> withLock(f: () -> R) = this.lock.withLock(f)fun unload(key: K) {this.lock.withLock {this.inMemory.remove(key)}writeToDisk()}fun queryOrLoad(key: K, compute: (K) -> V): V {return this.lock.withLock {if (!this.inMemory.containsKey(key)) {val computeEntry = CachedValue(Instant.now(), compute(key))this.inMemory[key] = computeEntrywriteToDisk()computeEntry} else {this.inMemory[key]!!}.value}}fun <I> queryOrLoad(key: K,compute: (K) -> I,filter: (I) -> Boolean,iMapper: (I) -> V,vMapper: (V) -> I): I {return this.lock.withLock {if (!this.inMemory.containsKey(key)) {val computeEntry = compute(key)if (!filter(computeEntry)) {return computeEntry}this.inMemory[key] = CachedValue(Instant.now(), iMapper(computeEntry!!))writeToDisk()computeEntry} else {vMapper(this.inMemory[key]!!.value)}}}/*** This call completes asynchronously.*/private fun writeToDisk() {this.manager.write(this.inMemory)}/*** @see CacheService.invalidate*/fun invalidate() {this.lock.withLock {this.inMemory.clear()this.writeToDisk()}}/*** @see CacheService.invalidate*/fun invalidate(key: K) {this.lock.withLock {this.inMemory.remove(key)this.writeToDisk()}}/*** @see CacheService.updateCache*/fun updateCache(key: K, newValueCompute: () -> V) {val instant = Instant.now()this.lock.withLock {val inMemoryValue = this.inMemory[key]if (inMemoryValue == null || inMemoryValue.instant.isBefore(instant)) {this.inMemory[key] = CachedValue(instant, newValueCompute())}}}/*** @see CacheService.updateCache*/fun updateCache(keys: List<K>, newValueCompute: (K) -> V) {val instant = Instant.now()this.lock.withLock {for (key in keys) {val inMemoryValue = this.inMemory[key]if (inMemoryValue == null || inMemoryValue.instant.isBefore(instant)) {this.inMemory[key] = CachedValue(instant, newValueCompute(key))}}}}}sealed class CacheCommand<K, V> {class CacheLoad<K, V>(val completion: CompletableDeferred<Map<K, V>?>): CacheCommand<K, V>()class CacheSave<K, V>(val data: Map<K, V>, val completion: CompletableDeferred<Unit>): CacheCommand<K, V>()}// Do async, accept MISS while locked// cache storage is very slow due to compressionclass ActorDataCacheManager<K, V>(val path: Path, val version: Long) {@Suppress("BlockingMethodInNonBlockingContext")private val loader = CoroutineScope(Dispatchers.IO).actor<CacheCommand<K, V>> {for (msg in channel) {when (msg) {is CacheCommand.CacheLoad<K, V> -> {msg.completion.complete(if (!Files.exists(path)) {null} else {try {Files.newInputStream(path).use { stream ->GZIPInputStream(stream).use { gz ->ObjectInputStream(gz).use { reader ->val version = reader.readLong()if (version != this@ActorDataCacheManager.version) {emptyMap()} else {reader.readObject() as Map<K, V>}}}}} catch (t: Throwable) {t.printStackTrace()Files.delete(path)null}})}is CacheCommand.CacheSave<K, V> -> {Files.newOutputStream(path, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING).use { writer ->GZIPOutputStream(writer).use { compress ->ObjectOutputStream(compress).use { oos ->oos.writeLong(this@ActorDataCacheManager.version)oos.writeObject(msg.data)msg.completion.complete(Unit)}}}}}}}init {Files.createDirectories(this.path.parent)}fun load(onLoad: suspend (Map<K, V>) -> Unit) {CoroutineScope(DRACON_SCHEDULER).launch {val complete = CompletableDeferred<Map<K, V>?>()loader.send(CacheCommand.CacheLoad(complete))val load = complete.await()if (load != null) {onLoad(load)}}}fun write(data: Map<K, V>, onComplete: suspend () -> Unit = {}) {CoroutineScope(DRACON_SCHEDULER).launch {val complete = CompletableDeferred<Unit>()loader.send(CacheCommand.CacheSave(data, complete))complete.await()onComplete()}}} - file addition: pijul-no-log-tree.png[1.0]
- file addition: pijul-log-tree.png[1.0]
- file addition: pijul-log-tree-would-be.png[1.0]
- replacement in "README.md" at line 31
Dracon is in a heavily experimental state, and it depends on latest IDEA EAP build.Dracon is in a heavily experimental state, and it depends on IntelliJ 212.5284.40. - edit in "README.md" at line 89[18.1213353][2.0]
### Dracon log history does not show related recordsDracon shows history like:Instead of something link:This is because the relation of Pijul patches are not linear, because of this, the following happens whenwe try to correctly build the relation between records: