apraga/org - Change X54SCM4CJGBLTUD4UPHVDKVAQHXDCRSWSRSQKLTMO2A32ZPXM6AAC

Bug is actually in xamscissors

Created by Alexis Praga on August 18, 2023

X54SCM4CJGBLTUD4UPHVDKVAQHXDCRSWSRSQKLTMO2A32ZPXM6AAC

Dependencies

In channels

main

Change contents

Replacement in projects/bisonex.org at line 4 [4.35]

B:BD[3.8438] → [3.8438:8469]

∅:D[3.8469] → [5.8221:9022]

B:BD[5.8221] → [5.8221:9022]

∅:D[5.9022] → [6.8279:8402]

B:BD[6.8279] → [6.8279:8402]

∅:D[6.8402] → [7.8221:8284]

B:BD[7.8221] → [7.8221:8284]

∅:D[7.8284] → [8.8221:8448]

B:BD[8.8221] → [8.8221:8448]

∅:D[8.8448] → [9.8221:9494]

B:BD[9.8221] → [9.8221:9494]

∅:D[9.9494] → [10.8221:13895]

B:BD[10.8221] → [10.8221:13895]

:00>
:LOGBOOK:
CLOCK: [2023-0
7-30 Sun 19:13]--[2023-07-30 Sun 20:50] =>  1:37
:END:
Modification nécessaire pour kent :
- plus de patch
- suppression d'une boucle dans postPatch
On supprime aussi NIX_BUILD_TOP
*** DONE [[https://github.com/NixOS/nixpkgs/pull/186459][BioDBHTS]]
CLOSED: [2023-05-06 Sat 08:49] SCHEDULED: <2023-04-15 Sat>
/Entered on/ [2022-08-10 Wed 14:28]
Correction pour review faites <2022-10-10 Mon>
*** DONE [[https://github.com/NixOS/nixpkgs/pull/186464][BioExtAlign]]
CLOSED: [2022-10-22 Sat 12:43] SCHEDULED: <2022-08-10 Wed>
/Entered on/ [2022-08-10 Wed 14:28]
Review <2022-10-10 Mon>, correction dans la journée.
Correction 2e passe, attente
Impossible de faire marcher les tests Car il ne trouve pas le module Bio::Tools::Align, qui est dans un dossier ailleurs dans le dépôt. Même e
n compilant tout le dépôt, cela ne fonctionne pas... On skip les tests.
*** TODO VEP
** WAIT [[https://github.com/N
ixOS/nixpkgs/pull/230394][rtg-tools]] :vcfeval:
Soumis
** WAI
T Package Spip https://github.com/NixOS/nixpkgs/pull/247476
** TODO Happy :happy:
*** PROJ PR python 3 upstream
*** PROJ nixpkgs en l'état
** TODO Bamsurgeon
/Entered on/ [2023-05-13 Sat 19:11]
*** TODO Velvet
** TO

[3.8438]

[10.13895]

:00>
:LOGBOOK:
CLOCK: [2023-07-30 Sun 19:13]--[2023-07-30 Sun 20:50] =>  1:37
:END:
Modification nécessaire pour kent :
- plus de patch
- suppression d'une boucle dans postPatch
On supprime aussi NIX_BUILD_TOP
*** DONE [[https://github.com/NixOS/nixpkgs/pull/186459][BioDBHTS]]
CLOSED: [2023-05-06 Sat 08:49] SCHEDULED: <2023-04-15 Sat>
/Entered on/ [2022-08-10 Wed 14:28]
Correction pour review faites <2022-10-10 Mon>
*** DONE [[https://github.com/NixOS/nixpkgs/pull/186464][BioExtAlign]]
CLOSED: [2022-10-22 Sat 12:43] SCHEDULED: <2022-08-10 Wed>
/Entered on/ [2022-08-10 Wed 14:28]
Review <2022-10-10 Mon>, correction dans la journée.
Correction 2e passe, attente
Impossible de faire marcher les tests Car il ne trouve pas le module Bio::Tools::Align, qui est dans un dossier ailleurs dans le dépôt. Même en compilant tout le dépôt, cela ne fonctionne pas... On skip les tests.
*** TODO VEP
** WAIT [[https://github.com/NixOS/nixpkgs/pull/230394][rtg-tools]] :vcfeval:
Soumis
** WAIT Package Spip https://github.com/NixOS/nixpkgs/pull/247476
** TODO Happy :happy:
*** PROJ PR python 3 upstream
*** PROJ nixpkgs en l'état
** TODO Bamsurgeon
/Entered on/ [2023-05-13 Sat 19:11]
*** TODO Velvet
** TODO PR Picard avec option pour gérer la mémoire
Similaire à
https://github.com/bioconda/bioconda-recipes/blob/master/recipes/picard/picard.sh
* Julia :julia:
** KILL XAM.jl: PR pour modification record :julia:
CLOSED: [2023-05-29 Mon 15:40] SCHEDULED: <2023-05-28 Sun>
/Entered on/ [2023-05-27 Sat 22:39]
** TODO XAMscissors.jl :xamscissors:
Modification de la séquence dans BAM.
*Pas de mise à jour de CIGAR*
On convertit en fastq et on lance le pipeline pour "corriger"
#+begin_src sh
cd /home/alex/code/bisonex/out/63003856/preprocessing/mapped
samtools view 63003856_S135.bam NC_000022.11 -o 63003856_S135_chr22.bam
cd /home/alex/recherche/bisonex/code/BamScissors.jl
cp ~/code/bisonex/out/63003856/preprocessing/mapped/63003856_S135_chr22.bam .
samtools index 63003856_chr22.bam
#+end_src
Le script va modifier le bam, le trier et générer le fastq. !!!
Attention: ne pas oublier l'option -n !!!
#+begin_src sh
time julia --project=.. insertVariant.jl
scp 63003856_S135_chr22_{1,2}.fq.gz meso:/Work/Users/apraga/bisonex/tests/bamscissors/
#+end_src
*** WAIT Implémenter les SNV avec VAF :snv:
Stratégie :
1. calculer la profondeur sur les positions
2. créer un dictionnaire { nom du reads : position dataframe }
3. itérer sur tous les reads et changer ceux marqués
**** DONE VAF = 1
CLOSED: [2023-05-29 Mon 15:34]
**** DONE VAF selon loi normale
CLOSED: [2023-05-29 Mon 15:35]
Tronquée si > 1
**** WAIT Tests unitaires
***** DONE NA12878: 1 gène sur chromosome 22
CLOSED: [2023-05-30 Tue 23:55]
root = "https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/"
#+begin_src sh
samtools view project.NIST_NIST7035_H7AP8ADXX_NA12878.bwa.markDuplicates.bam  chr22 -o project.NIST_NIST7035_H7AP8ADXX_NA12878_chr22.bam
samtools view project.NIST_NIST7035_H7AP8ADXX_NA12878_chr22.bam chr22:19419700-19424000 -o NIST7035_H7AP8ADXX_NA12878_chr22_MRPL40_hg19.bam
#+end_src
***** WAIT Pull request formatspeciment
https://github.com/BioJulia/FormatSpecimens.jl/pull/8
***** DONE Formatspecimens
CLOSED: [2023-05-29 Mon 23:03]
****** DONE 1 read
CLOSED: [2023-05-29 Mon 23:02]
****** DONE VAF sur 1 exon
CLOSED: [2023-05-29 Mon 23:03]
**** TODO [#A] Bug: perte de nombreux reads avec NA12878
SCHEDULED: <2023-08-18 Fri>
Ex: chrX:g.124056226T>G : on passe de 65 reads à 1
*** TODO Implémenter les indel avec VAF :indel:
*** TODO Soumission paquet
* Données
:PROPERTIES:
:CATEGORY: data
:END:
** DONE Remplacer bam par fastq sur mesocentre
CLOSED: [2023-04-16 Sun 16:33]
Commande
*** DONE Supprimer les fastq non "paired"
CLOSED: [2023-04-16 Sun 16:33]
nushell
Liste des fastq avec "paired-end" manquant
#+begin_src nu
ls **/*.fastq.gz | get name | path basename | split column "_" | get column1 | uniq -u | save single.txt
#+end_src
#+RESULTS:
: 62907927
: 62907970
: 62899606
: 62911287
: 62913201
: 62914084
: 62915905
: 62921595
: 62923065
: 62925220
: 62926503
: 62926502
: 62926500
: 62926499
: 62926498
: 62931719
: 62943423
: 62943400
: 62948290
: 62949205
: 62949206
: 62949118
: 62951284
: 62960792
: 62960785
: 62960787
: 62960617
: 62962561
: 62962692
: 62967473
: 62972194
: 62979102
On vérifie
#+begin_src nu
open single.txt  | lines | each {|e| ls $"fastq/*_($in)/*" | get 0  }
open single.txt  | lines | each {|e| ls $"fastq/*_($in)/*" | get 0.name }  | path basename | split column "_" | get column1 | uniq -c
#+end_src
On met tous dans un dossier (pas de suppression )
#+begin_src
open single.txt  | lines | each {|e| ls $"fastq/*_($in)/*" | get 0  }  | each {|e| ^mv $e.name bad-fastq/}
#+end_src
On vérifie que les dossiier sont videsj
 open single.txt  | lines | each {|e| ls $"fastq/*_($in)" | get 0.name } | ^ls -l $in
 Puis on supprime
 open single.txt  | lines | each {|e| ls $"fastq/*_($in)" | get 0.name } | ^rm -r $in
*** DONE Supprimer bam qui ont des fastq
CLOSED: [2023-04-16 Sun 16:33]
On liste les identifiants des fastq et bam dans un tableau avec leur type :
#+begin_src
let fastq = (ls fastq/*/*.fastq.gz | get name | parse "{dir}/{full_id}/{id}_{R}_001.fastq.gz"  | select dir id | uniq )
let bam = (ls bam/*/*.bam | get name | parse "{dir}/{full_id}/{id}_{S}.bqrt.bam"  | select dir id)
#+end_src
On groupe les résultat par identifiant (résultats = liste de records qui doit être convertie en table)
et on trie ceux qui n'ont qu'un fastq ou un bam
#+begin_src
let single = ( $bam | append $fastq | group-by id | transpose id files | get files | where {|x| ($x | length) == 1})
#+end_src
On convertit en table et on récupère seulement les bam
#+begin_src
$single | reduce {|it, acc| $acc | append $it} | where dir == bam | get id | each {|e| ^ls $"bam/*_($e)/*.bam"}
#+end_src
#+RESULTS:
: bam/2100656174_62913201/62913201_S52.bqrt.bam
: bam/2100733271_62925220/62925220_S33.bqrt.bam
: bam/2100738763_62926502/62926502_S108.bqrt.bam
: bam/2100746726_62926498/62926498_S105.bqrt.bam
: bam/2100787936_62931955/62931955_S4.bqrt.bam
: bam/2200066374_62948290/62948290_S130.bqrt.bam
: bam/2200074722_62948298/62948298_S131.bqrt.bam
: bam/2200074990_62948306/62948306_S218.bqrt.bam
: bam/2200214581_62967331/62967331_S267.bqrt.bam
: bam/2200225399_62972187/62972187_S85.bqrt.bam
: bam/2200293962_62979117/62979117_S63.bqrt.bam
: bam/2200423985_62999352/62999352_S1.bqrt.bam
: bam/2200495073_63010427/63010427_S20.bqrt.bam
: bam/2200511274_63012586/63012586_S114.bqrt.bam
: bam/2200669188_63036688/63036688_S150.bqrt.bam
* Nouveau workflow :workflow:
** TODO Bases de données
*** KILL Nix pour télécharger les données brutes
**** Conclusion
Non viable sur cluster car en dehors de /nix/store
On peut utiliser des symlink mais trop compliqué
**** KILL Axel au lieu de curl pour gérer les timeout?
CLOSED: [2022-08-19 Fri 15:18]
*** DONE Tester patch de @pennae pour gros fichiers
SCHEDULED: <2022-08-19 Fri>
*** KILL Télécharger les données avec nextflow: hg38
CLOSED: [2023-06-12 Mon 23:29]
**** DONE Genome de référence
**** DONE dbSNP
**** DONE VEP 20G
CLOSED: [2023-06-12 Mon 23:29]
Ajout vérification checksum -> à vérifier
**** DONE VEP version 1.10
CLOSED: [2023-08-06 Sun 09:45] SCHEDULED: <2023-08-06 Sun>
**** DONE transcriptome (spip)
CLOSED: [2023-06-12 Mon 23:29]
Rajouter checksum manuel
**** KILL Refseq
**** KILL OMIM
CLOSED: [2023-06-12 Mon 23:29]
codé, à vérifier
**** KILL ACMG incidental
CLOSED: [2023-06-12 Mon 23:29]
*** TODO Données :T2T:
:PROPERTIES:
:ID:       5d915178-ca96-44ef-87f1-6702af114f2b
:END:
**** DONE fasta
CLOSED: [2023-06-12 Mon 23:30]
***** DONE compatibilité hg38
CLOSED: [2023-06-12 Mon 23:30]
**** DONE fasta index
CLOSED: [2023-06-13 Tue 00:07]
***** DONE compatibilité hg38
CLOSED: [2023-06-13 Tue 00:07]
**** DONE fasta dictionnaire
CLOSED: [2023-06-13 Tue 00:07]
**** DONE dbSNP
CLOSED: [2023-06-12 Mon 23:30]
***** DONE compatibilité hg38
CLOSED: [2023-06-12 Mon 23:30]
**** DONE commonSNP
CLOSED: [2023-06-1

Replacement in projects/bisonex.org at line 56 [4.35]

B:BD[11.91465] → [11.91465:92858]

B:BD[11.92858] → [8.41827:42283]

B:BD[8.42283] → [6.8403:14746]

FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |       0.286285 |        0.519629 |                    NaN |                    NaN |                  2.428571 |                  2.465116 |
| INDEL | PASS   |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |       0.286285 |        0.519629 |                    NaN |                    NaN |                  2.428571 |                  2.465116 |
| SNP   | ALL    |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |       0.120397 |        0.686734 |                3.11461 |                2.85705 |                  3.640644 |                  2.114633 |
| SNP   | PASS   |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |       0.120397 |        0.686734 |                3.11461 |                2.85705 |                  3.640644 |                  2.114633 |
Hg38
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY
.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |       0.378198 |        0.888102 |                    NaN |                    NaN |    
              1.860963 |                  2.247273 |
| INDEL | PASS   |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |       0.378198 |        0.888102 |                    NaN |                    NaN |                  1.860963 |                  2.247273 |
| SNP   | ALL    |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |       0.162184 |        0.975588 |                3.00711 |               2.784686 |                  1.591810 |                  1.816145 |
| SNP   | PASS   |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |       0.162184 |        0.975588 |                3.00711 |               2.784686 |                  1.591810 |                  1.816145 |
******** Résumé
T2T
| Type  | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision |
| INDEL |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |
| SNP   |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |
Hg38
| Type  | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision |
| INDEL |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |
| SNP   |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |
****** DONE Interesection des bed: similaire
CLOSED: [2023-07-04 Tue 23:11]
HG38
 #+begin_src sh
 bedtools intersect -a capture/Agilent_SureSelect_All_Exons_v7_hg38_Regions.bed -b /Work/Groups/bisonex/data/giab/GRCh38/HG001_GRCh38_1_22_v4.2.1_benchmark.bed  | wc -l
 #+end_src
 204280
 T2T
 #+begin_src sh
 bedtools intersect -a /Work/Groups/bisonex/data/giab/T2T/Agilent_SureSelect_All_Exons_v7_hg38_Regions_hg38_T2T.bed -b /Work/Groups/bisonex/data/giab/T2T/HG001_GRCh38_1_22_v4.2.1_benchmark_hg38_T2T.bed  | wc -l
 #+end_src
 204021
****** DONE Vérifier la ligne de commande
CLOSED: [2023-07-04 Tue 23:38]
#+begin_src sh
hap.py \
    HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz \
    HG001-SRX11061486_SRR14724513-T2T.vcf.gz \
     \
    --reference chm13v2.0.fa \
    --threads 6 \
     \
    -T Agilent_SureSelect_All_Exons_v7_hg38_Regions_hg38_T2T.bed \
    --false-positives HG001_GRCh38_1_22_v4.2.1_benchmark_hg38_T2T.bed \
     \
    -o HG001
#+end_src
****** DONE Corriger FILTER : mieux mais toujours trop de négatifs. 3/4 SNP retrouvés
CLOSED: [2023-07-08 Sat 15:19] SCHEDULED: <2023-07-08 Sat>
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.Precision  METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
INDEL    ALL          413       246       167          751       289        215      2     98       0.595642          0.460821        0.286285         0.519629                     NaN                     NaN                   2.428571                   2.465116
INDEL   PASS          413       246       167          751       289        215      2     98       0.595642          0.460821        0.286285         0.519629                     NaN                     NaN                   2.428571                   2.465116
  SNP    ALL        15883     15479       404        23597      5277       2841     46     44       0.974564          0.745760        0.120397         0.844947                3.017198                 2.85705                   5.560099                   2.114633
  SNP   PASS        15883     15479       404        23597      5277       2841     46     44       0.974564          0.745760        0.120397         0.844947                3.017198                 2.85705                   5.560099                   2.114633
******* DONE Vérifier qu'il ne reste plus de filtre autre que PASS
CLOSED: [2023-07-08 Sat 15:19]
#+begin_src
$ zgrep -c 'PASS' HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz
3730505
$ zgrep -c '^chr' HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz
3730506
#+end_src
****** TODO 1/4 SNP manquant ?
******* DONE Regarder avec Julia si ce sont vraiment des FP: 61/5277 qui ne le sont pas
CLOSED: [2023-07-09 Sun 12:09]
******* DONE Examiner les FP
CLOSED: [2023-07-30 Sun 22:05]
******* DONE Tester un FP
CLOSED: [2023-07-30 Sun 22:05]
  2 │ chr1        608765  A           G           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:ti:SNP:homalt:188
  liftDown UCSC: rien en GIAB : vrai FP
 3 │ chr1        762943  A           G           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:ti:SNP:homalt:287
 4 │ chr1        762945  A           T           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:tv:SNP:homalt:287
 Remaniements complexes ? Pas dans le gène en HG38
******* DONE La plupart des FP (4705/5566) sont homozygotes: erreur de référence ?
CLOSED: [2023-07-12 Wed 21:10] SCHEDULED: <2023-07-09 Sun>
Sur les 2 premiers variants, ils montrent en fait la différence entre T2T et GRCh38
Erreur à l'alignement ?
******** KILL relancer l'alignement
CLOSED: [2023-07-09 Sun 17:36]
******** DONE vérifier reads identiques hg38 et T2T: oui
CLOSED: [2023-07-09 Sun 16:36]
T2T CHR1608765
38   	chr1:1180168-1180168 (
SRR14724513.24448214
SRR14724513.24448214
******* DONE Vérifier quelques variants sur IGV
CLOSED: [2023-07-09 Sun 17:36]
******* KILL Répartition des FP : cluster ?
CLOSED: [2023-07-09 Sun 17:36]
****** DONE Examiner les FP restant après correction selon séquence de référence
CLOSED: [2023-08-12 Sat 15:57]
****** HOLD Examiner les variants supprimé
****** TODO Enlever les FP qui correspondent à un changement dans le génome
SCHEDULED: <2023-08-17 Thu>
******* Condition:
- pas de variation à la position en GRCh38
- variantion homozygote
- la varation en T2T correspond au changement de pair de base GRC38 -> T2T
  pour les SNP:
  alt_T2T[i] = DNA_GRC38[j]
  avec i la position en T2T e

[11.91465]

[6.14746]

FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |       0.286285 |        0.519629 |                    NaN |                    NaN |                  2.428571 |                  2.465116 |
| INDEL | PASS   |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |       0.286285 |        0.519629 |                    NaN |                    NaN |                  2.428571 |                  2.465116 |
| SNP   | ALL    |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |       0.120397 |        0.686734 |                3.11461 |                2.85705 |                  3.640644 |                  2.114633 |
| SNP   | PASS   |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |       0.120397 |        0.686734 |                3.11461 |                2.85705 |                  3.640644 |                  2.114633 |
Hg38
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |       0.378198 |        0.888102 |                    NaN |                    NaN |                  1.860963 |                  2.247273 |
| INDEL | PASS   |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |       0.378198 |        0.888102 |                    NaN |                    NaN |                  1.860963 |                  2.247273 |
| SNP   | ALL    |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |       0.162184 |        0.975588 |                3.00711 |               2.784686 |                  1.591810 |                  1.816145 |
| SNP   | PASS   |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |       0.162184 |        0.975588 |                3.00711 |               2.784686 |                  1.591810 |                  1.816145 |
******** Résumé
T2T
| Type  | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision |
| INDEL |         413 |      246 |      167 |         751 |      289 |       215 |     2 |    93 |      0.595642 |         0.460821 |
| SNP   |       11236 |    10985 |      251 |       23597 |     9771 |      2841 |    26 |    58 |      0.977661 |         0.529245 |
Hg38
| Type  | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision |
| INDEL |         549 |      489 |       60 |         899 |       64 |       340 |     8 |    17 |      0.890710 |         0.885510 |
| SNP   |       21973 |    21462 |      511 |       26285 |      563 |      4263 |    68 |    16 |      0.976744 |         0.974435 |
****** DONE Interesection des bed: similaire
CLOSED: [2023-07-04 Tue 23:11]
HG38
 #+begin_src sh
 bedtools intersect -a capture/Agilent_SureSelect_All_Exons_v7_hg38_Regions.bed -b /Work/Groups/bisonex/data/giab/GRCh38/HG001_GRCh38_1_22_v4.2.1_benchmark.bed  | wc -l
 #+end_src
 204280
 T2T
 #+begin_src sh
 bedtools intersect -a /Work/Groups/bisonex/data/giab/T2T/Agilent_SureSelect_All_Exons_v7_hg38_Regions_hg38_T2T.bed -b /Work/Groups/bisonex/data/giab/T2T/HG001_GRCh38_1_22_v4.2.1_benchmark_hg38_T2T.bed  | wc -l
 #+end_src
 204021
****** DONE Vérifier la ligne de commande
CLOSED: [2023-07-04 Tue 23:38]
#+begin_src sh
hap.py \
    HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz \
    HG001-SRX11061486_SRR14724513-T2T.vcf.gz \
     \
    --reference chm13v2.0.fa \
    --threads 6 \
     \
    -T Agilent_SureSelect_All_Exons_v7_hg38_Regions_hg38_T2T.bed \
    --false-positives HG001_GRCh38_1_22_v4.2.1_benchmark_hg38_T2T.bed \
     \
    -o HG001
#+end_src
****** DONE Corriger FILTER : mieux mais toujours trop de négatifs. 3/4 SNP retrouvés
CLOSED: [2023-07-08 Sat 15:19] SCHEDULED: <2023-07-08 Sat>
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.Precision  METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
INDEL    ALL          413       246       167          751       289        215      2     98       0.595642          0.460821        0.286285         0.519629                     NaN                     NaN                   2.428571                   2.465116
INDEL   PASS          413       246       167          751       289        215      2     98       0.595642          0.460821        0.286285         0.519629                     NaN                     NaN                   2.428571                   2.465116
  SNP    ALL        15883     15479       404        23597      5277       2841     46     44       0.974564          0.745760        0.120397         0.844947                3.017198                 2.85705                   5.560099                   2.114633
  SNP   PASS        15883     15479       404        23597      5277       2841     46     44       0.974564          0.745760        0.120397         0.844947                3.017198                 2.85705                   5.560099                   2.114633
******* DONE Vérifier qu'il ne reste plus de filtre autre que PASS
CLOSED: [2023-07-08 Sat 15:19]
#+begin_src
$ zgrep -c 'PASS' HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz
3730505
$ zgrep -c '^chr' HG001_GRCh38_1_22_v4_lifted_merged.vcf.gz
3730506
#+end_src
****** TODO 1/4 SNP manquant ?
******* DONE Regarder avec Julia si ce sont vraiment des FP: 61/5277 qui ne le sont pas
CLOSED: [2023-07-09 Sun 12:09]
******* DONE Examiner les FP
CLOSED: [2023-07-30 Sun 22:05]
******* DONE Tester un FP
CLOSED: [2023-07-30 Sun 22:05]
  2 │ chr1        608765  A           G           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:ti:SNP:homalt:188
  liftDown UCSC: rien en GIAB : vrai FP
 3 │ chr1        762943  A           G           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:ti:SNP:homalt:287
 4 │ chr1        762945  A           T           ./.:.:.:.:NOCALL:nocall:.  1/1:FP:.:tv:SNP:homalt:287
 Remaniements complexes ? Pas dans le gène en HG38
******* DONE La plupart des FP (4705/5566) sont homozygotes: erreur de référence ?
CLOSED: [2023-07-12 Wed 21:10] SCHEDULED: <2023-07-09 Sun>
Sur les 2 premiers variants, ils montrent en fait la différence entre T2T et GRCh38
Erreur à l'alignement ?
******** KILL relancer l'alignement
CLOSED: [2023-07-09 Sun 17:36]
******** DONE vérifier reads identiques hg38 et T2T: oui
CLOSED: [2023-07-09 Sun 16:36]
T2T CHR1608765
38   	chr1:1180168-1180168 (
SRR14724513.24448214
SRR14724513.24448214
******* DONE Vérifier quelques variants sur IGV
CLOSED: [2023-07-09 Sun 17:36]
******* KILL Répartition des FP : cluster ?
CLOSED: [2023-07-09 Sun 17:36]
****** DONE Examiner les FP restant après correction selon séquence de référence
CLOSED: [2023-08-12 Sat 15:57]
****** HOLD Examiner les variants supprimé
****** TODO Enlever les FP qui correspondent à un changement dans le génome
******* Condition:
SCHEDULED: <2023-08-19 Sat>
- pas de variation à la position en GRCh38
- variantion homozygote
- la varation en T2T correspond au changement de pair de base GRC38 -> T2T
  pour les SNP:
  alt_T2T[i] = DNA_GRC38[j]
  avec i la position en T2T e

Replacement in projects/bisonex.org at line 71 [4.35]

B:BD[3.32425] → [3.32425:33017]

B:BD[3.33017] → [2.8814:11924]

gin_src sh :dir /home/alex/roam/research/bisonex/code/sanger
julia --project=.. xamscissors.jl
cd out-all
mv inserted_1.fq.gz NA12878-sanger-inserted-all_1.fq.gz
mv inserted_2.fq.gz NA12878-sanger-inserted-all_2.fq.gz
rsync -avz NA12878-sanger-inserted_* meso:/Work/Users/apraga/bisonex/data/
#+end_src
Fichier de configuration
patient,sample,fastq1,fastq2
NA12878,sanger-all,data/NA12878-sanger-inserted-all_1.fq.gz,data/NA12878-sanger-inserted-all_2.fq.gz
On lance la simulation
#+begin_src
nextflow run main.nf -profile standard,helios --input=samples-synthetic.csv --g
enome=GRCh38  -bg
#+end_src
***** DONE Résultat après haplotyecaller: 3 varinat perdus => ok
CLOSED: [2023-08-17 Thu 19:13]
Haplotypecaller 143 found over 146
3×3 DataFrame
| variant             | meanQual | depth |
|---------------------+----------+-------|
| chr12:g.13720138C>T |     60.0 |     1 |
| chr17:g.10296150T>A |     60.0 |     3 |
| chr21:g.43426167C>T |      0.0 |    88 |
Pas assez de read (1,2) et problème d'alignement (3)
***** TODO Résultat après filtre depth : +10 variants perduis
SCHEDULED: <2023-08-18 Fri>
filter depth : another 10 missed variants
10×3 DataFrame
| variant             | meanQual | depth |
|---------------------+----------+-------|
| chr3:g.71112628C>T  |     60.0 |    62 |
| chr12:g.40367710A>G |  58.0435 |    46 |
| chr14:g.58458545G>A |     60.0 |     9 |
| chr15:g.66703292C>T |     60.0 |    33 |
| chr16:g.30965737C>A |     60.0 |    18 |
| chr17:g.61968202A>C |     60.0 |    46 |
| chrX:g.124056226T>G |     60.0 |    40 |
| chrX:g.24737739G>T  |     60.0 |    16 |
| chrX:g.40591349C>T  |     60.0 |    37 |
| chrX:g.53193275G>A  |     60.0 |    32 |
|                     |          |       |
S'ils sont hétérozygotes, 0.5*depth est effectivement < 30 (notre filtre...)
****** TODO Discordance DP et vraie profondeur ?
SCHEDULED: <2023-08-18 Fri>
***** DONE Résultat après filtre common variant: +0 ok
CLOSED: [2023-08-17 Thu 19:32]
***** TODO Résultat après filtre VEP : +23 perdus ??
SCHEDULED: <2023-08-18 Fri>
filter vep : another 23 missed variants
23×3 DataFrame
 Row │ variant               meanQual  depth
     │ String                Float64   Int64
─────┼───────────────────────────────────────
   1 │ chr1:g.183222115C>T    60.0       168
   2 │ chr1:g.39388062C>T     60.0       285
   3 │ chr2:g.240719197G>C    60.0        77
   4 │ chr3:g.41227353G>C     60.0       105
   5 │ chr4:g.15536991T>G     60.0        41
   6 │ chr5:g.14474096G>A     60.0       191
   7 │ chr8:g.43122149C>T     60.0       237
   8 │ chr9:g.128603589A>C    60.0       304
   9 │ chr9:g.137452819G>C    60.0       107
  10 │ chr10:g.129957338T>C   60.0       116
  11 │ chr10:g.247389T>G      60.0        56
  12 │ chr11:g.61313668G>A    60.0        83
  13 │ chr12:g.45850467C>T    60.0       291
  14 │ chr14:g.64216315C>G    60.0       263
  15 │ chr15:g.60514655G>A    60.0       259
  16 │ chr17:g.61966475G>T    60.0       144
  17 │ chr17:g.7852503T>C     60.0       190
  18 │ chr19:g.13230158G>A    60.0       172
  19 │ chr19:g.38523211C>G    60.0        93
  20 │ chr19:g.4110557G>C     59.9929    425
  21 │ chr20:g.62334188G>A    60.0        62
  22 │ chrX:g.47575255G>A     60.0       244
  23 │ chrX:g.53409112G>A     60.0       136
**** TODO Données simuscop 200x
SCHEDULED: <2023-08-22 Tue>
* Résultats
** TODO Speed-up BWA-mem
SCHEDULED: <2023-08-26 Sat>
** TODO Speed-up Hapotypecaller
SCHEDULED: <2023-08-26 Sat>

[3.32425]

gin_src sh :dir /home/alex/roam/research/bisonex/code/sanger
julia --project=.. xamscissors.jl
cd out-all
mv inserted_1.fq.gz NA12878-sanger-inserted-all_1.fq.gz
mv inserted_2.fq.gz NA12878-sanger-inserted-all_2.fq.gz
rsync -avz NA12878-sanger-inserted_* meso:/Work/Users/apraga/bisonex/data/
#+end_src
Fichier de configuration
patient,sample,fastq1,fastq2
NA12878,sanger-all,data/NA12878-sanger-inserted-all_1.fq.gz,data/NA12878-sanger-inserted-all_2.fq.gz
On lance la simulation
#+begin_src
nextflow run main.nf -profile standard,helios --input=samples-synthetic.csv --genome=GRCh38  -bg
#+end_src
***** DONE Résultat après haplotyecaller: 3 varinat perdus => ok
CLOSED: [2023-08-17 Thu 19:13]
Haplotypecaller 143 found over 146
3×3 DataFrame
| variant             | meanQual | depth |
|---------------------+----------+-------|
| chr12:g.13720138C>T |     60.0 |     1 |
| chr17:g.10296150T>A |     60.0 |     3 |
| chr21:g.43426167C>T |      0.0 |    88 |
Pas assez de read (1,2) et problème d'alignement (3)
***** TODO Résultat après filtre depth : +10 variants perduis
SCHEDULED: <2023-08-18 Fri>
filter depth : another 10 missed variants
10×3 DataFrame
| variant             | meanQual | depth |
|---------------------+----------+-------|
| chr3:g.71112628C>T  |     60.0 |    62 |
| chr12:g.40367710A>G |  58.0435 |    46 |
| chr14:g.58458545G>A |     60.0 |     9 |
| chr15:g.66703292C>T |     60.0 |    33 |
| chr16:g.30965737C>A |     60.0 |    18 |
| chr17:g.61968202A>C |     60.0 |    46 |
| chrX:g.124056226T>G |     60.0 |    40 |
| chrX:g.24737739G>T  |     60.0 |    16 |
| chrX:g.40591349C>T  |     60.0 |    37 |
| chrX:g.53193275G>A  |     60.0 |    32 |
|                     |          |       |
S'ils sont hétérozygotes, 0.5*depth est effectivement < 30 (notre filtre...)
****** TODO Problème d'inserstion des reads: on en perd de nombreux !
SCHEDULED: <2023-08-18 Fri>
Ex: chrX:g.124056226T>G : on passe de 65 reads à 1
***** DONE Résultat après filtre common variant: +0 ok
CLOSED: [2023-08-17 Thu 19:32]
***** TODO Résultat après filtre VEP : +23 perdus ??
SCHEDULED: <2023-08-18 Fri>
filter vep : another 23 missed variants
23×3 DataFrame
 Row │ variant               meanQual  depth
     │ String                Float64   Int64
─────┼───────────────────────────────────────
   1 │ chr1:g.183222115C>T    60.0       168
   2 │ chr1:g.39388062C>T     60.0       285
   3 │ chr2:g.240719197G>C    60.0        77
   4 │ chr3:g.41227353G>C     60.0       105
   5 │ chr4:g.15536991T>G     60.0        41
   6 │ chr5:g.14474096G>A     60.0       191
   7 │ chr8:g.43122149C>T     60.0       237
   8 │ chr9:g.128603589A>C    60.0       304
   9 │ chr9:g.137452819G>C    60.0       107
  10 │ chr10:g.129957338T>C   60.0       116
  11 │ chr10:g.247389T>G      60.0        56
  12 │ chr11:g.61313668G>A    60.0        83
  13 │ chr12:g.45850467C>T    60.0       291
  14 │ chr14:g.64216315C>G    60.0       263
  15 │ chr15:g.60514655G>A    60.0       259
  16 │ chr17:g.61966475G>T    60.0       144
  17 │ chr17:g.7852503T>C     60.0       190
  18 │ chr19:g.13230158G>A    60.0       172
  19 │ chr19:g.38523211C>G    60.0        93
  20 │ chr19:g.4110557G>C     59.9929    425
  21 │ chr20:g.62334188G>A    60.0        62
  22 │ chrX:g.47575255G>A     60.0       244
  23 │ chrX:g.53409112G>A     60.0       136
**** TODO Données simuscop 200x
SCHEDULED: <2023-08-22 Tue>
* Résultats
** TODO Speed-up BWA-mem
SCHEDULED: <2023-08-26 Sat>
** TODO Speed-up Hapotypecaller
SCHEDULED: <2023-08-26 Sat>