apraga/org - Change TUIGPRCLOWRHD2UQD3ZSWHWKKIJTRXB7LP3DDMTGEL4OFORXY24AC

Simpler nix configuration for hakyll

Created by Alexis Praga on April 8, 2023

TUIGPRCLOWRHD2UQD3ZSWHWKKIJTRXB7LP3DDMTGEL4OFORXY24AC

Dependencies

In channels

main

Change contents

File deletion: templates
BF:BFD[5.2] → [6.2:22]
BF:BFD[6.22] → [6.1:1]
File deletion: posts
BF:BFD[5.2] → [7.150:166]
BF:BFD[7.166] → [7.149:149]
File deletion: files
BF:BFD[5.2] → [2.298:314]
BF:BFD[2.314] → [2.297:297]
File deletion: mandelbrot
BF:BFD[2.297] → [2.23487272:23487293]
BF:BFD[2.23487293] → [2.23487271:23487271]
File deletion: maps
BF:BFD[2.297] → [2.14387368:14387383]
BF:BFD[2.14387383] → [2.14387367:14387367]

File deletion: haskell-overlay.nix

BF:BFD[5.2] → [6.4851:4894]

BF:BFD[6.4894] → [6.2435:2435]

B:BD[6.2435] → [6.2436:2568]

B:BD[6.2586] → [6.2586:2649]

B:BD[6.2683] → [6.2683:3049]

B:BD[6.3290] → [6.3290:3390]

B:BD[6.3429] → [6.3429:3607]

B:BD[6.3641] → [6.3641:4727]

∅:D[8.314] → [6.4801:4850]

B:BD[6.4801] → [6.4801:4850]

B:BD[6.4727] → [8.237:314]

final: prev:
  let
    inherit (prev.stdenv) mkDerivation;
    inherit (prev.lib.trivial) flip pipe;
    inherit (prev.haskell.lib)
      appendConfigureFlags
      dontCheck
      doJailbreak;
    withFlags = flip appendConfigureFlags;
    haskellCompiler = "ghc902";
  in {
    myHaskellPackages = prev.haskell.packages.${haskellCompiler}.override {
      overrides = hpFinal: hpPrev:
        let
          hakyll-src = hpPrev.callHackage "hakyll" "4.15.1.1" {};
          pandoc-src = hpPrev.callHackage "pandoc" "2.17.1.1" {}; # latest version on nixpkgs
        in rec {
          hakyll = pipe hakyll-src [
            doJailbreak
            dontCheck
            (withFlags [ "-f" "watchServer" "-f" "previewServer" ])
          ];
          pandoc = pipe pandoc-src [
            doJailbreak
            dontCheck
          ];
          blog = hpPrev.callCabal2nix "blog-src" ./blog {};
          blog-nix = prev.stdenv.mkDerivation {
            #__contentAddressed = true; # uncomment if using cas: https://www.tweag.io/blog/2020-09-10-nix-cas/
            name = "blog-nix";
            buildInputs = [ blog ];
            src = prev.nix-gitignore.gitignoreSourcePure [
              ./.gitignore
              ".git"
              ".github"
            ] ./.;
            # LANG and LOCALE_ARCHIVE are fixes pulled from the community:
            #   https://github.com/jaspervdj/hakyll/issues/614#issuecomment-411520691
            #   https://github.com/NixOS/nix/issues/318#issuecomment-52986702
            #   https://github.com/MaxDaten/brutal-recipes/blob/source/default.nix#L24
            LANG = "en_US.UTF-8";
            LOCALE_ARCHIVE = prev.lib.optionalString
              (prev.buildPlatform.libc == "glibc")
              "${prev.glibcLocales}/lib/locale/locale-archive";
            buildPhase = ''
              hakyll-site build --verbose
            '';
            installPhase = ''
            '';
          };
        };
    };
}
              mkdir -p "$out/_site"
              cp -r _site/* "$out/_site"

File deletion: css
BF:BFD[5.2] → [6.7510:7524]
BF:BFD[6.7524] → [6.7509:7509]

Replacement in projects/bisonex.org at line 30 [10.35]

B:BD[9.237371] → [9.237371:237584]

B:BD[9.237584] → [11.31094:32397]

B:BD[11.32397] → [12.17918:24594]

|
| --help false                                                                                                  | --help false                                                                            |
| --v
ersion false                                                                                               | --version false                                                                         |
| --showHidden false                                                                                            | --showHidden false                                                                      |
| --QUIET false                                                                                                 | --QUIET false                                                                           |
| --use-jdk-deflater false                                                                                      | --use-jdk-deflater false                                                                |
| --use-jdk-inflater false                                                                                      | --use-jdk-inflater false                                                                |
| --gcs-max-retries 20                                                                                          | --gcs-max-retries 20                                                                    |
| --gcs-project-for-requester-pays                                            
                                  | --gcs-project-for-requester-pays                                                        |
| --disable-tool-default-read-filters false	PN:GATK ApplyBQSR                                                 | --disable-tool-default-read-filters false     PN:GATK ApplyBQSR                         |
****** KILL Vérifier sha256sum
CLOSED: [2023-01-24 Tue 23:00]
alignment: différent
****** KILL Comparer bam
CLOSED: [2023-01-25 Wed 21:58]
/Work/Users/apraga/bisonex/script/files〉picard CompareSAMs LENIENT_LOW_MQ_ALIGNMENT=true LENIENT_DUP=true tmp_63003856_S135/63003856_S135.bam /Work/Groups/bisonex/ref/tmp_63003856_S135/63003856_S135.bam O=compare-bam.tsv
picard CompareSAMs -LENIENT_LOW_MQ_ALIGNMENT true -LENIENT_DUP true tmp_63003856_S135/63003856_S135.bam /Work/Groups/bisonex/ref/tmp_63003856_S135/63003856_S135.bam -O compare-bam.tsv
VN Program Record attribute differs.
File 1: 1.13
File 2: 1.10
SAM files differ.
[Tue Jan 24 23:12:50 CET 2023] picard.sam.CompareSAMs done. Elapsed time: 7.32 minutes.
***** DONE Relancer avec la même version de samtools
CLOSED: [2023-01-25 Wed 21:58]
Pas d'impact
***** TODO Comparer tsv de sortie
***** TODO Regarder où sont les variants différents
** TODO GIAB Validation : GIAB
https://github.com/ga4gh/benchmarking-tools
Prérequis :
- [[*hap.py][hap.py]]
- [[*NA12878][NA12878]]
*** TODO GIAB : exome :giab:
**** Notes
https://github.com/genome-in-a-bottle/giab_FAQ
**** TODO télécharger données avec Nextflow
***** DONE Renommer les chromosomes
CLOSED: [2023-02-17 Fri 19:30]
****** DONE Genome de reference NCBI
CLOSED: [2023-02-25 Sat 19:46]
****** DONE Bed avec les exons
CLOSED: [2023-03-29 Wed 23:04]
****** DONE hg19
CLOSED: [2023-02-26 Sun 22:37]
****** DONE hg38
CLOSED: [2023-03-29 Wed 23:04]
- [X] Télécharger hg19 : ok
- [X] convertir bed en interval list
picard BedToIntervalList -I exons_illumina.bed  -O exons_illumina.list -SD  ../../genome/GRCh19/genomeRef.dict
- [X] puis en hg38
picard LiftOverIntervalList -I exons_illumina.list  -O exons_illumina_hg38.list --CHAIN hg19ToHg38.over.chain -SD  ../../genome/GRCh38.p13/genomeRef.dict
- [X] puis en bed
***** TODO VCF de référence
****** DONE NA12878 (HG001)
CLOSED: [2023-02-25 Sat 19:46]
******* DONE Fastq HiSeq
CLOSED: [2023-02-25 Sat 19:46]
On prend le Hiseq, qui est probablement ce qu'utilise Centogène :
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/
On utilisé les données "trimmés" (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1069-7), i.e qui ont enlevé les fragments plus petits que la taille d'un read.
Informations:
- https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/Garvan_NA12878_HG001_HiSeq_Exome.README
- Sequencer: HiSeq2500
- kit: Nextera Rapid Capture Exome and Expanded Exome
Il y a 2 samples (NIST7035 et NIST7086), chacun sur 2 lanes -> à concaténer
NB : liste techno illumina https://www.illumina.com/systems/sequencing-platforms.html
Hiseq postérieur nextseq 550
******* DONE Capture : Exons (bed)
CLOSED: [2023-02-25 Sat 19:46]
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/nexterarapidcapture_expandedexome_targetedregions.bed.gz
******* DONE Bed, vcf
CLOSED: [2023-02-24 Fri 23:45]
****** TODO Ashkenazy trio HG002, HG003, HGQ004
SCHEDULED: <2023-04-01 Sat>
****** TODO Chinese trio HG005, 6, 7
***** TODO Fastq
****** DONE NA12878 (HG001)
CLOSED: [2023-02-25 Sat 19:46]
******* DONE Fastq HiSeq
CLOSED: [2023-02-25 Sat 19:46]
On prend le Hiseq, qui est probablement ce qu'utilise Centogène :
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/
On utilisé les données "trimmés" (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1069-7), i.e qui ont enlevé les fragments plus petits que la taille d'un read.
Informations:
- https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/Garvan_NA12878_HG001_HiSeq_Exome.README
- Sequencer: HiSeq2500
- kit: Nextera Rapid Capture Exome and Expanded Exome
Il y a 2 samples (NIST7035 et NIST7086), chacun sur 2 lanes -> à concaténer
NB : liste techno illumina https://www.illumina.com/systems/sequencing-platforms.html
Hiseq postérieur nextseq 550
******* DONE Capture : Exons (bed)
CLOSED: [2023-02-25 Sat 19:46]
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/nexterarapidcapture_expandedexome_targetedregions.bed.gz
******* DONE Bed, vcf
CLOSED: [2023-02-24 Fri 23:45]
****** TODO Ashkenazy trio HG002, HG003, HG004
SCHEDULED: <2023-04-01 Sat>
******* TODO Capture
https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/OsloUniversityHospital_Exome_GATK_jointVC_11242015/wex_Agilent_SureSelect_v05_b37.baits.slop50.merged.list
******* TODO Bam à partir des fastq
Bam + index + checksum
https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/AshkenazimTrio/alignment.index.AJtrio_OsloUniversityHospital_IlluminaExome_bwamem_GRCh37_11252015
*
****** TODO Chinese trio
Whole exome pour HG005 seulement
******* TODO HG005
https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/ChineseTrio/alignment.index.Chinesetrio_HG005_OsloUniversityHospital_IlluminaExome_bwamem_GRCh37_11252015
**** TODO NA12878 / HG001 :na12878:
***** DONE Discussion alexis : Mail
CLOSED: [2023-03-29 Wed 22:40]
Avec le patient NA12878 et comparaison avec hap.py du VCF de Genome In A Bottle ("gold" standard), on avait pour rappel
- sensibilité (=recall) 71% pour indel, 85% SNP
- précision  (= VPP) 69 et 97% respectivement
| Type  | TRUTH |    TP |   FN | QUERY |   FP |  UNK | FP.gt | FP.al |   Recall | Precision |
| INDEL |  4871 |  3461 | 1410 |  7048 | 1554 | 1987 |   193 |   346 | 0.710532 |  0.692946 |
| SNP   | 46032 | 39369 | 6663 | 44600 | 1186 | 4041 |   304 |    30 | 0.855253 |  0.970759 |
Les statistiques sur les génomes sont bien meilleurs (cf precisionFDA challenge).
Pour les exome, un article [1] a fait a des meilleures stats sur ce patient avec BWA et GATK mais ils ont moins de variant (on a presque un facteur 2 !).
Je soupçonne qu'on ne travaille pas sur les mêmes zones de capture (pas réussi à récupérer leur .bed)
| Exome | Type  |    TP |   FP |  FN | Sensitivity | Precision | F-Score |   FDR |
|     1 | SNV   | 23689 | 1397 | 613 |       0.975 |     0.944 |

[9.237371]

[12.24594]

|
| --help false                                                                                                  | --help false                                                                            |
| --version false                                                                                               | --version false                                                                         |
| --showHidden false                                                                                            | --showHidden false                                                                      |
| --QUIET false                                                                                                 | --QUIET false                                                                           |
| --use-jdk-deflater false                                                                                      | --use-jdk-deflater false                                                                |
| --use-jdk-inflater false                                                                                      | --use-jdk-inflater false                                                                |
| --gcs-max-retries 20                                                                                          | --gcs-max-retries 20                                                                    |
| --gcs-project-for-requester-pays                                                                              | --gcs-project-for-requester-pays                                                        |
| --disable-tool-default-read-filters false	PN:GATK ApplyBQSR                                                 | --disable-tool-default-read-filters false     PN:GATK ApplyBQSR                         |
****** KILL Vérifier sha256sum
CLOSED: [2023-01-24 Tue 23:00]
alignment: différent
****** KILL Comparer bam
CLOSED: [2023-01-25 Wed 21:58]
/Work/Users/apraga/bisonex/script/files〉picard CompareSAMs LENIENT_LOW_MQ_ALIGNMENT=true LENIENT_DUP=true tmp_63003856_S135/63003856_S135.bam /Work/Groups/bisonex/ref/tmp_63003856_S135/63003856_S135.bam O=compare-bam.tsv
picard CompareSAMs -LENIENT_LOW_MQ_ALIGNMENT true -LENIENT_DUP true tmp_63003856_S135/63003856_S135.bam /Work/Groups/bisonex/ref/tmp_63003856_S135/63003856_S135.bam -O compare-bam.tsv
VN Program Record attribute differs.
File 1: 1.13
File 2: 1.10
SAM files differ.
[Tue Jan 24 23:12:50 CET 2023] picard.sam.CompareSAMs done. Elapsed time: 7.32 minutes.
***** DONE Relancer avec la même version de samtools
CLOSED: [2023-01-25 Wed 21:58]
Pas d'impact
***** TODO Comparer tsv de sortie
***** TODO Regarder où sont les variants différents
** TODO GIAB Validation : GIAB
https://github.com/ga4gh/benchmarking-tools
Prérequis :
- [[*hap.py][hap.py]]
- [[*NA12878][NA12878]]
*** TODO GIAB : exome :giab:
**** Notes
https://github.com/genome-in-a-bottle/giab_FAQ
**** TODO télécharger données avec Nextflow
***** DONE Renommer les chromosomes
CLOSED: [2023-02-17 Fri 19:30]
****** DONE Genome de reference NCBI
CLOSED: [2023-02-25 Sat 19:46]
****** DONE Bed avec les exons
CLOSED: [2023-03-29 Wed 23:04]
****** DONE hg19
CLOSED: [2023-02-26 Sun 22:37]
****** DONE hg38
CLOSED: [2023-03-29 Wed 23:04]
- [X] Télécharger hg19 : ok
- [X] convertir bed en interval list
picard BedToIntervalList -I exons_illumina.bed  -O exons_illumina.list -SD  ../../genome/GRCh19/genomeRef.dict
- [X] puis en hg38
picard LiftOverIntervalList -I exons_illumina.list  -O exons_illumina_hg38.list --CHAIN hg19ToHg38.over.chain -SD  ../../genome/GRCh38.p13/genomeRef.dict
- [X] puis en bed
***** TODO VCF de référence
****** DONE NA12878 (HG001)
CLOSED: [2023-02-25 Sat 19:46]
******* DONE Fastq HiSeq
CLOSED: [2023-02-25 Sat 19:46]
On prend le Hiseq, qui est probablement ce qu'utilise Centogène :
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/
On utilisé les données "trimmés" (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1069-7), i.e qui ont enlevé les fragments plus petits que la taille d'un read.
Informations:
- https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/Garvan_NA12878_HG001_HiSeq_Exome.README
- Sequencer: HiSeq2500
- kit: Nextera Rapid Capture Exome and Expanded Exome
Il y a 2 samples (NIST7035 et NIST7086), chacun sur 2 lanes -> à concaténer
NB : liste techno illumina https://www.illumina.com/systems/sequencing-platforms.html
Hiseq postérieur nextseq 550
******* DONE Capture : Exons (bed)
CLOSED: [2023-02-25 Sat 19:46]
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/nexterarapidcapture_expandedexome_targetedregions.bed.gz
******* DONE Bed, vcf
CLOSED: [2023-02-24 Fri 23:45]
****** DONE Ashkenazy trio HG002, HG003, HGQ004
CLOSED: [2023-04-06 Thu 21:43] SCHEDULED: <2023-04-01 Sat>
****** TODO Chinese trio HG005, 6, 7
***** TODO Fastq :fastq:
****** DONE NA12878 (HG001)
CLOSED: [2023-02-25 Sat 19:46]
******* DONE Fastq HiSeq
CLOSED: [2023-02-25 Sat 19:46]
On prend le Hiseq, qui est probablement ce qu'utilise Centogène :
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/
On utilisé les données "trimmés" (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-016-1069-7), i.e qui ont enlevé les fragments plus petits que la taille d'un read.
Informations:
- https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/Garvan_NA12878_HG001_HiSeq_Exome.README
- Sequencer: HiSeq2500
- kit: Nextera Rapid Capture Exome and Expanded Exome
Il y a 2 samples (NIST7035 et NIST7086), chacun sur 2 lanes -> à concaténer
NB : liste techno illumina https://www.illumina.com/systems/sequencing-platforms.html
Hiseq postérieur nextseq 550
******* DONE Capture : Exons (bed)
CLOSED: [2023-02-25 Sat 19:46]
https://ftp-trace.ncbi.nih.gov/ReferenceSamples/giab/data/NA12878/Garvan_NA12878_HG001_HiSeq_Exome/nexterarapidcapture_expandedexome_targetedregions.bed.gz
****** TODO Ashkenazy trio HG002, HG003, HG004
SCHEDULED: <2023-04-05 Wed>
******* TODO Capture
https://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/data/AshkenazimTrio/analysis/OsloUniversityHospital_Exome_GATK_jointVC_11242015/wex_Agilent_SureSelect_v05_b37.baits.slop50.merged.list
******* TODO Bam à partir des fastq
Bam + index + checksum
https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/AshkenazimTrio/alignment.index.AJtrio_OsloUniversityHospital_IlluminaExome_bwamem_GRCh37_11252015
****** TODO Chinese trio
Whole exome pour HG005 seulement
******* TODO HG005
https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/ChineseTrio/alignment.index.Chinesetrio_HG005_OsloUniversityHospital_IlluminaExome_bwamem_GRCh37_11252015
**** TODO NA12878 / HG001 :na12878:
***** DONE Discussion alexis : Mail
CLOSED: [2023-03-29 Wed 22:40]
Avec le patient NA12878 et comparaison avec hap.py du VCF de Genome In A Bottle ("gold" standard), on avait pour rappel
- sensibilité (=recall) 71% pour indel, 85% SNP
- précision  (= VPP) 69 et 97% respectivement
| Type  | TRUTH |    TP |   FN | QUERY |   FP |  UNK | FP.gt | FP.al |   Recall | Precision |
| INDEL |  4871 |  3461 | 1410 |  7048 | 1554 | 1987 |   193 |   346 | 0.710532 |  0.692946 |
| SNP   | 46032 | 39369 | 6663 | 44600 | 1186 | 4041 |   304 |    30 | 0.855253 |  0.970759 |
Les statistiques sur les génomes sont bien meilleurs (cf precisionFDA challenge).
Pour les exome, un article [1] a fait a des meilleures stats sur ce patient avec BWA et GATK mais ils ont moins de variant (on a presque un facteur 2 !).
Je soupçonne qu'on ne travaille pas sur les mêmes zones de capture (pas réussi à récupérer leur .bed)
| Exome | Type  |    TP |   FP |  FN | Sensitivity | Precision | F-Score |   FDR |
|     1 | SNV   | 23689 | 1397 | 613 |       0.975 |     0.944 |

Replacement in projects/bisonex.org at line 35 [10.35]

B:BD[13.65044] → [13.65044:66723]

B:BD[13.66723] → [12.26274:32787]

B:BD[12.32787] → [4.16:6381]

T
45515164        AA -> A
NB: testée avec pre.py
#+begin_src
grep '^#' NA12878_NIST7035_DP_over_30.vcf > NA12878_NIST7035_DP_over_30_chr21.vcf
grep '^NC_000021' NA12878_NIST7035_DP_over_30.vcf >> NA12878_NIST7035_DP_over_30_chr21.vcf
pre.py NA12878_NIST7035_DP_over_30_chr21.vcf out.vcf.gz -r GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
#+end_src
Selon l'algorithm, on devrait avoir TA -> T et non AA -> A (non left-align)
Si on débug:
#+begin_src
grep "^#" NA12878_NIST7035_DP_over_30_chr21.vcf > test_align.vcf
grep 45515163 NA12878_NIST7035_DP_over_30_chr21.vcf  >> test_align.vcf.gz
bgzip test_align.vcf.gz
bcftools index test_align.vcf.gz
multimerge test_align.vcf.gz -o out.vcf.gz -r GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
#+end_src
Idem... Si on utile un genome de référence plus récent ?
#+begin_src
multimerge test_align.vcf.gz -o out.vcf.gz -r /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna --process-full=1
#+end_src
Idem. Et vcfeval ?
#+begin_src
tabix HG001_GRCh38_1_22_v4.2.1_benchmark_chr21.vcf.gz
tabix test_align.vcf.gz
rtg vcfeval -b HG001_GRCh38_1_22_v4.2.1_benchmark_chr21.vcf.gz -c test_align.vcf.gz -t /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.sdf -o vcfeval-test
#+end_src
Selected score threshold using: maximized F-measure
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
   99.000                  0              0          1      54828     0.0000       0.0000     0.0000
     None                  0              0     
     1      54828     0.0000       0.0000     0.0000
     On essaie les différentes étapes
#+begin_src
multimerge test_align.vcf.gz -o out.vcf.gz -r /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna --homref-split 1 --homref-vcf-out 1 --trimalleles 1 --splitalleles 1
#+end_src
1er changement
TAA T
TAA TA
Après réflexion, c'est la référence qui n'est pas normalisée ! (left-trimmed)
******** DONE NC_000021.9     14108836  T -> C
CLOSED: [2023-03-04 Sat 11:01]
Dans le bam mais filtré par DP
******* DONE variant calling seul : meilleur score pour l'instant (77% recall, 95% precision)
CLOSED: [2023-03-04 Sat 11:01]
tests/chr21-alexis
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.Precision
INDEL    ALL           76        43        33           82        18         20      3      5       0.565789          0.709677
  SNP    ALL          582       448       134          530        25         57      6      1       0.769759          0.947146
 METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
       0.243902         0.629617                     NaN                     NaN                   1.181818                   1.612903
       0.107547         0.849289                3.098592                2.925926                   1.530435                   1.774869
******** NC_000021.9:14144627 : FN, dans le bam mais pas dans le vcf (2 reads/9)
 On récupére tout le vcf: pas dedans
 Dans le bam : 9/2
 Idem pour le bam dans notre pipeline
 - base qualité 33 sur les 2 reads
https://gatk.broadinstitute.org/hc/en-us/articles/360043491652-When-HaplotypeCaller-and-Mutect2-do-not-call-an-expected-variant
https://gatk.broadinstitute.org/hc/en-us/articles/360035891111-Expected-variant-at-a-specific-site-was-not-called
On debug
#+begin_src
cd /Work/Users/apraga/bisonex/out/NA12878_NIST7035/preprocessing/applybqsr
samtools view -b NA12878_NIST7035.bam NC_000021.9 -o NA12878_NIST7035_chr21.bam
samtools index NA12878_NIST7035_chr21.bam
#+end_src
********* --debug
#+begin_src
gatk --java-options "-Xmx3g" HaplotypeCaller --input NA12878_NIST7035_chr21.bam \
    --output debug_chr1.vcf.gz \
    --reference /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna \
    --dbsnp /Work/Groups/bisonex/data/dbSNP/GRCh38.p13/dbSNP.gz \
    --tmp-dir . \
    --max-mnp-distance 2 --debug &> lol.txt
#+end_src
Les allèles sont bien retrouvées
#+begin_quote
14:41:05.530 INFO  EventMap - === Best Haplotypes ===
14:41:05.530 INFO  EventMap - AATTTAATTTTCTTACCTTTCTGGGTATGTAAGTGATTTTA
14:41:05.530 INFO  EventMap - > Cigar = 41M
14:41:05.530 INFO  EventMap - >> Events = EventMap{}
14:41:05.530 INFO  EventMap - AATTTAATTTTCTTACCTTTTTGGGTATGTAAGTGATTTTA
14:41:05.530 INFO  EventMap - > Cigar = 41M
14:41:05.530 INFO  EventMap - >> Events = EventMap{NC_000021.9:14144627-14144627 [C*, T],}
14:41:05.530 INFO  HaplotypeCallerGenotypingEngine - Genotyping event at 14144627 with alleles = [C*, T]
#+end_quote
NB: même en filtranrt sur le chromosome 21 avec julia, haplotypecaller parcourt tous les chromosomes
********* DONE --linked-de-bruijn-graph : idem
CLOSED: [2023-02-26 Sun 17:26]
********* DONE examine sortie --bamout : non présent
CLOSED: [2023-02-26 Sun 19:53]
#+begin_src
cd test/chr21-alexis
gatk --java-options "-Xmx3g" HaplotypeCaller \
    --input /Work/Users/apraga/bisonex/script/files/bam/NA12878_chr21.bam \
    --output debug_chr1.vcf.gz \
    --reference /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna \
    --dbsnp /Work/Groups/bisonex/data/dbSNP/GRCh38.p13/dbSNP.gz \
    --tmp-dir . \
    --max-mnp-distance 2 -bamout debug.bam
#+end_src
Pas de reads
 #+begin_quote
If you see nothing overlapping your region, then it might not have been flagged as active, or could have failed to assemble.
 #+end_quote
********* 14582339: FN mais pas de reads...
********* 14583327 idem
********* 17512551 idem
********* 17567111: difference d'haplotype
********* 17567621 pas de reads
****** DONE Comparer avec sortie du variant calling vcf donné par GIAB
CLOSED: [2023-04-02 Sun 17:11]
******* DONE vcfeval
CLOSED: [2023-04-01 Sat 11:59] SCHEDULED: <2023-04-01 Sat>
#+begin_src sh
 nextflow run workflows/test.nf -profile standard,helios -resume --test.vcfeval --test.giabVCF --outdir=test-giabVCF
cat test-giabVCF/vcfeval/output/summary.txt
#+end_src
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
    1.000              44818          44818       2892       6087     0.9394       0.8804     0.9089
     None              44819          44819       2896       6086     0.9393       0.8804     0.9089
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
    1.000              44818          44818       2892       6087     0.9394       0.8804     0.9089
     None              44819          44819       2896       6086     0.9393       0.8804     0.9089
******* DONE happy
CLOSED: [2023-04-01 Sat 11:56]
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.PrecisioN
INDEL   PASS         4871      3678      1193         7036      1299       2011    208    217       0.755081          0.741493
  SNP   PASS        46032     41138      4894        47694      1622       4930    362     31       0.893683          0.962071
 METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
       0.285816         0.748225                     NaN                     NaN                   1.617499                   2.524051
       0.103367         0.926617                2.529552                2.412446                   1.620686                   1.688868
****** DONE Statistiques avec vcfeval
CLOSED: [2023-04-02 Sun 17:10] SCHEDULED: <2023-04-01 Sat>
**** TODO Résumer résultats pour Paul + article :resultats:
SCHEDULED: <2023-04-02 Sun>
***** TODO HG001 :
SCHEDULED: <2023-04-02 Sun>
****** Données brutes
Version GIAB avec hap.py + vcfeva
l:
#+begin_src sh
NXF_OPTS=-D"user.name=${USER}" nextflow run workflows/test.nf -profile standard,helios -resume --outdir=compareNA12878-giab --test.happy  --test.query=giab --test.vcfeval
#+end_src
Notre version avec hap.py + vcfeval
#+begin_src sh
NXF_OPTS=-D"user.name=${USER}" nextflow run workflows/test.nf -profile standard,helios -resume --outdir=compareNA12878 --test.vcfeval --test.query="out/NA12878_NIST/variantCalling/haplotypecaller/NA12878_NIST.vcf.gz" --test.happy
#+end_src
On concatene les csv avec une colonne indicant le type
# awk '{if (NR==1) {print "Data,Algorithm" $0} else {print "bisonx,happy,"$0}}' compareNA12878/happy/NA12878.summary.csv
compareNA12878/happy/NA12878.summary.csv
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |        4871 |     3461 |     1410 |        7048 |     1554 |      1987 |   193 |   346 |      0.710532 |         0.692946 |       0.281924 |        0.701629 |                        |                        |        1.6174985978687606 |        3.0674091441969518 |
| INDEL | PASS   |        4871 |     3461 |     1410 |        7048 |     1554 |      1987 |   193 |   346 |      0.710532 |         0.692946 |       0.281924 |        0.701629 |                        |                        |        1.6174985978687606 |        3.0674091441969518 |
| SNP   | ALL    |       46032 |    39367 |     6665 |       44599 |     1186 |      4042 |   304 |    30 |      0.855209 |         0.970757 |        0.09063 |        0.909327 |      2.529551552318896 |      2.402150701647346 |        1.6206857273037931 |        1.6273423688862698 |
| SNP   | PASS   |       46032 |    39367 |     6665 |       44599 |     1186 |      4042 |   304 |    30 |      0.855209 |         0.970757 |        0.09063 |        0.909327 |      2.529551552318896 |      2.402150701647346 |        1.6206857273037931 |        1.6273423688862698 |
compareNA12878/vcfeval/NA12878.summary.txt
| Threshold | True-pos-baseline | True-pos-call | False-pos | False-neg | Precision | Sensitivity | F-measure |
|-----------+-------------------+---------------+-----------+-----------+-----------+-------------+-----------|
| 3.000     |             42789 |         42416 |      2598 |      8080 |    0.9423 |      0.8412 |    0.8889 |
| None      |             42798 |         42425 |      2616 |      8071 |    0.9419 |      0.8413 |    0.8888 |
Indel avec le plus petit seuil : zcat NA12878.non_snp_roc.tsv.gz
Attention à inverser precision et recall !
 zcat NA12878.non_snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.71390.7136
SNP avec le plus petit seuil : zcat NA12878.non_snp_roc.tsv.gz
Attention à inverser precision et recall !
$ zcat NA12878.snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.85470.9727
compareNA12878-giab/vcfeval/NA12878.summary.txt
| Threshold | True-pos-baseline | True-pos-call | False-pos | False-neg | Precision | Sensitivity | F-measure |
| 1.000     |             44812 |         44812 |      2878 |      6057 |    0.9397 |      0.8809 |    0.9093 |
| None      |             44813 |         44813 |      2882 |      6056 |    0.9396 |      0.8809 |    0.9093 |
SNP:
$ zcat NA12878.snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.89370.9621
indel
$ zcat NA12878.non_snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.75980.7445
compareNA12878-giab/happy/NA12878.summary.csv
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
|-------+--------+-------------+----------+----------+-------------+----------+-----------+-------+-------+---------------+------------------+----------------+-----------------+------------------------+------------------------+---------------------------+---------------------------|
| INDEL | ALL    |        4871 |     3678 |     1193 |        7036 |     1299 |      2011 |   208 |   217 |      0.755081 |         0.741493 |       0.285816 |        0.748225 |                        |                        |        1.6174985978687606 |        2.5240506329113925 |
| INDEL | PASS   |        4871 |     3678 |     1193 |        7036 |     1299 |      2011 |   208 |   217 |      0.755081 |         0.741493 |       0.285816 |        0.748225 |                        |                        |        1.6174985978687606 |        2.5240506329113925 |
| SNP   | ALL    |       46032 |    41138 |     4894 |       47694 |     1622 |      4930 |   362 |    31 |      0.893683 |         0.962071 |       0.103367 |        0.926617 |      2.529551552318896 |     2.4124463519313304 |        1.6206857273037931 |        1.6888675840288743 |
| SNP   | PASS   |       46032 |    41138 |     4894 |       47694 |     1622 |      4930 |   362 |    31 |      0.893683 |         0.962071 |       0.103367 |        0.926617 |      2.529551552318896 |     2.4124463519313304 |        1.6206857273037931 |         1.688867584028874 |
****** Résumé
| Données | Algorithm | Type    | Recall | Precision |
|---------+-----------+---------+--------+-----------|
| Bisonex | Happy     | SNP     | 0.8552 |    0.9708 |
| Bisonex | vcfeval   | SNP     | 0.8547 |    0.9727 |
| Bisonex | Happy     | INDEL   | 0.7105 |    0.6929 |
| Bisonex | vcfeval   | Non-SNP | 0.7139 |    0.7136 |
|---------+-----------+---------+--------+-----------|
| GIAB    | happy     | INDEL   | 0.7551 |    0.7415 |
| GIAB    | vcfeval   | INDEL   | 0.7598 |    0.7445 |
| GIAB    | happy     | SNP     | 0.8937 |    0.9621 |
| giab    | vcfeval   | SNP     | 0.8937 |    0.9621 |
***** WAIT Ashkenazi trio (père, mère)
SCHEDULED: <2023-04-02 Sun>
*** TODO Platinum genome
https://emea.illumina.com/platinumgenomes.html
*** TODO Séquencer NA12878
Discussion avec Paul : sous-traitant ne nous donnera pas les données, il faut commander l'ADN
** Divers
*** DONE Vérifier nombre de reads fastq - bam
CLOSED: [2022-10-09 Sun 22:31]

[13.65044]

T
45515164        AA -> A
NB: testée avec pre.py
#+begin_src
grep '^#' NA12878_NIST7035_DP_over_30.vcf > NA12878_NIST7035_DP_over_30_chr21.vcf
grep '^NC_000021' NA12878_NIST7035_DP_over_30.vcf >> NA12878_NIST7035_DP_over_30_chr21.vcf
pre.py NA12878_NIST7035_DP_over_30_chr21.vcf out.vcf.gz -r GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
#+end_src
Selon l'algorithm, on devrait avoir TA -> T et non AA -> A (non left-align)
Si on débug:
#+begin_src
grep "^#" NA12878_NIST7035_DP_over_30_chr21.vcf > test_align.vcf
grep 45515163 NA12878_NIST7035_DP_over_30_chr21.vcf  >> test_align.vcf.gz
bgzip test_align.vcf.gz
bcftools index test_align.vcf.gz
multimerge test_align.vcf.gz -o out.vcf.gz -r GCA_000001405.15_GRCh38_no_alt_analysis_set.fasta
#+end_src
Idem... Si on utile un genome de référence plus récent ?
#+begin_src
multimerge test_align.vcf.gz -o out.vcf.gz -r /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna --process-full=1
#+end_src
Idem. Et vcfeval ?
#+begin_src
tabix HG001_GRCh38_1_22_v4.2.1_benchmark_chr21.vcf.gz
tabix test_align.vcf.gz
rtg vcfeval -b HG001_GRCh38_1_22_v4.2.1_benchmark_chr21.vcf.gz -c test_align.vcf.gz -t /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.sdf -o vcfeval-test
#+end_src
Selected score threshold using: maximized F-measure
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
   99.000                  0              0          1      54828     0.0000       0.0000     0.0000
     None                  0              0          1      54828     0.0000       0.0000     0.0000
     On essaie les différentes étapes
#+begin_src
multimerge test_align.vcf.gz -o out.vcf.gz -r /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna --homref-split 1 --homref-vcf-out 1 --trimalleles 1 --splitalleles 1
#+end_src
1er changement
TAA T
TAA TA
Après réflexion, c'est la référence qui n'est pas normalisée ! (left-trimmed)
******** DONE NC_000021.9     14108836  T -> C
CLOSED: [2023-03-04 Sat 11:01]
Dans le bam mais filtré par DP
******* DONE variant calling seul : meilleur score pour l'instant (77% recall, 95% precision)
CLOSED: [2023-03-04 Sat 11:01]
tests/chr21-alexis
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.Precision
INDEL    ALL           76        43        33           82        18         20      3      5       0.565789          0.709677
  SNP    ALL          582       448       134          530        25         57      6      1       0.769759          0.947146
 METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
       0.243902         0.629617                     NaN                     NaN                   1.181818                   1.612903
       0.107547         0.849289                3.098592                2.925926                   1.530435                   1.774869
******** NC_000021.9:14144627 : FN, dans le bam mais pas dans le vcf (2 reads/9)
 On récupére tout le vcf: pas dedans
 Dans le bam : 9/2
 Idem pour le bam dans notre pipeline
 - base qualité 33 sur les 2 reads
https://gatk.broadinstitute.org/hc/en-us/articles/360043491652-When-HaplotypeCaller-and-Mutect2-do-not-call-an-expected-variant
https://gatk.broadinstitute.org/hc/en-us/articles/360035891111-Expected-variant-at-a-specific-site-was-not-called
On debug
#+begin_src
cd /Work/Users/apraga/bisonex/out/NA12878_NIST7035/preprocessing/applybqsr
samtools view -b NA12878_NIST7035.bam NC_000021.9 -o NA12878_NIST7035_chr21.bam
samtools index NA12878_NIST7035_chr21.bam
#+end_src
********* --debug
#+begin_src
gatk --java-options "-Xmx3g" HaplotypeCaller --input NA12878_NIST7035_chr21.bam \
    --output debug_chr1.vcf.gz \
    --reference /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna \
    --dbsnp /Work/Groups/bisonex/data/dbSNP/GRCh38.p13/dbSNP.gz \
    --tmp-dir . \
    --max-mnp-distance 2 --debug &> lol.txt
#+end_src
Les allèles sont bien retrouvées
#+begin_quote
14:41:05.530 INFO  EventMap - === Best Haplotypes ===
14:41:05.530 INFO  EventMap - AATTTAATTTTCTTACCTTTCTGGGTATGTAAGTGATTTTA
14:41:05.530 INFO  EventMap - > Cigar = 41M
14:41:05.530 INFO  EventMap - >> Events = EventMap{}
14:41:05.530 INFO  EventMap - AATTTAATTTTCTTACCTTTTTGGGTATGTAAGTGATTTTA
14:41:05.530 INFO  EventMap - > Cigar = 41M
14:41:05.530 INFO  EventMap - >> Events = EventMap{NC_000021.9:14144627-14144627 [C*, T],}
14:41:05.530 INFO  HaplotypeCallerGenotypingEngine - Genotyping event at 14144627 with alleles = [C*, T]
#+end_quote
NB: même en filtranrt sur le chromosome 21 avec julia, haplotypecaller parcourt tous les chromosomes
********* DONE --linked-de-bruijn-graph : idem
CLOSED: [2023-02-26 Sun 17:26]
********* DONE examine sortie --bamout : non présent
CLOSED: [2023-02-26 Sun 19:53]
#+begin_src
cd test/chr21-alexis
gatk --java-options "-Xmx3g" HaplotypeCaller \
    --input /Work/Users/apraga/bisonex/script/files/bam/NA12878_chr21.bam \
    --output debug_chr1.vcf.gz \
    --reference /Work/Groups/bisonex/data/genome/GRCh38.p13/genomeRef.fna \
    --dbsnp /Work/Groups/bisonex/data/dbSNP/GRCh38.p13/dbSNP.gz \
    --tmp-dir . \
    --max-mnp-distance 2 -bamout debug.bam
#+end_src
Pas de reads
 #+begin_quote
If you see nothing overlapping your region, then it might not have been flagged as active, or could have failed to assemble.
 #+end_quote
********* 14582339: FN mais pas de reads...
********* 14583327 idem
********* 17512551 idem
********* 17567111: difference d'haplotype
********* 17567621 pas de reads
****** DONE Comparer avec sortie du variant calling vcf donné par GIAB
CLOSED: [2023-04-02 Sun 17:11]
******* DONE vcfeval
CLOSED: [2023-04-01 Sat 11:59] SCHEDULED: <2023-04-01 Sat>
#+begin_src sh
 nextflow run workflows/test.nf -profile standard,helios -resume --test.vcfeval --test.giabVCF --outdir=test-giabVCF
cat test-giabVCF/vcfeval/output/summary.txt
#+end_src
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
    1.000              44818          44818       2892       6087     0.9394       0.8804     0.9089
     None              44819          44819       2896       6086     0.9393       0.8804     0.9089
Threshold  True-pos-baseline  True-pos-call  False-pos  False-neg  Precision  Sensitivity  F-measure
----------------------------------------------------------------------------------------------------
    1.000              44818          44818       2892       6087     0.9394       0.8804     0.9089
     None              44819          44819       2896       6086     0.9393       0.8804     0.9089
******* DONE happy
CLOSED: [2023-04-01 Sat 11:56]
 Type Filter  TRUTH.TOTAL  TRUTH.TP  TRUTH.FN  QUERY.TOTAL  QUERY.FP  QUERY.UNK  FP.gt  FP.al  METRIC.Recall  METRIC.PrecisioN
INDEL   PASS         4871      3678      1193         7036      1299       2011    208    217       0.755081          0.741493
  SNP   PASS        46032     41138      4894        47694      1622       4930    362     31       0.893683          0.962071
 METRIC.Frac_NA  METRIC.F1_Score  TRUTH.TOTAL.TiTv_ratio  QUERY.TOTAL.TiTv_ratio  TRUTH.TOTAL.het_hom_ratio  QUERY.TOTAL.het_hom_ratio
       0.285816         0.748225                     NaN                     NaN                   1.617499                   2.524051
       0.103367         0.926617                2.529552                2.412446                   1.620686                   1.688868
****** DONE Statistiques avec vcfeval
CLOSED: [2023-04-02 Sun 17:10] SCHEDULED: <2023-04-01 Sat>
**** DONE Résumer résultats pour Paul + article :resultats:
CLOSED: [2023-04-06 Thu 21:41] SCHEDULED: <2023-04-02 Sun>
***** DONE HG001 :
CLOSED: [2023-04-06 Thu 21:41] SCHEDULED: <2023-04-02 Sun>
****** Données brutes
Version GIAB avec hap.py + vcfeval:
#+begin_src sh
NXF_OPTS=-D"user.name=${USER}" nextflow run workflows/compareVCF.nf -profile standard,helios -resume --outdir=compareNA12878-giab --test.happy  --test.query=giab --test.vcfeval
#+end_src
Notre version avec hap.py + vcfeval
#+begin_src sh
NXF_OPTS=-D"user.name=${USER}" nextflow run workflows/compareVCF.nf -profile standard,helios -resume --outdir=compareNA12878 --test.vcfeval --test.query="out/NA12878_NIST/variantCalling/haplotypecaller/NA12878_NIST.vcf.gz" --test.happy
#+end_src
On concatene les csv avec une colonne indicant le type
# awk '{if (NR==1) {print "Data,Algorithm" $0} else {print "bisonx,happy,"$0}}' compareNA12878/happy/NA12878.summary.csv
compareNA12878/happy/NA12878.summary.csv
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
| INDEL | ALL    |        4871 |     3461 |     1410 |        7048 |     1554 |      1987 |   193 |   346 |      0.710532 |         0.692946 |       0.281924 |        0.701629 |                        |                        |        1.6174985978687606 |        3.0674091441969518 |
| INDEL | PASS   |        4871 |     3461 |     1410 |        7048 |     1554 |      1987 |   193 |   346 |      0.710532 |         0.692946 |       0.281924 |        0.701629 |                        |                        |        1.6174985978687606 |        3.0674091441969518 |
| SNP   | ALL    |       46032 |    39367 |     6665 |       44599 |     1186 |      4042 |   304 |    30 |      0.855209 |         0.970757 |        0.09063 |        0.909327 |      2.529551552318896 |      2.402150701647346 |        1.6206857273037931 |        1.6273423688862698 |
| SNP   | PASS   |       46032 |    39367 |     6665 |       44599 |     1186 |      4042 |   304 |    30 |      0.855209 |         0.970757 |        0.09063 |        0.909327 |      2.529551552318896 |      2.402150701647346 |        1.6206857273037931 |        1.6273423688862698 |
compareNA12878/vcfeval/NA12878.summary.txt
| Threshold | True-pos-baseline | True-pos-call | False-pos | False-neg | Precision | Sensitivity | F-measure |
|-----------+-------------------+---------------+-----------+-----------+-----------+-------------+-----------|
| 3.000     |             42789 |         42416 |      2598 |      8080 |    0.9423 |      0.8412 |    0.8889 |
| None      |             42798 |         42425 |      2616 |      8071 |    0.9419 |      0.8413 |    0.8888 |
Indel avec le plus petit seuil : zcat NA12878.non_snp_roc.tsv.gz
Attention à inverser precision et recall !
 zcat NA12878.non_snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.71390.7136
SNP avec le plus petit seuil : zcat NA12878.non_snp_roc.tsv.gz
Attention à inverser precision et recall !
$ zcat NA12878.snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.85470.9727
compareNA12878-giab/vcfeval/NA12878.summary.txt
| Threshold | True-pos-baseline | True-pos-call | False-pos | False-neg | Precision | Sensitivity | F-measure |
| 1.000     |             44812 |         44812 |      2878 |      6057 |    0.9397 |      0.8809 |    0.9093 |
| None      |             44813 |         44813 |      2882 |      6056 |    0.9396 |      0.8809 |    0.9093 |
SNP:
$ zcat NA12878.snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.89370.9621
indel
$ zcat NA12878.non_snp_roc.tsv.gz  | tail -n 1 | awk '{print $7 $6}'
0.75980.7445
compareNA12878-giab/happy/NA12878.summary.csv
| Type  | Filter | TRUTH.TOTAL | TRUTH.TP | TRUTH.FN | QUERY.TOTAL | QUERY.FP | QUERY.UNK | FP.gt | FP.al | METRIC.Recall | METRIC.Precision | METRIC.Frac_NA | METRIC.F1_Score | TRUTH.TOTAL.TiTv_ratio | QUERY.TOTAL.TiTv_ratio | TRUTH.TOTAL.het_hom_ratio | QUERY.TOTAL.het_hom_ratio |
|-------+--------+-------------+----------+----------+-------------+----------+-----------+-------+-------+---------------+------------------+----------------+-----------------+------------------------+------------------------+---------------------------+---------------------------|
| INDEL | ALL    |        4871 |     3678 |     1193 |        7036 |     1299 |      2011 |   208 |   217 |      0.755081 |         0.741493 |       0.285816 |        0.748225 |                        |                        |        1.6174985978687606 |        2.5240506329113925 |
| INDEL | PASS   |        4871 |     3678 |     1193 |        7036 |     1299 |      2011 |   208 |   217 |      0.755081 |         0.741493 |       0.285816 |        0.748225 |                        |                        |        1.6174985978687606 |        2.5240506329113925 |
| SNP   | ALL    |       46032 |    41138 |     4894 |       47694 |     1622 |      4930 |   362 |    31 |      0.893683 |         0.962071 |       0.103367 |        0.926617 |      2.529551552318896 |     2.4124463519313304 |        1.6206857273037931 |        1.6888675840288743 |
| SNP   | PASS   |       46032 |    41138 |     4894 |       47694 |     1622 |      4930 |   362 |    31 |      0.893683 |         0.962071 |       0.103367 |        0.926617 |      2.529551552318896 |     2.4124463519313304 |        1.6206857273037931 |         1.688867584028874 |
****** Résumé
| Données | Algorithm | Type    | Recall | Precision |
|---------+-----------+---------+--------+-----------|
| Bisonex | Happy     | SNP     | 0.8552 |    0.9708 |
| Bisonex | vcfeval   | SNP     | 0.8547 |    0.9727 |
| Bisonex | Happy     | INDEL   | 0.7105 |    0.6929 |
| Bisonex | vcfeval   | Non-SNP | 0.7139 |    0.7136 |
|---------+-----------+---------+--------+-----------|
| GIAB    | happy     | INDEL   | 0.7551 |    0.7415 |
| GIAB    | vcfeval   | INDEL   | 0.7598 |    0.7445 |
| GIAB    | happy     | SNP     | 0.8937 |    0.9621 |
| giab    | vcfeval   | SNP     | 0.8937 |    0.9621 |
***** TODO Ashkenazi trio (père, mère)
SCHEDULED: <2023-04-02 Sun>
*** TODO Platinum genome
https://emea.illumina.com/platinumgenomes.html
*** TODO Séquencer NA12878
Discussion avec Paul : sous-traitant ne nous donnera pas les données, il faut commander l'ADN
** Divers
*** DONE Vérifier nombre de reads fastq - bam
CLOSED: [2022-10-09 Sun 22:31]

File addition: templates (d--x------)
[6.9382]
File move: sitemap.xml → sitemap.xml
BF:BFD[6.1] → [6.743:778]
BF:BF[6.778] → [6.24:24]
[0.23190]
[6.24]
File move: post.html → post.html
BF:BFD[6.1] → [6.977:1010]
BF:BF[6.1010] → [6.780:780]
[0.23190]
[6.780]
File move: post-list.html → post-list.html
BF:BFD[6.1] → [6.1131:1169]
BF:BF[6.1169] → [6.1012:1012]
[0.23190]
[6.1012]
File move: notitle.html → notitle.html
BF:BFD[6.1] → [14.868:904]
BF:BF[14.904] → [14.5:5]
[0.23190]
[14.5]
File move: notes-list.html → notes-list.html
BF:BFD[6.1] → [14.1016:1055]
BF:BF[14.1055] → [14.906:906]
[0.23190]
[14.906]
File move: default.html → default.html
BF:BFD[6.1] → [6.2120:2156]
BF:BF[6.2156] → [6.1171:1171]
[0.23190]
[6.1171]
File move: archive.html → archive.html
BF:BFD[6.1] → [6.2239:2275]
BF:BF[6.2275] → [6.2158:2158]
[0.23190]
[6.2158]
File move: publish.el → publish.el
BF:BFD[5.2] → [7.3:37]
BF:BF[7.37] → [15.223:223]
[6.9382]
[15.223]
File addition: posts (d--x------)
[6.9382]
File move: 2021-06-01-emacs.org → 2021-06-01-emacs.org
BF:BFD[7.149] → [16.39:83]
BF:BF[16.83] → [17.586648:586648]
[0.23516]
[17.586648]
File move: 2021-03-01-stream-media.org → 2021-03-01-stream-media.org
BF:BFD[7.149] → [16.85:136]
BF:BF[16.136] → [18.110854:110854]
[0.23516]
[18.110854]
File move: 2021-01-02-command-line.org → 2021-01-02-command-line.org
BF:BFD[7.149] → [16.138:189]
BF:BF[16.189] → [17.669758:669758]
[0.23516]
[17.669758]
File move: 2020-10-01-mail.org → 2020-10-01-mail.org
BF:BFD[7.149] → [16.191:234]
BF:BF[16.234] → [19.258514:258514]
[0.23516]
[19.258514]
File move: index.html → index.html
BF:BFD[5.2] → [20.463:497]
BF:BF[20.497] → [20.206:206]
[6.9382]
[20.206]
File move: flake.nix → flake.nix
BF:BFD[5.2] → [6.6408:6441]
BF:BF[6.6441] → [6.5074:5074]
[6.9382]
[6.5074]
File move: flake.lock → flake.lock
BF:BFD[5.2] → [6.7441:7475]
BF:BF[6.7475] → [6.6443:6443]
[6.9382]
[6.6443]
File addition: files (d--x------)
[6.9382]
File move: walls_of_metz_small.png → walls_of_metz_small.png
BF:BFD[2.297] → [2.382653:382707]
BF:BF[2.382707] → [2.316:316]
[0.23839]
[2.316]
File move: walls_of_metz_large.png → walls_of_metz_large.png
BF:BFD[2.297] → [2.2729688:2729742]
BF:BF[2.2729742] → [2.382709:382709]
[0.23839]
[2.382709]
File move: vivaldi_310_1.tar.gz → vivaldi_310_1.tar.gz
BF:BFD[2.297] → [2.2733069:2733120]
BF:BF[2.2733120] → [2.2729744:2729744]
[0.23839]
[2.2729744]
File move: vivaldi_310_1.pdf → vivaldi_310_1.pdf
BF:BFD[2.297] → [2.3445954:3446002]
BF:BF[2.3446002] → [2.2733122:2733122]
[0.23839]
[2.2733122]
File move: sir.pdf → sir.pdf
BF:BFD[2.297] → [2.3761056:3761094]
BF:BF[2.3761094] → [2.3446004:3446004]
[0.23839]
[2.3446004]
File move: pangolin_200809.swf → pangolin_200809.swf
BF:BFD[2.297] → [2.10014658:10014708]
BF:BF[2.10014708] → [2.3761096:3761096]
[0.23839]
[2.3761096]
File move: pangolin_200809.ogv → pangolin_200809.ogv
BF:BFD[2.297] → [2.12544157:12544207]
BF:BF[2.12544207] → [2.10014710:10014710]
[0.23839]
[2.10014710]
File move: pangolin_200809.mp4 → pangolin_200809.mp4
BF:BFD[2.297] → [2.14386560:14386610]
BF:BF[2.14386610] → [2.12544209:12544209]
[0.23839]
[2.12544209]
File move: movie.html → movie.html
BF:BFD[2.297] → [2.14387331:14387365]
BF:BF[2.14387365] → [2.14386612:14386612]
[0.23839]
[2.14386612]
File addition: maps (d--x------)
[0.23839]
File move: metz_remparts_old.pdf → metz_remparts_old.pdf
BF:BFD[2.14387367] → [2.17644346:17644398]
BF:BF[2.17644398] → [2.14387385:14387385]
[0.24305]
[2.14387385]
File move: metz_remparts_current.pdf → metz_remparts_current.pdf
BF:BFD[2.14387367] → [2.19062127:19062183]
BF:BF[2.19062183] → [2.17644400:17644400]
[0.24305]
[2.17644400]
File move: metz_old_and_new.pdf → metz_old_and_new.pdf
BF:BFD[2.14387367] → [2.23254763:23254814]
BF:BF[2.23254814] → [2.19062185:19062185]
[0.24305]
[2.19062185]
File move: mandelbrot_noalias.png → mandelbrot_noalias.png
BF:BFD[2.297] → [2.23390678:23390731]
BF:BF[2.23390731] → [2.23254816:23254816]
[0.23839]
[2.23254816]
File move: mandelbrot_bw.png → mandelbrot_bw.png
BF:BFD[2.297] → [2.23396857:23396905]
BF:BF[2.23396905] → [2.23390733:23390733]
[0.23839]
[2.23390733]
File move: mandelbrot_alias.png → mandelbrot_alias.png
BF:BFD[2.297] → [2.23487218:23487269]
BF:BF[2.23487269] → [2.23396907:23396907]
[0.23839]
[2.23396907]
File addition: mandelbrot (d--x------)
[0.23839]
File move: Mandelbrot.java → Mandelbrot.java
BF:BFD[2.23487271] → [2.23493866:23493905]
BF:BF[2.23493905] → [2.23487295:23487295]
[0.24646]
[2.23487295]
File move: Mandelbrot.jar → Mandelbrot.jar
BF:BFD[2.23487271] → [2.23494555:23494600]
BF:BF[2.23494600] → [2.23493907:23493907]
[0.24646]
[2.23493907]
File move: Mandelbrot.class → Mandelbrot.class
BF:BFD[2.23487271] → [2.23499661:23499708]
BF:BF[2.23499708] → [2.23494602:23494602]
[0.24646]
[2.23494602]
File move: Mandelbrot → Mandelbrot
BF:BFD[2.23487271] → [2.23500057:23500098]
BF:BF[2.23500098] → [2.23499710:23499710]
[0.24646]
[2.23499710]
File move: gollum_blender.png → gollum_blender.png
BF:BFD[2.297] → [2.23634177:23634226]
BF:BF[2.23634226] → [2.23500100:23500100]
[0.23839]
[2.23500100]
File move: gollum.jpg → gollum.jpg
BF:BFD[2.297] → [2.23657620:23657661]
BF:BF[2.23657661] → [2.23634228:23634228]
[0.23839]
[2.23634228]
File move: git_pres.pdf → git_pres.pdf
BF:BFD[2.297] → [2.24159688:24159731]
BF:BF[2.24159731] → [2.23657663:23657663]
[0.23839]
[2.23657663]
File move: cv_alexis_praga_fr.pdf → cv_alexis_praga_fr.pdf
BF:BFD[2.297] → [2.24220370:24220423]
BF:BF[2.24220423] → [2.24159733:24159733]
[0.23839]
[2.24159733]
File move: cv_alexis_praga_en.pdf → cv_alexis_praga_en.pdf
BF:BFD[2.297] → [2.24294240:24294293]
BF:BF[2.24294293] → [2.24220425:24220425]
[0.23839]
[2.24220425]
File move: .htaccess → .htaccess
BF:BFD[2.297] → [2.24294343:24294376]
BF:BF[2.24294376] → [2.24294295:24294295]
[0.23839]
[2.24294295]

File addition: default.nix (----------)

[6.9382]

let
  pkgs = import <nixpkgs> { }; # pin the channel to ensure reproducibility!
in
pkgs.haskellPackages.developPackage {
  root = ./.;
}

File addition: css (d--x------)
[6.9382]
File move: default.css → default.css
BF:BFD[6.7509] → [6.9334:9369]
BF:BF[6.9369] → [6.7526:7526]
[0.25310]
[6.7526]
Replacement in blog/blog.cabal at line 11 [6.9400]
B:BD[6.9620] → [6.9620:9653]
```
  build-depends:     base >= 4.8
```
[6.9620]
[3.660]
```
  build-depends:     base
```

Replacement in blog/blog.cabal at line 13 [6.9400]

∅:D[3.690] → [6.9653:9767]

B:BD[6.9653] → [6.9653:9767]

                   , hakyll >= 4.15
                   , pandoc == 2.17.*
                   , slugger >= 0.1.0.1

[3.690]

[3.691]

                   , hakyll

Deletion in blog/blog.cabal at line 15 [6.9400]
∅:D[3.718] → [6.9767:9800]
B:BD[6.9767] → [6.9767:9800]
```
                   , text >= 1.2
```

Replacement in blog/Main.hs at line 3 [6.9907]

B:BD[6.9943] → [16.629:692]

B:BD[16.692] → [3.720:816]

import           Data.Monoid (mappend)
import           Hakyll
import Data.List.Split
import Data.List
import Data.Char
import           System.FilePath.Posix

[6.9943]

[6.10485]

import  Data.Monoid (mappend)
import  Hakyll
import  Data.List.Split
import  Data.List
import  Data.Char
import  System.FilePath.Posix

Replacement in blog/Main.hs at line 34 [6.9907]
B:BD[14.1902] → [14.1902:1933]
```
            >>= relativizeUrls
```
[14.1902]
[6.11459]
```
            >>= relativizeUrl-- s
```

Simpler nix configuration for hakyll

Dependencies

In channels

Change contents

File deletion: templates

File deletion: posts

File deletion: files

File deletion: mandelbrot

File deletion: maps

File deletion: haskell-overlay.nix

File deletion: css

Replacement in projects/bisonex.org at line 30 [10.35]

Replacement in projects/bisonex.org at line 35 [10.35]

File addition: templates (d--x------)

File move: sitemap.xml → sitemap.xml

File move: post.html → post.html

File move: post-list.html → post-list.html

File move: notitle.html → notitle.html

File move: notes-list.html → notes-list.html

File move: default.html → default.html

File move: archive.html → archive.html

File move: publish.el → publish.el

File addition: posts (d--x------)

File move: 2021-06-01-emacs.org → 2021-06-01-emacs.org

File move: 2021-03-01-stream-media.org → 2021-03-01-stream-media.org

File move: 2021-01-02-command-line.org → 2021-01-02-command-line.org

File move: 2020-10-01-mail.org → 2020-10-01-mail.org

File move: index.html → index.html

File move: flake.nix → flake.nix

File move: flake.lock → flake.lock

File addition: files (d--x------)

File move: walls_of_metz_small.png → walls_of_metz_small.png

File move: walls_of_metz_large.png → walls_of_metz_large.png

File move: vivaldi_310_1.tar.gz → vivaldi_310_1.tar.gz

File move: vivaldi_310_1.pdf → vivaldi_310_1.pdf

File move: sir.pdf → sir.pdf

File move: pangolin_200809.swf → pangolin_200809.swf

File move: pangolin_200809.ogv → pangolin_200809.ogv

File move: pangolin_200809.mp4 → pangolin_200809.mp4

File move: movie.html → movie.html

File addition: maps (d--x------)

File move: metz_remparts_old.pdf → metz_remparts_old.pdf

File move: metz_remparts_current.pdf → metz_remparts_current.pdf

File move: metz_old_and_new.pdf → metz_old_and_new.pdf

File move: mandelbrot_noalias.png → mandelbrot_noalias.png

File move: mandelbrot_bw.png → mandelbrot_bw.png

File move: mandelbrot_alias.png → mandelbrot_alias.png

File addition: mandelbrot (d--x------)

File move: Mandelbrot.java → Mandelbrot.java

File move: Mandelbrot.jar → Mandelbrot.jar

File move: Mandelbrot.class → Mandelbrot.class

File move: Mandelbrot → Mandelbrot

File move: gollum_blender.png → gollum_blender.png

File move: gollum.jpg → gollum.jpg

File move: git_pres.pdf → git_pres.pdf

File move: cv_alexis_praga_fr.pdf → cv_alexis_praga_fr.pdf

File move: cv_alexis_praga_en.pdf → cv_alexis_praga_en.pdf

File move: .htaccess → .htaccess

File addition: default.nix (----------)

File addition: css (d--x------)

File move: default.css → default.css

Replacement in blog/blog.cabal at line 11 [6.9400]

Replacement in blog/blog.cabal at line 13 [6.9400]

Deletion in blog/blog.cabal at line 15 [6.9400]

Replacement in blog/Main.hs at line 3 [6.9907]

Replacement in blog/Main.hs at line 34 [6.9907]