KJNYAKSJRMLIGUAM22N7CTOBWYHZ2TFD2NY5WSFWRRDU5IEWOMIAC LGNTJMWSZ3DVZNAQMQQWSRSP46RXH5WXMBQ546LL6PVUTLSFJAUAC RHWQQAAHNHFO3FLCGVB3SIDKNOUFJGZTDNN57IQVBMXXCWX74MKAC 3WBY7ELOD4XA65WEV3UCLE6VUAGGF4T27WUYXN75QAOSYRPFEEMAC FXA3ZBV64FML7W47IPHTAJFJHN3J3XHVHFVNYED47XFSBIGMBKRQC YWWEIWM4CNSJTZE2FPFYBB3OMQOP62BKPL5PXEDUL5M3KLNRJQ3QC T3IPJM6TYF25RE2EQGASVSGIYJTJSJNZXBCEK3BNJ2LUYIBMTSSAC MHIFI3P3R5PVLHHZRDZH3FPFN5IEDJRTWIVCNEUHNUKVSGYZW6YQC QWCRR5CXIUZYZADCPUIE35VYCH4IEIEPPCOP2RV6TETADPS6H6SAC * <2023-01-09 Mon> ParkourDébloqué tic-tac + atterissage sur mur, kong* <2023-01-14 Sat> TrickingTraavil enchaînement : font, tornado, b-kick
** Validation*** GIAB: NA12878- methode https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/NA12878/analysis/Illumina_PlatinumGenomes_NA12877_NA12878_09162015/IlluminaPlatinumGenomes-user-guide.pdf- vcf https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/NA12878/analysis/Illumina_PlatinumGenomes_NA12877_NA12878_09162015/hg38/2.0.1/NA12878/Article comparant les variant calling : https://www.biorxiv.org/content/10.1101/2020.12.11.422022v1.full.pdfArticle pour vcfeval : https://www.nature.com/articles/s41587-019-0054-x
puis on compare le VCF avec les "high confidence" (Article : https://www.nature.com/articles/s41587-019-0054-x)- methode https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/NA12878/analysis/Illumina_PlatinumGenomes_NA12877_NA12878_09162015/IlluminaPlatinumGenomes-user-guide.pdf- vcf https://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/NA12878/analysis/Illumina_PlatinumGenomes_NA12877_NA12878_09162015/hg38/2.0.1/NA12878/2. On séquence directement NA12878
puis on compare le VCF avec les "high confidence"2. On séquence directement NA12878 -> inutile pour le pipeline seul
** TODO MAJ avec picardNormalement, GATK inclut picard mais la dernière version utilise picard pour certains outilshttps://gatk.broadinstitute.org/hc/en-us/articles/9570266920219--Tool-Documentation-IndexA compléter après validation*** TODO markduplicatesLa dernière version dans la documentation utilise picard !!** TODO Bwa-mem2 au lieu de bwa memhttps://github.com/bwa-mem2/bwa-mem2** TODO Parallélisation haplotypecallerspark est en beta, ne pas utiliserparallélisation du pauvre : se restreindre à un chromosome avec -L et paralléliser sur le nombre de chromosome** KILL CRAM au lieu de SAM ?CLOSED: [2022-12-30 Fri 20:38]Version compressée de bam mais :#+begin_quoteAll GATK tools that take in mapped read data expect a BAM file as primary format. Some support the CRAM format, but we have observed performance issues when working directly from CRAM files, so in our own work we convert CRAM to BAM first, and we only use CRAM for archival purposes#+end_quoteSource: https://gatk.broadinstitute.org/hc/en-us/articles/360035890791-SAM-or-BAM-or-CRAM-Mapped-sequence-data-formats
**** Relancer avec la version de prod + nix
**** TODO Relancer avec la version de prod + nix/Work/Projects/bisonex/ref-63003856_S135〉ls *.vcf* | insert nblines {|e| (^zgrep -v '^#' $e.name | wc -l)} | select name nblines 01/14/2023 08:04:02 PM╭───┬───────────────────────────────────────────────────────────────────────────┬─────────╮│ # │ name │ nblines │├───┼───────────────────────────────────────────────────────────────────────────┼─────────┤│ 0 │ 63003856_S135_DP_over_30.vcf.gz │ 84708 ││ 1 │ 63003856_S135_DP_over_30.vcf.gz.tbi │ 1 ││ 2 │ 63003856_S135_DP_over_30_not_SNP.recode.vcf │ 11362 ││ 3 │ 63003856_S135_DP_over_30_not_SNP_consensual_sequence.vcf │ 8864 ││ 4 │ 63003856_S135_DP_over_30_not_SNP_consensual_sequence_not_technical.vcf.gz │ 6478 │╰───┴───────────────────────────────────────────────────────────────────────────┴─────────╯/Work/Users/apraga/bisonex/script/files/tmp_63003856_S135〉ls *.vcf* | insert nblines {|e| (^zgrep -v '^#' $e.name | wc -l)} | select name nblines 01/14/2023 08:05:23 PM╭───┬───────────────────────────────────────────────────────────────────────────┬─────────╮│ # │ name │ nblines │├───┼───────────────────────────────────────────────────────────────────────────┼─────────┤│ 0 │ 63003856_S135_DP_over_30.vcf │ 84724 ││ 1 │ 63003856_S135_DP_over_30_not_SNP.recode.vcf │ 11377 ││ 2 │ 63003856_S135_DP_over_30_not_SNP_consensual_sequence.vcf │ 8884 ││ 3 │ 63003856_S135_DP_over_30_not_SNP_consensual_sequence_not_technical.vcf.gz │ 6759 │╰───┴───────────────────────────────────────────────────────────────────────────┴─────────╯**** TODO GATK : tests de non régression ??**** TODO Compiler gatk***** DONE CompilationCLOSED: [2023-01-14 Sat 22:45]Requirements :- java8 avec jDK- git lfs#+begin_src shgit clone https://github.com/broadinstitute/gatkgit checkout tags/4.2.4.0 -b 4.2.4.0nix-shell -p jdk8 git-lfs# We need the datasets for testing (otherwise it fails)git lfs installgit lfs pull --include src/main/resources/large./gradlew bundle#+end_src***** WAIT Vérifier tests#+begin_src./gradlew test#+end_src267064 tests completed, 444 failed, 1966 skipped> There were failing tests. See the report at: file:///Home/Users/apraga/gatk/build/reports/tests/test/index.htmlBUILD FAILED in 37m 1s6 actionable tasks: 1 executed, 5 up-to-dateCeux qui ont planté sont liés à spark + erreurs d'authentification kerberos***** TODO Lancer calcul : maisonJAVA_HOME=/nix/store/r1r5jr7gv6hcchpiggjmfqjkzbi8y5ja-openjdk-8u322-ga/lib/openjdk PATH=$PATH:$JAVA_HOME/bin ./gatk --version**** DONE Version prod à la maisonCLOSED: [2023-01-15 Sun 23:22]preprocessing + variant calling sans alignement***** DONE GATK nixCLOSED: [2023-01-15 Sun 23:22]❯ samtools view -c 63003856_S135_marked_dup.bam128077211On a le même nombre***** DONE GATK compilé: idemCLOSED: [2023-01-15 Sun 23:22]script/files-src/tmp_63003856_S135 on prod [$!?]❯ samtools view -c 63003856_S135_marked_dup.bam128077211
* Améliorations
* Amélioration** TODO MAJ avec picardNormalement, GATK inclut picard mais la dernière version utilise picard pour certains outilshttps://gatk.broadinstitute.org/hc/en-us/articles/9570266920219--Tool-Documentation-IndexA compléter après validation*** TODO markduplicatesLa dernière version dans la documentation utilise picard !!** TODO Bwa-mem2 au lieu de bwa memhttps://github.com/bwa-mem2/bwa-mem2** TODO Parallélisation haplotypecallerspark est en beta, ne pas utiliserparallélisation du pauvre : se restreindre à un chromosome avec -L et paralléliser sur le nombre de chromosome** KILL CRAM au lieu de SAM ?CLOSED: [2022-12-30 Fri 20:38]Version compressée de bam mais :#+begin_quoteAll GATK tools that take in mapped read data expect a BAM file as primary format. Some support the CRAM format, but we have observed performance issues when working directly from CRAM files, so in our own work we convert CRAM to BAM first, and we only use CRAM for archival purposes#+end_quoteSource: https://gatk.broadinstitute.org/hc/en-us/articles/360035890791-SAM-or-BAM-or-CRAM-Mapped-sequence-data-formats
* HOLD Implémenter d’autres pipelineVoir https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-021-04407-x** KILL GATKCLOSED: [2022 -11-11 Fri 20:01]https://broadinstitute.github.io/warp/docs/Pipelines/Exome_Germline_Single_Sample_Pipeline/READMEA priori, respecte les bonnes pratiques** KILL Essayer snmake avec bonne pratiqueshttps://github.com/snakemake-workflows/dna-seq-gatk-variant-calling/blob/main/.github/workflows/main.ymlInstaller Mamba (micromamba ne fonctionne pas sous nix)Ne fonctionne pas sous WSL2... MultiQC n’est pas assez à jourProblèmes de versions...** KILL SarekCLOSED: [2022-12-11 Sun 11:09]*** Dépendences**** Nix#+begin_src shnix profile install nixpkgs#mosdepth nixpkgs#python3nix-shell -p python310Packages.pyyaml --run "nextflow run nf-core/sarek -profile test --executor slurm --queue smp --outdir test -resume"#+end_src***** KILL derivation nix pour profile completCLOSED: [2022-12-11 Sun 11:09]**** KILL Sans nixCLOSED: [2022-09-24 Sat 10:20]On utilise conda#+begin_src shmodule unload nixmodule load anaconda3@2021.05/gcc-12.1.0module load nextflow@22.04.0/gcc-12.1.0module load openjdk@11.0.14.1_1/gcc-12.1.0nextflow run nf-core/sarek -profile conda,test --executor slurm --queue smp --outdir test -resume#+end_srcEssai 1: erreurs de permissions, corrigé en relancant le programme#+begin_quoteFailed to create Conda environmentcommand: conda create --mkdir --yes --quiet --prefix /Work/Users/apraga/test-sarek/work/conda/env-2d53b1db50de676670cf1a91ef0cf6db bioconda::tabix=1.11status : 1message:NotWritableError: The current user does not have write permissions to a required path.path: /Home/Users/apraga/.conda/pkgs/urls.txtuid: 1696gid: 513If you feel that permissions on this path are set incorrectly, you can manuallychange them by executing$ sudo chown 1696:513 /Home/Users/apraga/.conda/pkgs/urls.txt#+end_quoteCorrigé avec#+begin_src shchown 1696:513 /Home/Users/apraga/.conda/pkgs/urls.txt#+end_srcMais problème de proxy*** KILL Dérivation nix pour modules pythonCLOSED: [2022-12-11 Sun 11:09]*** KILL Lancer sarek en mode testCLOSED: [2022-12-11 Sun 11:09]#+begin_src shnix-shell -p python310Packages.pyyaml --run "nextflow run nf-core/sarek -profile test --executor slurm --queue smp --outdir test -resume"#+end_src*** KILL Lancer sarek sur données allégéesCLOSED: [2022-12-11 Sun 11:09]