{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Explore sorters weaknesses with ground-truth comparison\n\nHere a synthetic dataset will demonstrate some weaknesses.\n\nStandard weaknesses :\n\n  * not all units are detected\n  * a unit is detected, but not all of its spikes (false negatives)\n  * a unit is detected, but it detects too many spikes (false positives)\n\nOther weaknesses:\n\n  * detect too many units (false positive units)\n  * detect units twice (or more) (redundant units = oversplit units)\n  * several units are merged into one units (overmerged units)\n\n\nTo demonstrate this the script `generate_erroneous_sorting.py` generate a ground truth sorting with 10 units.\nWe duplicate the results and modify it a bit to inject some \"errors\":\n\n  * unit 1 2 are perfect\n  * unit 3 4 have medium agreement\n  * unit 5 6 are over merge\n  * unit 7 is over split in 2 part\n  * unit 8 is redundant 3 times\n  * unit 9 is missing\n  * unit 10 have low agreement\n  * some units in tested do not exist at all in GT (15, 16, 17)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Import\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport spikeinterface.extractors as se\nimport spikeinterface.sorters as ss\nimport spikeinterface.comparison as sc\nimport spikeinterface.widgets as sw\n\nfrom generate_erroneous_sorting import generate_erroneous_sorting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here the agreement matrix\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "sorting_true, sorting_err = generate_erroneous_sorting()\ncomp = sc.compare_sorter_to_ground_truth(sorting_true, sorting_err, exhaustive_gt=True)\nsw.plot_agreement_matrix(comp, ordered=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here the same matrix but **ordered**\n\u00a0It is now quite trivial to check that fake injected errors are enlighted here.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "sw.plot_agreement_matrix(comp, ordered=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we can see that only Units 1 2 and 3 are well detected with 'accuracy'>0.75\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('well_detected', comp.get_well_detected_units(well_detected_score=0.75))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we can explore **\"false positive units\"** units that do not exists in ground truth\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('false_positive', comp.get_false_positive_units(redundant_score=0.2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we can explore **\"redundant units\"** units that do not exists in ground truth\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('redundant', comp.get_redundant_units(redundant_score=0.2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we can explore **\"overmerged units\"** units that do not exists in ground truth\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('overmerged', comp.get_overmerged_units(overmerged_score=0.2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we can explore **\"bad units\"** units that a mixed a several possible errors.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('bad', comp.get_bad_units())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There is a convenient function to summary everything.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comp.print_summary(well_detected_score=0.75, redundant_score=0.2, overmerged_score=0.2)\n\n\n\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}