• Live Demo
• Animated US COVID-19 Deaths By State
• 3D Scatter Chart
• Surface Chart
• Network Analysis

What is it?

D-Tale is the combination of a Flask back-end and a React front-end to bring you an easy way to view & analyze Pandas data structures. It integrates seamlessly with ipython notebooks & python/ipython terminals. Currently this tool supports such Pandas objects as DataFrame, Series, MultiIndex, DatetimeIndex & RangeIndex.

Origins

D-Tale was the product of a SAS to Python conversion. What was originally a perl script wrapper on top of SAS's insight function is now a lightweight web client on top of Pandas data structures.

In The News

• 4 Libraries that can perform EDA in one line of python code
• React Status
• KDNuggets
• Man Institute (warning: contains deprecated functionality)
• Python Bytes
• FlaskCon 2020
• San Diego Python
• Medium: towards data science
• Medium: Exploratory Data Analysis – Using D-Tale
• EOD Notes: Using python and dtale to analyze correlations
• Data Exploration is Now Super Easy w/ D-Tale
• Practical Business Python

Tutorials

• Pip Install Python YouTube Channel
• machine_learning_2019
• D-Tale The Best Library To Perform Exploratory Data Analysis Using Single Line Of Code🔥🔥🔥🔥
• Explore and Analyze Pandas Data Structures w/ D-Tale
• Data Preprocessing simplest method 🔥

Related Resources

• Adventures In Flask While Developing D-Tale
• Adding Range Selection to react-virtualized
• Building Draggable/Resizable Modals
• Embedding Flask Apps within Streamlit

Contents

• Where To Get It
• Getting Started
  • Python Terminal
  • As A Script
  • Jupyter Notebook
  • Jupyterhub w/ Jupyter Server Proxy
  • Jupyterhub w/ Kubernetes
  • Docker Container
  • Google Colab
  • Kaggle
  • Binder
  • R with Reticulate
  • Startup with No Data
  • Command-line
  • Custom Command-line Loaders
  • Embedding Within Your Own Flask App
  • Embedding Within Your Own Django App
  • Embedding Within Streamlit
  • Running D-Tale On Gunicorn w/ Redis
  • Configuration
  • Authentication
  • Predefined Filters
  • Using Swifter
  • Behavior for Wide Dataframes
• UI
  • Dimensions/Ribbon Menu/Main Menu
  • Header
  • Resize Columns
  • Editing Cells
  • Copy Cells Into Clipboard
  • Main Menu Functions
    • XArray Operations, Describe, Outlier Detection, Custom Filter, Dataframe Functions, Merge & Stack, Summarize Data, Duplicates, Missing Analysis, Correlations, Predictive Power Score, Heat Map, Highlight Dtypes, Highlight Missing, Highlight Outliers, Highlight Range, Low Variance Flag, Instances, Code Exports, Export CSV, Load Data & Sample Datasets, Refresh Widths, About, Theme, Reload Data, Unpin/Pin Menu, Language, Shutdown
  • Column Menu Functions
    • Filtering, Moving Columns, Hiding Columns, Delete, Rename, Replacements, Lock, Unlock, Sorting, Formats, Describe (Column Analysis)
  • Charts
  • Network Viewer
  • Hotkeys
  • Menu Functions Depending on Browser Dimensions
• For Developers
  • Cloning
  • Running Tests
  • Linting
  • Formatting JS
  • Docker Development
  • Adding Language Support
• Global State/Data Storage
• Startup Behavior
• Documentation
• Dependencies
• Acknowledgements
• License

Where To get It

The source code is currently hosted on GitHub at: https://github.com/man-group/dtale

Binary installers for the latest released version are available at the Python package index and on conda using conda-forge.

# conda
conda install dtale -c conda-forge
# if you want to also use "Export to PNG" for charts
conda install -c plotly python-kaleido

# or PyPI
pip install dtale

Getting Started

PyCharm	jupyter

Python Terminal

This comes courtesy of PyCharm Feel free to invoke python or ipython directly and use the commands in the screenshot above and it should work

Issues With Windows Firewall

If you run into issues with viewing D-Tale in your browser on Windows please try making Python public under "Allowed Apps" in your Firewall configuration. Here is a nice article: How to Allow Apps to Communicate Through the Windows Firewall

Additional functions available programmatically

import dtale
import pandas as pd

df = pd.DataFrame([dict(a=1,b=2,c=3)])

# Assigning a reference to a running D-Tale process.
d = dtale.show(df)

# Accessing data associated with D-Tale process.
tmp = d.data.copy()
tmp['d'] = 4

# Altering data associated with D-Tale process
# FYI: this will clear any front-end settings you have at the time for this process (filter, sorts, formatting)
d.data = tmp

# Get raw dataframe w/ any sorting or edits made through the UI
d.data

# Get raw dataframe similar to '.data' along with any filters applied using the UI
d.view_data

# Shutting down D-Tale process
d.kill()

# Using Python's `webbrowser` package it will try and open your server's default browser to this process.
d.open_browser()

# There is also some helpful metadata about the process.
d._data_id  # The process's data identifier.
d._url  # The url to access the process.

d2 = dtale.get_instance(d._data_id)  # Returns a new reference to the instance running at that data_id.

dtale.instances()  # Prints a list of all ids & urls of running D-Tale sessions.

Duplicate data check

To help guard against users loading the same data to D-Tale multiple times and thus eating up precious memory, we have a loose check for duplicate input data. The check runs the following:

• Are row & column count the same as a previously loaded piece of data?
• Are the names and order of columns the same as a previously loaded piece of data?

If both these conditions are true then you will be presented with an error and a link to the previously loaded data. Here is an example of how the interaction looks:

As A Script

D-Tale can be run as script by adding subprocess=False to your dtale.show command. Here is an example script:

import dtale
import pandas as pd

if __name__ == '__main__':
      dtale.show(pd.DataFrame([1,2,3,4,5]), subprocess=False)

Jupyter Notebook

Within any jupyter (ipython) notebook executing a cell like this will display a small instance of D-Tale in the output cell. Here are some examples:

dtale.show	assignment	instance

If you are running ipython & Instances popup)| |--debug|turn on Flask's "debug" mode for your D-Tale instance| |--no-reaper|flag to turn off auto-reaping subprocess (kill D-Tale instances after an hour of inactivity), good for long-running displays | |--open-browser|flag to automatically open up your server's default browser to your D-Tale instance| |--force`|flag to force D-Tale to try an kill any pre-existing process at the port you've specified so it can use it|

Loading data from ArcticDB(high performance, serverless DataFrame database) (this requires either installing arcticdb or dtale[arcticdb])

dtale --arcticdb-uri lmdb:///<path> --arcticdb-library jdoe.my_lib --arcticdb-symbol my_symbol

If you would like to change your storage mechanism to ArcticDB then add the --arcticdb-use_store flag

dtale --arcticdb-uri lmdb:///<path> --arcticdb-library my_lib --arcticdb-symbol my_symbol --arcticdb-use_store

Loading data from arctic(high performance datastore for pandas dataframes) (this requires either installing arctic or dtale[arctic])

dtale --arctic-uri mongodb://localhost:27027 --arctic-library my_lib --arctic-symbol my_symbol --arctic-start 20130101 --arctic-end 20161231

Loading data from CSV

dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date

Loading data from EXCEL

dtale --excel-path /home/jdoe/my_csv.xlsx --excel-parse_dates date
dtale --excel-path /home/jdoe/my_csv.xls --excel-parse_dates date

Loading data from JSON

dtale --json-path /home/jdoe/my_json.json --json-parse_dates date

or

dtale --json-path http://json-endpoint --json-parse_dates date

Loading data from R Datasets

dtale --r-path /home/jdoe/my_dataset.rda

Loading data from SQLite DB Files

dtale --sqlite-path /home/jdoe/test.sqlite3 --sqlite-table test_table

Custom Command-line Loaders

Loading data from a Custom loader

• Using the DTALE_CLI_LOADERS environment variable, specify a path to a location containing some python modules
• Any python module containing the global variables LOADER_KEY & LOADER_PROPS will be picked up as a custom loader
  • LOADER_KEY: the key that will be associated with your loader. By default you are given arctic & csv (if you use one of these are your key it will override these)
  • LOADER_PROPS: the individual props available to be specified.
    • For example, with arctic we have host, library, symbol, start & end.
    • If you leave this property as an empty list your loader will be treated as a flag. For example, instead of using all the arctic properties we would simply specify --arctic (this wouldn't work well in arctic's case since it depends on all those properties)
• You will also need to specify a function with the following signature def find_loader(kwargs) which returns a function that returns a dataframe or None
• Here is an example of a custom loader:

from dtale.cli.clickutils import get_loader_options

'''
  IMPORTANT!!! This global variable is required for building any customized CLI loader.
  When find loaders on startup it will search for any modules containing the global variable LOADER_KEY.
'''
LOADER_KEY = 'testdata'
LOADER_PROPS = ['rows', 'columns']


def test_data(rows, columns):
    import pandas as pd
    import numpy as np
    import random
    from past.utils import old_div
    from pandas.tseries.offsets import Day
    from dtale.utils import dict_merge
    import string

    now = pd.Timestamp(pd.Timestamp('now').date())
    dates = pd.date_range(now - Day(364), now)
    num_of_securities = max(old_div(rows, len(dates)), 1)  # always have at least one security
    securities = [
        dict(security_id=100000 + sec_id, int_val=random.randint(1, 100000000000),
             str_val=random.choice(string.ascii_letters) * 5)
        for sec_id in range(num_of_securities)
    ]
    data = pd.concat([
        pd.DataFrame([dict_merge(dict(date=date), sd) for sd in securities])
        for date in dates
    ], ignore_index=True)[['date', 'security_id', 'int_val', 'str_val']]

    col_names = ['Col{}'.format(c) for c in range(columns)]
    return pd.concat([data, pd.DataFrame(np.random.randn(len(data), columns), columns=col_names)], axis=1)


# IMPORTANT!!! This function is required for building any customized CLI loader.
def find_loader(kwargs):
    test_data_opts = get_loader_options(LOADER_KEY, LOADER_PROPS, kwargs)
    if len([f for f in test_data_opts.values() if f]):
        def _testdata_loader():
            return test_data(int(test_data_opts.get('rows', 1000500)), int(test_data_opts.get('columns', 96)))

        return _testdata_loader
    return None

In this example we simplying building a dataframe with some dummy data based on dimensions specified on the command-line:

• --testdata-rows
• --testdata-columns

Here's how you would use this loader:

DTALE_CLI_LOADERS=./path_to_loaders bash -c 'dtale --testdata-rows 10 --testdata-columns 5'

Authentication

You can choose to use optional authentication by adding the following to your D-Tale .ini file (directions here):

[auth]
active = True
username = johndoe
password = 1337h4xOr

Or you can call the following:

import dtale.global_state as global_state

global_state.set_auth_settings({'active': True, 'username': 'johndoe', 'password': '1337h4x0r'})

If you have done this before initially starting D-Tale it will have authentication applied. If you are adding this after starting D-Tale you will have to kill your service and start it over.

When opening your D-Tale session you will be presented with a screen like this:

From there you can enter the credentials you either set in your .ini file or in your call to dtale.global_state.set_auth_settings and you will be brought to the main grid as normal. You will now have an additional option in your main menu to logout:

Instance Settings

Users can set front-end properties on their instances programmatically in the dtale.show function or by calling the update_settings function on their instance. For example:

import dtale
import pandas as pd

df = pd.DataFrame(dict(
  a=[1,2,3,4,5],
  b=[6,7,8,9,10],
  c=['a','b','c','d','e']
))
dtale.show(
  df,
  locked=['c'],
  column_formats={'a': {'fmt': '0.0000'}},
  nan_display='...',
  background_mode='heatmap-col',
  sort=[('a','DESC')],
  vertical_headers=True,
)

or

import dtale
import pandas as pd

df = pd.DataFrame(dict(
  a=[1,2,3,4,5],
  b=[6,7,8,9,10],
  c=['a','b','c','d','e']
))
d = dtale.show(
  df
)
d.update_settings(
  locked=['c'],
  column_formats={'a': {'fmt': '0.0000'}},
  nan_display='...',
  background_mode='heatmap-col',
  sort=[('a','DESC')],
  vertical_headers=True,
)
d

Here's a short description of each instance setting available:

column_formats

A dictionary of column name keys and their front-end display configuration. Here are examples of the different format configurations:

Example:

column_formats={
    'price': {'fmt': '0.00'},
    'url': {'fmt': {'link': True}},
    'description': {'fmt': {'truncate': 50}},
    'created_date': {'fmt': 'YYYY-MM-DD'}
}

sort

List of tuples which sort your dataframe. Each tuple contains the column name and sort direction ('ASC' or 'DESC').

Example:

sort=[('date', 'DESC'), ('name', 'ASC')]

background_mode

A string denoting one of the many background displays available in D-Tale.

range_highlights

Dictionary of column name keys and range configurations. If the value for that column matches a condition, it will be shaded with the specified color.

Example:

range_highlights={
    'a': {
        'active': True,
        'equals': {'active': True, 'value': 3, 'color': {'r': 255, 'g': 245, 'b': 157, 'a': 1}},      # light yellow
        'greaterThan': {'active': True, 'value': 3, 'color': {'r': 80, 'g': 227, 'b': 194, 'a': 1}},  # mint green
        'lessThan': {'active': True, 'value': 3, 'color': {'r': 245, 'g': 166, 'b': 35, 'a': 1}},     # orange
    }
}

Each column can have the following range conditions:

Each condition has these properties:

• active: Whether this condition is enabled (True/False)
• value: The value to compare against
• color: RGBA color object with keys r, g, b, a (values 0-255 for RGB, 0-1 for alpha)

vertical_headers

If set to True then the headers in your grid will be rotated 90 degrees vertically to conserve width.

column_edit_options

Dictionary of column name keys and lists of allowed values. When editing a cell in the specified column, users will be presented with a dropdown of the allowed values instead of free-form text input.

Example:

column_edit_options={
    'status': ['pending', 'approved', 'rejected'],
    'priority': ['low', 'medium', 'high', 'critical']
}

Predefined Filters

Users can build their own custom filters which can be used from the front-end using the following code snippet:

import pandas as pd
import dtale
import dtale.predefined_filters as predefined_filters
import dtale.global_state as global_state

global_state.set_app_settings(dict(open_predefined_filters_on_startup=True))

predefined_filters.set_filters([
    {
        "name": "A and B > 2",
        "column": "A",
        "description": "Filter A with B greater than 2",
        "handler": lambda df, val: df[(df["A"] == val) & (df["B"] > 2)],
        "input_type": "input",
        "default": 1,
        "active": False,
    },
    {
        "name": "A and (B % 2) == 0",
        "column": "A",
        "description": "Filter A with B mod 2 equals zero (is even)",
        "handler": lambda df, val: df[(df["A"] == val) & (df["B"] % 2 == 0)],
        "input_type": "select",
        "default": 1,
        "active": False,
    },
    {
        "name": "A in values and (B % 2) == 0",
        "column": "A",
        "description": "A is within a group of values and B mod 2 equals zero (is even)",
        "handler": lambda df, val: df[df["A"].isin(val) & (df["B"] % 2 == 0)],
        "input_type": "multiselect",
        "default": [1],
        "active": True,
    }
])

df = pd.DataFrame(
    ([[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12], [13, 14, 15, 16, 17, 18]]),
    columns=['A', 'B', 'C', 'D', 'E', 'F']
)
dtale.show(df)

This code illustrates the types of inputs you can have on the front end:

• input: just a simple text input box which users can enter any value they want (if the value specified for "column" is an int or float it will try to convert the string to that data type) and it will be passed to the handler
• select: this creates a dropdown populated with the unique values of "column" (an asynchronous dropdown if the column has a large amount of unique values)
• multiselect: same as "select" but it will allow you to choose multiple values (handy if you want to perform an isin operation in your filter)

Here is a demo of the functionality:

If there are any new types of inputs you would like available please don't hesitate to submit a request on the "Issues" page of the repo.

Using Swifter

Swifter is a package which will increase performance on any apply() function on a pandas series or dataframe. If install the package in your virtual environment

pip install swifter
# or
pip install dtale[swifter]

It will be used for the following operations:

• Standard dataframe formatting in the main grid & chart display
• Column Builders
  • Type Conversions
    • string hex -> int or float
    • int or float -> hex
    • mixed -> boolean
    • int -> timestamp
    • date -> int
  • Similarity Distance Calculation
• Handling of empty strings when calculating missing counts
• Building unique values by data type in "Describe" popup

Behavior for Wide Dataframes

There is currently a performance bottleneck on the front-end when loading "wide dataframes" (dataframes with many columns). The current solution to this problem is that upon initial load of these dataframes to D-Tale any column with an index greater than 100 (going from left to right) will be hidden on the front-end. You can still unhide these columns the same way you would any other and you still have the option to show all columns using the "Describe" popup. Here's a sample of this behavior:

Say you loaded this dataframe into D-Tale.

import pandas as pd
import dtale

dtale.show(pd.DataFrame(
    {'col{}'.format(i): list(range(1000)) for i in range(105)}
))

You will now have access to a new "Jump To Column" menu item.

It would be too hard to scroll to the column you're looking for. So now you'll be able to type in the name of the column you're looking for and select it.

And now you'll see only the columns you've had locked (we've locked no columns in this example) and the column you chose to jump to.

Accessing CLI Loaders in Notebook or Console

I am pleased to announce that all CLI loaders will be available within notebooks & consoles. Here are some examples (the last working if you've installed dtale[arctic]):

• dtale.show_csv(path='test.csv', parse_dates=['date'])
• dtale.show_csv(path='http://csv-endpoint', index_col=0)
• dtale.show_excel(path='test.xlsx', parse_dates=['date'])
• dtale.show_excel(path='test.xls', sheet=)
• dtale.show_excel(path='http://excel-endpoint', index_col=0)
• dtale.show_json(path='http://json-endpoint', parse_dates=['date'])
• dtale.show_json(path='test.json', parse_dates=['date'])
• dtale.show_r(path='text.rda')
• dtale.show_arctic(uri='uri', library='library', symbol='symbol', start='20200101', end='20200101')
• dtale.show_arcticdb(uri='uri', library='library', symbol='symbol', start='20200101', end='20200101', use_store=True)

Using ArcticDB as your data store for D-Tale

So one of the major drawbacks of using D-Tale is that it stores a copy of your dataframe in memory (unless you specify the inplace=True when calling dtale.show). One way around this is to switch your storage mechanism to ArcticDB. This will use ArcticDB's QueryBuilder to perform all data loading and filtering. This will significantly drop your memory footprint, but it will remove a lot of the original D-Tale functionality:

• Custom Filtering
• Range filtering in Numeric Column Filters
• Regex filtering on String Column Filters
• Editing Cells
• Data Reshaping
• Dataframe Functions
• Drop Filtered Rows
• Sorting

If the symbol you're loading from ArcticDB contains more than 1,000,000 rows then you will also lose the following:

• Column Filtering using dropdowns of unique values (you'll have to manually type your values)
• Outlier Highlighting
• Most of the details in the "Describe" screen

In order to update your storage mechanism there are a few options, the first being use_arcticdb_store:

import dtale.global_state as global_state
import dtale

global_state.use_arcticdb_store(uri='lmdb:///<path>')
dtale.show_arcticdb(library='my_lib', symbol='my_symbol')

Or you can set your library ahead of time so you can use dtale.show:

import dtale.global_state as global_state
import dtale

global_state.use_arcticdb_store(uri='lmdb:///<path>', library='my_lib')
dtale.show('my_symbol')

Or you can set your library using dtale.show with a pipe-delimited identifier:

import dtale.global_state as global_state
import dtale

global_state.use_arcticdb_store(uri='lmdb:///<path>')
dtale.show('my_lib|my_symbol')

You can also do everything using dtale.show_arcticdb:

import dtale

dtale.show_arcticdb(uri='lmdb:///<path>', library='my_lib', symbol='my_symbol', use_store=True)

Navigating to different libraries/symbols in your ArcticDB database

When starting D-Tale with no data

import dtale.global_state as global_state
import dtale

global_state.use_arcticdb_store(uri='lmdb:///<path>')
dtale.show()

you'll be presented with this screen on startup

Once you choose a library and a symbol you can click "Load" and it will bring you to the main grid comprised of the data for that symbol.

You can also view information about the symbol you've selected before loading it by clicking the "Info" button

UI

Once you have kicked off your D-Tale session please copy & paste the link on the last line of output in your browser

Dimensions/Ribbon Menu/Main Menu

The information in the upper right-hand corner gives grid dimensions

• lower-left => row count
• upper-right => column count

Ribbon Menu

• hovering around to top of your browser will display a menu items (similar to the ones in the main menu) across the top of the screen
• to close the menu simply click outside the menu and/or dropdowns from the menu

Main Menu

• clicking the triangle displays the menu of standard functions (click outside menu to close it)

Header

The header gives users an idea of what operations have taken place on your data (sorts, filters, hidden columns). These values will be persisted across broswer instances. So if you perform one of these operations and then send a link to one of your colleagues they will see the same thing :)

Notice the "X" icon on the right of each display. Clicking this will remove those operations.

When performing multiple of the same operation the description will become too large to display so the display will truncate the description and if users click it they will be presented with a tooltip where you can crop individual operations. Here are some examples:

Sorts	Filters	Hidden Columns

Resize Columns

Currently there are two ways which you can resize columns.

• Dragging the right border of the column's header cell.

• Altering the "Maximum Column Width" property from the ribbon menu.

• Side Note: You can also set the max_column_width property ahead of time in your global configuration or programmatically using:

import dtale.global_state as global_state

global_state.set_app_settings(dict(max_column_width=100))

Editing Cells

You may edit any cells in your grid (with the exception of the row indexes or headers, the ladder can be edited using the Rename column menu function).

In order to edit a cell simply double-click on it. This will convert it into a text-input field and you should see a blinking cursor. In addition to turning that cell into an input it will also display an input at the top of the screen for better viewing of long strings. It is assumed that the value you type in will match the data type of the column you editing. For example:

• integers -> should be a valid positive or negative integer
• float -> should be a valid positive or negative float
• string -> any valid string will do
• category -> either a pre-existing category or this will create a new category for (so beware!)
• date, timestamp, timedelta -> should be valid string versions of each
• boolean -> any string you input will be converted to lowercase and if it equals "true" then it will make the cell True, otherwise False

Users can make use of two protected values as well:

• "nan" -> numpy.nan
• "inf" -> numpy.inf

To save your change simply press "Enter" or to cancel your changes press "Esc".

If there is a conversion issue with the value you have entered it will display a popup with the specific exception in question.

Here's a quick demo:

Here's a demo of editing cells with long strings:

Copy Cells Into Clipboard

Select	Copy	Paste

One request that I have heard time and time again while working on D-Tale is "it would be great to be able to copy a range of cells into excel". Well here is how that is accomplished:

1. Shift + Click on a cell
2. Shift + Click on another cell (this will trigger a popup)
3. Choose whether you want to include headers in your copy by clicking the checkbox
4. Click Yes
5. Go to your excel workbook and execute Ctrl + V or manually choose "Paste"

• You can also paste this into a standard text editor and what you're left with is tab-delimited data

Main Menu Functions

XArray Operations

• Convert To XArray: If you have are currently viewing a pandas dataframe in D-Tale you will be given this option to convert your data to an xarray.Dataset. It is as simple as selecting one or many columns as an index and then your dataframe will be converted to a dataset (df.set_index([...]).to_xarray()) which makes toggling between indexes slices much easier.

• XArray Dimensions: If you are currently viewing data associated with an xarray.Dataset you will be given the ability to toggle which dimension coordinates you're viewing by clicking this button. You can select values for all, some or none (all data - no filter) of your coordinates and the data displayed in your grid will match your selections. Under the hood the code being executed is as follows: ds.sel(dim1=coord1,...).to_dataframe()

Describe

View all the columns & their data types as well as individual details of each column

Data Type	Display	Notes
date
string		If you have less than or equal to 100 unique values they will be displayed at the bottom of your popup
int		Anything with standard numeric classifications (min, max, 25%, 50%, 75%) will have a nice boxplot with the mean (if it exists) displayed as an outlier if you look closely.
float

Outlier Detection

When viewing integer & float columns in the "Describe" popup you will see in the lower right-hand corner a toggle for Uniques & Outliers.

Outliers	Filter

If you click the "Outliers" toggle this will load the top 100 outliers in your column based on the following code snippet:

s = df[column]
q1 = s.quantile(0.25)
q3 = s.quantile(0.75)
iqr = q3 - q1
iqr_lower = q1 - 1.5 * iqr
iqr_upper = q3 + 1.5 * iqr
outliers = s[(s < iqr_lower) | (s > iqr_upper)]

If you click on the "Apply outlier filter" link this will add an addtional "outlier" filter for this column which can be removed from the header or the custom filter shown in picture above to the right.

Custom Filter

Starting with version 3.7.0 this feature will be turned off by default. Custom filters are vulnerable to code injection attacks, please only use in trusted environments.

You can turn this feature on by doing one of the following:

• add enable_custom_filters=True to your dtale.show call
• add enable_custom_filters = True to the [app] section of your dtale.ini config file (more info)
• run this code before calling dtale.show:

import dtale.global_state as global_state
global_state.set_app_settings(dict(enable_custom_filters=True))

Apply a custom pandas query to your data (link to pandas documentation included in popup)

Editing	Result

You can also see any outlier or column filters you've applied (which will be included in addition to your custom query) and remove them if you'd like.

Context Variables are user-defined values passed in via the context_variables argument to dtale.show(); they can be referenced in filters by prefixing the variable name with '@'.

For example, here is how you can use context variables in a pandas query:

import pandas as pd

df = pd.DataFrame([
  dict(name='Joe', age=7),
  dict(name='Bob', age=23),
  dict(name='Ann', age=45),
  dict(name='Cat', age=88),
])
two_oldest_ages = df['age'].nlargest(2)
df.query('age in @two_oldest_ages')

And here is how you would pass that context variable to D-Tale: dtale.show(df, context_variables=dict(two_oldest_ages=two_oldest_ages))

Here's some nice documentation on the performance of pandas queries

Dataframe Functions

This video shows you how to build the following:

• Numeric: adding/subtracting two columns or columns with static values
• Bins: bucketing values using pandas cut & qcut as well as assigning custom labels
• Dates: retrieving date properties (hour, weekday, month...) as well as conversions (month end)
• Random: columns of data type (int, float, string & date) populated with random uniformly distributed values.
• Type Conversion: switch columns from one data type to another, fun. :smile:

Merge & Stack

This feature allows users to merge or stack (vertically concatenate) dataframes they have loaded into D-Tale. They can also upload additional data to D-Tale while wihin this feature. The demo shown above goes over the following actions:

• Editing of parameters to either a pandas merge or stack (vertical concatenation) of dataframes
  • Viewing direct examples of each from the pandas documentation
• Selection of dataframes
• Uploading of additional dataframes from an excel file
• Viewing code & resulting data from a merge or stack

Summarize Data

This is very powerful functionality which allows users to create a new data from currently loaded data. The operations currently available are:

• Aggregation: consolidate data by running different aggregations on columns by a specific index
• Pivot: this is simple wrapper around pandas.Dataframe.pivot and pandas.pivot_table
• Transpose: transpose your data on a index (be careful dataframes can get very wide if your index has many unique values)

Function	Data
No Reshaping

Duplicates

Remove duplicate columns/values from your data as well as extract duplicates out into separate instances.

The folowing screen shots are for a dataframe with the following data:

Function	Description	Preview
Remove Duplicate Columns	Remove any columns that contain the same data as another and you can either keep the first, last or none of these columns that match this criteria. You can test which columns will be removed by clicking the "View Duplicates" button.
Remove Duplicate Column Names	Remove any columns with the same name (name comparison is case-insensitive) and you can either keep the first, last or none of these columns that match this criteria. You can test which columns will be removed by clicking the "View Duplicates" button.
Remove Duplicate Rows	Remove any rows from your dataframe where the values of a subset of columns are considered duplicates. You can choose to keep the first, last or none of the rows considered duplicated.
Show Duplicates	Break any duplicate rows (based on a subset of columns) out into another dataframe viewable in your D-Tale session. You can choose to view all duplicates or select specific groups based on the duplicated value.

Missing Analysis

Display charts analyzing the presence of missing (NaN) data in your dataset using the missingno pacakage. You can also open them in a tab by themselves or export them to a static PNG using the links in the upper right corner. You can also close the side panel using the ESC key.

Chart	Sample
Matrix
Bar
Heatmap
Dendrogram

Charts

Build custom charts based off your data(powered by plotly/dash).

• The Charts will open in a tab because of the fact there is so much functionality offered there you'll probably want to be able to reference the main grid data in the original tab
• To build a chart you must pick a value for X & Y inputs which effectively drive what data is along the X & Y axes
  • If you are working with a 3-Dimensional chart (heatmap, 3D Scatter, Surface) you'll need to enter a value for the Z axis as well
• Once you have entered all the required axes a chart will be built
• If your data along the x-axis (or combination of x & y in the case of 3D charts) has duplicates you have three options:
  • Specify a group, which will create series for each group
  • Specify an aggregation, you can choose from one of the following: Count, First, Last, Mean, Median, Minimum, Maximum, Standard Deviation, Variance, Mean Absolute Deviation, Product of All Items, Sum, Rolling
    • Specifying a "Rolling" aggregation will also require a Window & a Computation (Correlation, Count, Covariance, Kurtosis, Maximum, Mean, Median, Minimum, Skew, Standard Deviati