Tutorial 1 - Rat retina
In [1]:
Copied!
import linmo.resample as resample
import linmo.plot as plot
import linmo.resample as resample
import linmo.plot as plot
First, we will import the rat retina dataset, Gomes et al., 2011. This dataset consists of 129 trees with at least 3 cells, acquired by time lapse microscopy of postnatal rat retinal progenitors grown in vitro. During this time stage, the progenitors give rise to mostly rod (R) cells, some bipolar (B) and amacrine (A) cells, and few Müller glia (M).
The data should first be formatted into a list where each entry is a tree is represented in NEWICK format, without branch lengths or interior nodes, separated by semi-colons.
In [2]:
Copied!
dataset = 'rat_retina'
dataset = 'rat_retina'
In [3]:
Copied!
all_trees_sorted = resample.read_dataset(f'datasets/{dataset}.txt')
# print the first ten trees
all_trees_sorted[:10]
all_trees_sorted = resample.read_dataset(f'datasets/{dataset}.txt')
# print the first ten trees
all_trees_sorted[:10]
Out[3]:
['((R,R),((A,B),((A,R),(B,M))))', '(R,((R,R),((R,R),((B,(R,R)),((B,M),(R,R))))))', '((A,R),(R,(R,(M,R))))', '(R,(M,R))', '(R,(M,R))', '(B,(M,(R,R)))', '(R,(R,(R,(M,R))))', '(B,(B,M))', '(R,(M,(R,(R,R))))', '(A,(M,R))']
We will first conduct doublet motif analysis by resampling the dataset num_resamples times with replacement. The following function resample_trees_doublets will automatically detect all cell fates in the dataset. It will output the doublet dictionary, a list of the detected cell fates, and a DataFrame that lists the number of occurrences for each doublet in each resample and in the original dataset. The DataFrame will also contain the expected number of occurrences for each doublet based on the probabilities of observing each of its constituent cell fates.
Doublet motif analysis¶
In [4]:
Copied!
(doublet_dict,
cell_fates,
dfs_c) = resample.resample_trees_doublets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
(doublet_dict,
cell_fates,
dfs_c) = resample.resample_trees_doublets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
100%|██████████| 10000/10000 [00:15<00:00, 641.65it/s] 0it [00:00, ?it/s] 0it [00:00, ?it/s]
In [5]:
Copied!
doublet_dict
doublet_dict
Out[5]:
{'(A,A)': 0,
'(A,B)': 1,
'(A,M)': 2,
'(A,R)': 3,
'(B,B)': 4,
'(B,M)': 5,
'(B,R)': 6,
'(M,M)': 7,
'(M,R)': 8,
'(R,R)': 9}
In [6]:
Copied!
cell_fates
cell_fates
Out[6]:
['A', 'B', 'M', 'R']
In [7]:
Copied!
dfs_c.head()
dfs_c.head()
Out[7]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 9992 | 9993 | 9994 | 9995 | 9996 | 9997 | 9998 | 9999 | observed | expected | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1.0 | 1.0 | 0.0 | 0.0 | 2.0 | 0.0 | 6.0 | 0.0 | 1.0 | 1.0 | ... | 1.0 | 2.0 | 1.0 | 1.0 | 3.0 | 1.0 | 1.0 | 3.0 | 5.0 | 1.468910 |
| 1 | 2.0 | 1.0 | 2.0 | 3.0 | 3.0 | 6.0 | 1.0 | 2.0 | 0.0 | 4.0 | ... | 2.0 | 3.0 | 2.0 | 4.0 | 5.0 | 3.0 | 5.0 | 8.0 | 12.0 | 3.655953 |
| 2 | 2.0 | 3.0 | 0.0 | 2.0 | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 | 1.0 | ... | 2.0 | 1.0 | 1.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.783419 |
| 3 | 18.0 | 20.0 | 19.0 | 20.0 | 15.0 | 25.0 | 20.0 | 21.0 | 20.0 | 13.0 | ... | 12.0 | 21.0 | 16.0 | 19.0 | 24.0 | 21.0 | 22.0 | 17.0 | 15.0 | 21.609295 |
| 4 | 2.0 | 4.0 | 2.0 | 2.0 | 2.0 | 2.0 | 2.0 | 1.0 | 2.0 | 4.0 | ... | 3.0 | 1.0 | 4.0 | 2.0 | 2.0 | 3.0 | 3.0 | 2.0 | 4.0 | 2.274815 |
5 rows × 10002 columns
We will now reformat the DataFrame for plotting. The next function dfs_for_plotting generates several DataFrames.
df_true_melt_subsetwill contain various characteristics about each of the doublets (i.e., the observed and expected number, z-score, adjusted p-value, etc.)df_melt_subsetwill contain the number of occurences for each subtree in all of the resamples.df_melt_100resamples_subsetwill contain the number of occurences for each subtree in only 100 resamples.df_null_zscores_i_c_melt_subsetwill contain the null z-score for each subtree innum_nullresamples. The null z-scores are calculated by comparison of each resample set to the rest of the resample sets.df_null_zscores_i_c_melt_100resamples_subsetwill contain the null z-score for each subtree in 100 resamples.
We can also specify a cutoff number for how many subtrees we would like to include in the final plot. We can use None to keep all subtrees, auto to keep all subtrees with an absolute z-score above 1, or any number of top significant subtrees to keep (in order of highest to lowest absolute z-score).
In [8]:
Copied!
(doublet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
doublet_dict,
cutoff=None,
num_null=1000,
use_expected=True,
min_cell_types=1)
(doublet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
doublet_dict,
cutoff=None,
num_null=1000,
use_expected=True,
min_cell_types=1)
100%|██████████| 10/10 [00:00<00:00, 1318.38it/s] 100%|██████████| 1000/1000 [00:13<00:00, 73.72it/s]
In [9]:
Copied!
df_true_melt_subset.head()
df_true_melt_subset.head()
Out[9]:
| subtree_val | observed | expected | z-score | abs z-score | label | null min | null mean | null max | p_val | adj_p_val_fdr_bh | adj_p_val_fdr_tsbh | null z-score min | null z-score mean | null z-score max | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 12.0 | 3.655953 | 4.425798 | 4.425798 | (A,B) | 0.0 | 3.6680 | 12.0 | 0.000300 | 0.001500 | 0.001200 | -1.948839 | 0.029239 | 3.897770 |
| 5 | 5 | 4.0 | 0.974921 | 2.998022 | 2.998022 | (B,M) | 0.0 | 0.9903 | 7.0 | 0.020398 | 0.050995 | 0.040796 | -0.986555 | 0.034628 | 5.997437 |
| 0 | 0 | 5.0 | 1.468910 | 2.954300 | 2.954300 | (A,A) | 0.0 | 1.4643 | 7.0 | 0.015498 | 0.050995 | 0.040796 | -1.223668 | -0.009408 | 3.792777 |
| 4 | 4 | 4.0 | 2.274815 | 1.160519 | 1.160519 | (B,B) | 0.0 | 2.2709 | 10.0 | 0.193481 | 0.280258 | 0.224206 | -1.524411 | 0.012854 | 3.848234 |
| 9 | 9 | 85.0 | 79.474185 | 0.934716 | 0.934716 | (R,R) | 58.0 | 79.4696 | 104.0 | 0.196180 | 0.280258 | 0.224206 | -2.954047 | 0.003111 | 3.302881 |
In [10]:
Copied!
df_melt_subset.head()
df_melt_subset.head()
Out[10]:
| subtree_val | observed | label | |
|---|---|---|---|
| 10000 | 1 | 2.0 | (A,B) |
| 10001 | 1 | 1.0 | (A,B) |
| 10002 | 1 | 2.0 | (A,B) |
| 10003 | 1 | 3.0 | (A,B) |
| 10004 | 1 | 3.0 | (A,B) |
In [11]:
Copied!
df_melt_100resamples_subset.head()
df_melt_100resamples_subset.head()
Out[11]:
| subtree_val | observed | label | |
|---|---|---|---|
| 100 | 1 | 2.0 | (A,B) |
| 101 | 1 | 1.0 | (A,B) |
| 102 | 1 | 2.0 | (A,B) |
| 103 | 1 | 3.0 | (A,B) |
| 104 | 1 | 3.0 | (A,B) |
In [12]:
Copied!
df_null_zscores_i_c_melt_subset.head()
df_null_zscores_i_c_melt_subset.head()
Out[12]:
| subtree_val | observed | label | |
|---|---|---|---|
| 1000 | 1 | -0.886089 | (A,B) |
| 1001 | 1 | -1.417404 | (A,B) |
| 1002 | 1 | -0.886089 | (A,B) |
| 1003 | 1 | -0.354849 | (A,B) |
| 1004 | 1 | -0.354849 | (A,B) |
In [13]:
Copied!
df_null_zscores_i_c_melt_100resamples_subset.head()
df_null_zscores_i_c_melt_100resamples_subset.head()
Out[13]:
| subtree_val | observed | label | |
|---|---|---|---|
| 100 | 1 | -0.886089 | (A,B) |
| 101 | 1 | -1.417404 | (A,B) |
| 102 | 1 | -0.886089 | (A,B) |
| 103 | 1 | -0.354849 | (A,B) |
| 104 | 1 | -0.354849 | (A,B) |
We will specify a dictionary of cell fates with assigned colors for plotting purposes.
In [14]:
Copied!
cell_fates
cell_fates
Out[14]:
['A', 'B', 'M', 'R']
In [15]:
Copied!
cell_color_dict = plot.make_color_dict(cell_fates, ['#F89A3A',
'#9C80B8',
'#F071AB',
'#7EC352',
])
cell_color_dict = plot.make_color_dict(cell_fates, ['#F89A3A',
'#9C80B8',
'#F071AB',
'#7EC352',
])
Lastly, we will plot the frequency and deviation plot using the DataFrame outputs from dfs_for_plotting and cell color dictionary.
In [16]:
Copied!
plot.plot_frequency('doublet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=None,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_frequency('doublet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=None,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
In [18]:
Copied!
plot.plot_deviation('doublet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=None,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_deviation('doublet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=None,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
A similar analysis can be done for triplet and quartet motifs, as shown below.
Triplet motif analysis¶
In [47]:
Copied!
(triplet_dict,
cell_fates,
dfs_c) = resample.resample_trees_triplets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
(triplet_dict,
cell_fates,
dfs_c) = resample.resample_trees_triplets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
100%|██████████| 10000/10000 [00:16<00:00, 589.90it/s] 100%|██████████| 8/8 [00:00<00:00, 441.51it/s] 0it [00:00, ?it/s] 100%|██████████| 2/2 [00:00<00:00, 4192.21it/s]
In [48]:
Copied!
(triplet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
triplet_dict,
cutoff=15,
num_null=1000,
use_expected=True,
min_cell_types=1)
(triplet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
triplet_dict,
cutoff=15,
num_null=1000,
use_expected=True,
min_cell_types=1)
100%|██████████| 40/40 [00:00<00:00, 909.89it/s] 100%|██████████| 1000/1000 [00:47<00:00, 21.07it/s]
In [49]:
Copied!
plot.plot_frequency('triplet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_frequency('triplet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
In [51]:
Copied!
plot.plot_deviation('triplet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_deviation('triplet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
Quartet motif analysis¶
In [54]:
Copied!
(quartet_dict,
cell_fates,
dfs_c) = resample.resample_trees_quartets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
(quartet_dict,
cell_fates,
dfs_c) = resample.resample_trees_quartets(all_trees_sorted,
num_resamples=10000,
replacement_bool=True,
cell_fates='auto',
calc_expected=True
)
100%|██████████| 10000/10000 [00:14<00:00, 673.70it/s] 100%|██████████| 19/19 [00:00<00:00, 555.05it/s] 100%|██████████| 2/2 [00:00<00:00, 4082.05it/s]
In [55]:
Copied!
(quartet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
quartet_dict,
cutoff=15,
num_null=1000,
use_expected=True,
min_cell_types=1)
(quartet_dict,
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset) = plot.dfs_for_plotting(dfs_c,
10000,
quartet_dict,
cutoff=15,
num_null=1000,
use_expected=True,
min_cell_types=1)
100%|██████████| 55/55 [00:00<00:00, 856.57it/s] 100%|██████████| 1000/1000 [01:12<00:00, 13.76it/s]
In [56]:
Copied!
plot.plot_frequency('quartet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_frequency('quartet',
df_true_melt_subset,
df_melt_subset,
df_melt_100resamples_subset,
cell_color_dict,
use_expected=True,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
In [57]:
Copied!
plot.plot_deviation('quartet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
plot.plot_deviation('quartet',
df_true_melt_subset,
df_null_zscores_i_c_melt_subset,
df_null_zscores_i_c_melt_100resamples_subset,
cell_color_dict,
fdr_type='fdr_tsbh',
cutoff=15,
title='auto',
multiple_datasets=False,
legend_bool=True,
legend_pos='outside',
save=False,
image_format='png',
dpi=300,
image_save_path=None)
