Ascher Friedman’s Data Science Blog

Cross Validation Strategies

2022-10-30T00:00:00+00:00

High Level Idea

During my career as a Data Scientist, I’ve learned there are a multitude of strategies for cross-validation, and that each have their time and place. Pictured below is an overview of what we’ll be discussing in this blog post.

If it looks a bit confusing, don’t worry! At the top of the graphic we have the benefits of the various strategies, and the bottom we have the situations that would lead you to lean toward various strategies. Upon explaining the various strategies it will serve as a great high-level overview of the pros, cons and decision criteria that should be considered when choosing a cross validation strategy. We will explain all of the CV strategies, and then discuss a few use cases and why you would use one over the other.

High Level Idea
CV Methods
Use cases:
Closing Thoughts

CV Methods

Train-Test and Train-Test-Valid Split

Train-Test

The most simple cross-validation strategy, one portion of the data is used for training and selecting hyper-parameters, and a portion is used to validate the performance of the model.

Pros:

The most computationally cheap strategy
(Great for large datasets and computationally expensive models)

Cons:

Prone to Bias
Bad for small datasets with high dimensionality

Train-Test-Valid

The second most simple cross-validation strategy, again we use the train and valid iteratively to select the best hyper-parameters. Here we have an additional test set, to see the performance of a model that hasn’t seen this data in any portion of the modeling process. Generally a better option than Train-Test due to less bias, but with slightly less data for training and validating.

K-fold CV

A middle of the road approach concerning the techniques referenced here. We pick a k, (5 in this example, 10 is more common), and run the whole process k times, using a different 20% (1/k) of the data as a validation set each time. Then we combine these 5 parts, and we have an estimate for the performance of the model, using every datapoint instead of just the Valid portion. This is now 5 times more computationally expensive than our previous methods but gives us a much larger sample of data to validate our results on.

Pros:

Less biased results than a regular Train-Test split.

Cons:

Somewhat computationally expensive (k* more expensive).

Robust K-fold CV

We use 80% of the data for k-fold cross validation, and save 20% as a test set that the process has not seen yet. Similar to K-fold, but with slightly less data available for the K-fold process and slightly less bias.

Nested K-fold CV

Here we repeat this entire K-fold process 5 times (for k=5), so that we have a test set score independent of the K-fold procedure for every data point.

Pros:

Extremely unbiased estimate

Cons:

K times more expensive than K-fold, k^2 more computationally expensive than Train-Test.
Is unrealistic for computationally expensive applications as a training loop may take weeks.
In the given time we can train less hyper-parameters.

Use cases:

Train-Test-Valid (Or Train-Test)

Situation: We are doing image classification with a massive dataset with millions of rows.
Explanation: The model is computationally expensive making K-fold unrealistically long to compute results, and due to the dataset size, the chances of bias due to a bad test split is extremely low. I prefer Train-Test_Valid

Nested K-Fold

Situation: We have a small tabular dataset, with large dimensionality, and extremely high business impact for each decision.
Explanation: Small tabular dataset means the modeling computational cost is low and Nested K-Fold will still be light on computing power even with a computationally expensive CV procedure. High business impact means we want as accurate an estimation of the performance as possible.

Robust K-Fold (Or K-Fold)

Situation: We have a medium sized dataset, and still want to be careful to avoid bias and accurately represent performance. (Middle of the road)
Explanation: A dataset of a few hundred thousand rows being trained with a K-Fold CV will not be unreasonably computationally expensive. The results are still very robust and unbiased. Good middle of the road solution. I prefer Robust K-Fold generally.

Closing Thoughts

I took my time putting this together as I feel its a nuanced discussion that doesnt always have perfect answers. Some of your choices will come down to practical considerations, and some down to how you personally weight the pros and cons of each strategy. This was by no means an exuastive list of CV strategies or factors that can influence these decisions, but I hope its serves as a concise high level view of the topic.

NLP Series Part 1, Shapley and Interpretability is a must!

2022-09-27T00:00:00+00:00

Disclaimer, for the sake of keeping these posts relatively short, I will not be conducting robust cross-validation, or using more advanced NLP techniques. This is purely to show a simple NLP multi-class classification task, and how we can interpret the results.

High Level Idea

Trying to convince your stakeholder that your model is a black box of magic that they can simply trust is, understandably, a hard sell. Shapley values are an absolute must and in my professional experience have been responsible for major buy in to a modeling effort.

Were going to do multi-class text classification, with 4 classes, show how to do summary plots, interpret an example for each of the 4 categories, and then show how an ambiguous sample is interpreted by all 4 classes.

To run this code you should use the requirements.txt as the versions of some packages are extremely important, and copy the model folder with the torch dense neural net implementation

Dataset: https://www.kaggle.com/datasets/tunguz/200000-jeopardy-questions

Code to run this yourself: https://github.com/AscherFriedman/AscherFriedman.github.io/tree/master/NLP_P1_Code

Includes modules that are in other directories and imports of packages

#Lets grab a csv containing historical information from the show jeapordy. Each Question belongs to a Category.
df = pd.read_csv('JEOPARDY_CSV.csv')
#Some of the columns come with odd names so lets strip white spaces before and after
df.rename(columns = {x:x.lstrip().rstrip() for x in df.columns}, inplace=True) 
df = df[['Question','Category']]
df.Category.value_counts().head(10)

BEFORE & AFTER             547
SCIENCE                    519
LITERATURE                 496
AMERICAN HISTORY           418
POTPOURRI                  401
WORLD HISTORY              377
WORD ORIGINS               371
COLLEGES & UNIVERSITIES    351
HISTORY                    349
SPORTS                     342
Name: Category, dtype: int64

As we can see there are many many categories and they are all relatively small.

Lets create 4 categories by combining others: History, Literature, Art, and Geography

#rename_cols = 
df.rename(columns = {x:x.lstrip().rstrip() for x in df.columns}, inplace=True) #EDA found spaces before and after column names
#Lets make some categories with 1500 observations each
df.Category.replace({'WORLD HISTORY':'HISTORY','AMERICAN HISTORY':'HISTORY','U.S. HISTORY':'HISTORY','THE CIVIL WAR':'HISTORY'},inplace=True)
df.Category.replace({'U.S. GEOGRAPHY':'GEOGRAPHY','WORLD GEOGRAPHY':'GEOGRAPHY',
                    'BODIES OF WATER':'GEOGRAPHY','ISLANDS':'GEOGRAPHY'},inplace=True)
df.Category.replace({'BOOKS & AUTHORS':'LITERATURE','SHAKESPEARE':'LITERATURE','FICTIONAL CHARACTERS':'LITERATURE','AMERICAN LITERATURE':'LITERATURE'},inplace=True)
df.Category.replace({'U.S. CITIES':'CITY_COUNTRY','STATE CAPITALS':'CITY_COUNTRY','WORLD CAPITALS':'CITY_COUNTRY'},inplace=True)
df.Category.replace({'ART & ARTISTS':'ART','BALLET':'ART','OPERA':'ART','MUSIC':'ART',
                    'CLASSICAL MUSIC':'ART'},inplace=True)
df = df[df.Category.isin(['HISTORY', 'LITERATURE', 'ART', 'GEOGRAPHY'])]
df.Category.value_counts()

HISTORY       1600
LITERATURE    1559
ART           1542
GEOGRAPHY     1507
Name: Category, dtype: int64

Lemmatizing and Speed

For a larger dataset I would suggest using the package pandarallel or using multiprocessing with an entire column at once as lemmatizing is known to be a slow process on large datasets. Mostly because it involves repeatedly using lookup tables for every word. Given this dataset is under 10,000 rows, lemmatizing is still quite fast.

stop_words = set(stopwords.words('english'))
wnl = WordNetLemmatizer()
# Two functions to do some basic cleaning, stripping odd characters and numbers, leaving alpha numeric.
def basic_clean(text):
    "Strips all non-alpha numeric text"
    # Regex pattern for only alphanumeric, hyphenated text
    return(re.sub(r'[^a-zA-Z -]', '', text))

def pre_processing(text):
    '''
    Tokenizes, Lemmatizes, Lower Cases, Removes Stopwords
    Joins back to Sentance'''
    lemma_list = ' '.join([wnl.lemmatize(x.lower()) for x in word_tokenize(text) if x.lower() not in stop_words])
    return lemma_list

df['clean'] = df['Question'].apply(basic_clean).apply(pre_processing)

Split Data

In another tutorial i’ll show how to better analyse results from a multi-class classification

#Label encoder for the the response variable
#Lets make the train and valid each 15% of the dataset
le = LabelEncoder().fit(df['Category'])
df['encoded'] = le.transform(df['Category'])
X_train, X_test, y_train, y_test = train_test_split(df['clean'], df['encoded'], test_size=.2, random_state=42,stratify=df['encoded'])

Very basic NLP algorithm

Convert to a tfidf matrix with 100 features, feed it into a 128,64,32 dense neural net with 4 classes using torch Then

#Vectorize to a tf-idf matrix max_features = 300 X_test_original = X_test.copy() #We save this tfidf matrix for the intepretation portion of this analysis tfidf = TfidfVectorizer(max_features=max_features) X_train = tfidf.fit_transform(X_train) X_test = tfidf.transform(X_test) feature_names = tfidf.get_feature_names_out ()

X_train = torch.tensor(X_train.todense()).to(device).float()
X_test = torch.tensor(X_test.todense()).to(device).float()

y_train = torch.tensor(y_train.values).to(device).float()
y_test = torch.tensor(y_test.values).to(device).float()

train_dataset = TFIDFDataset(X_train,y_train)
test_dataset = TFIDFDataset(X_test,y_test)

train_loader = DataLoader(dataset = train_dataset, batch_size = 32)
test_loader = DataLoader(dataset = test_dataset, batch_size = 16)

dataiter = iter(train_loader)
train_feat_ex, train_label_ex = dataiter.next()

model = dense_model(hidden_layers= [128,64,43], num_features = max_features, num_classes = len(le.classes_), p_drop=.15)

Again, this is a fairly small dataset and a non-robust cross validation procedure so these results are to be taken with a grain of salt, the point here is to highlight interpretation techniques, not performance

model, stats = train_nn(model, train_loader, test_loader, NUM_EPOCHS = 50, LR = .0001, LR_patience=5, early_stopping_callback= EarlyStoppingCallback())

  2%|█▋                                                                                 | 1/50 [00:00<00:43,  1.11it/s]

EPOCH 001: Loss: [Train: 1.36238 | test 1.24256 ] Accuracy: [Train: 31.964 | test 49.038] LR: 0.0001

  4%|███▎                                                                               | 2/50 [00:01<00:45,  1.05it/s]

EPOCH 002: Loss: [Train: 1.19640 | test 1.09368 ] Accuracy: [Train: 49.546 | test 64.103] LR: 0.0001

  6%|████▉                                                                              | 3/50 [00:02<00:43,  1.07it/s]

EPOCH 003: Loss: [Train: 1.04410 | test 0.94591 ] Accuracy: [Train: 61.679 | test 71.554] LR: 0.0001

  8%|██████▋                                                                            | 4/50 [00:03<00:43,  1.06it/s]

EPOCH 004: Loss: [Train: 0.92014 | test 0.84137 ] Accuracy: [Train: 68.717 | test 75.689] LR: 0.0001

 10%|████████▎                                                                          | 5/50 [00:04<00:41,  1.08it/s]

Message from callback (Early Stopping) counter: 1/5
no
EPOCH 005: Loss: [Train: 0.81978 | test 0.75311 ] Accuracy: [Train: 72.583 | test 77.420] LR: 0.0001

 12%|█████████▉                                                                         | 6/50 [00:05<00:40,  1.09it/s]

EPOCH 006: Loss: [Train: 0.72940 | test 0.68310 ] Accuracy: [Train: 75.935 | test 78.141] LR: 0.0001

 14%|███████████▌                                                                       | 7/50 [00:06<00:38,  1.11it/s]

Message from callback (Early Stopping) counter: 1/5
no
EPOCH 007: Loss: [Train: 0.66856 | test 0.63342 ] Accuracy: [Train: 77.958 | test 78.622] LR: 0.0001

 16%|█████████████▎                                                                     | 8/50 [00:07<00:37,  1.12it/s]

Message from callback (Early Stopping) counter: 2/5
no
EPOCH 008: Loss: [Train: 0.62259 | test 0.63164 ] Accuracy: [Train: 80.195 | test 78.990] LR: 1e-05

 18%|██████████████▉                                                                    | 9/50 [00:08<00:36,  1.12it/s]

Message from callback (Early Stopping) counter: 3/5
no
EPOCH 009: Loss: [Train: 0.61857 | test 0.62941 ] Accuracy: [Train: 80.442 | test 78.429] LR: 1e-05

 20%|████████████████▍                                                                 | 10/50 [00:09<00:36,  1.11it/s]

Message from callback (Early Stopping) counter: 4/5
no
EPOCH 010: Loss: [Train: 0.60936 | test 0.62494 ] Accuracy: [Train: 80.863 | test 79.151] LR: 1e-05

 20%|████████████████▍                                                                 | 10/50 [00:10<00:40,  1.00s/it]

Message from callback (Early Stopping) counter: 5/5
here
Training stopped -> Eearly Stopping Callback : test_loss: 0.6211988784563847

For the Shap package, deep explainer generally works fairly quick to nn.torch problems that arnt image classification

explainer = shap.DeepExplainer(model, shap.sample(X_test,25))
# Get the shap values from my test data
shap_values = explainer.shap_values(X_test)

%matplotlib inline

Functions to generate plots

Worth noting that the title option on shap plots is buggy, so we turn it off, add it manually matplotlib

def summary(feature):
    encoded = le.transform([feature])[0]
    shap.summary_plot(shap_values[encoded],X_test.cpu(), feature_names=feature_names, class_names=le.classes_,show=False)
    plt.title(feature,fontsize=16)
    plt.savefig(feature,bbox_inches='tight')
    plt.close()
    
def specific_example(feature,threshold,sample_num, save_fig = False):
    encoded = le.transform([feature])[0]
    sample_index = np.where(y_test.cpu().numpy()==encoded)[0][sample_num]
    shap_vals = shap_values[encoded][sample_index]
    use = shap_vals*[abs(shap_vals)>threshold][0]
    shap.bar_plot(use,feature_names=feature_names,max_display=(abs(use)>0).sum(),show=False)
    sentence  = df.loc[X_test_original.index[sample_index],'Question']
    plt.title(f'{feature} sample: "{sentence}"')
    if save_fig:
        plt.savefig(feature+'_barplot',bbox_inches='tight')
    plt.close()

Create Summary plot for all 4 classes

for feature in le.classes_[:]:
    summary(feature)

Note that in these shaply plots a word either exists or it doesnt, so blue just means no word, and red means words are in the sentance we are plotting.

ART

Here we can see that words such as “Ballet”, “Artist”, “Painter” and “Opera” tell the model that the question belongs to category ART. If it sees “Island” or “Novel”, it is a good indication that it is not art.

Geography

“Island”, “Gulf”, “Lake” and “Mile” tell the model that the question belongs to category Geography. If it sees “Wrote” or again “Novel”, it is a good indication that it is not geography.

History

History is an interesting case because the model is primarily defined by which words are not present. So if it sees “Novel” or “Artist” or “Wrote”, it knows its not history. Some words are still positive Indicators, like “War”.

Literature

“Novel”, “Wrote”, “Published” and “Book” tell the model that the question belongs to category Literature. If it sees “island” or “Country”, it is a good indication that it is not literature.

“Novel” ends up being important because theres a category that we combined into Literature called “Novels” so each question in that subcategory had the word Novel in it.

Picked some specific examples that are interesting to look at

specific_example('ART',.1,3, save_fig=True)
specific_example('HISTORY', .1, 1, save_fig=True)
specific_example('GEOGRAPHY', .1, 5, save_fig=True)
specific_example('LITERATURE', .1, 9, save_fig=True)

ART

“Opera” and “Composer” indicate its ART, but “California” and “West” indicate its not art! Still we can see the positives are much larger than the negatives and the model still likely choses art.

Geography

“Lake and “Part” tell the model its geography, but “Home” is a small hint it might not be (home probably belongs frequently to literature and history.

History

“Treaty” and “Country” tell the model it is likely History, but “French” says it may not be, probably because of many French Artists.

Literature

“Shakespear”, “Comedy”, “Love”, all hallmarks of literature, but so many literature questions contained the word “Novel” that the lack of this word makes the model uncertain

Lets Pick an interesting and very confliting case and see how the model evaluates the likelyhood of each category.

“Tradition says this country’s first emperor, Menelik 1, was the son of King Solomon and the Queen of Sheba” This is a history question.

for feature in le.classes_:
    encoded = le.transform(['HISTORY'])[0]
    sample_index = np.where(y_test.cpu().numpy() == encoded)[0][32]
    shap_vals = shap_values[le.transform([feature])[0]][sample_index]
    use = shap_vals*[abs(shap_vals) > .05][0]
    shap.bar_plot(use,feature_names = feature_names,max_display = (abs(use) > 0).sum(),show = False)
    sentence  = df.loc[X_test_original.index[sample_index],'Question']
    plt.title(f'{feature} sample: "{sentence}"')
    plt.savefig(feature+'_one_sample_mult_feat_',bbox_inches='tight')
    plt.show()

ART

The presence of words like “Country” and “Say” and “First”, as well as the absence of words like “Artist” and “Painter” all indicate this is not ART. Art is ruled out!

Geography

Geography sees words like country that give it some confidence, but “Say” and “King” are enough to know its not Geography. Geography is ruled out!

History

Here the model sees “King” and “Country” giving it confidence, but also “Say”, “First” and “Son” which makes it doubtful. Say usually belongs to literature, as does Son.

Literature

Here the model sees “Say”, “First” and “Son”, which are frequently found in literature, but it still has doubts given that it sees “Country” and “Emperor” and no “Novel”

print('True Label:',df.loc[X_test_original.index[sample_index]].Category)
probs = np.array(softmax(model(X_test[sample_index:sample_index+1]),dim=1).detach().cpu())[0]
print()
for category,prob in zip(le.classes_,probs):
    print(f'{round(prob*100,2)}% chance of being {category}')
print()
print('Modle Predicts: ', le.classes_[probs.argmax()])

True Label: HISTORY

14.71% chance of being ART
17.11% chance of being GEOGRAPHY
33.44% chance of being HISTORY
34.74% chance of being LITERATURE

Modle Predicts:  LITERATURE

The model is just barely wrong! With a little hyperparameter tuning, or simple tricks like adding in bi-grams, i’m sure it can figure it out!

Python for Bigquerey and SQL

2022-09-19T00:00:00+00:00

import pandas as pd
import numpy as np
import os
from google.cloud import bigquery

JSON_PATH = "path to JSON FILE from Google"
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]=JSON_PATH
client = bigquery.Client()

High Level Idea

Pulling very specific GA4 data takes a bit of SQL knowledge, and intricate queries.

Writing these yourself is a headache, but with some python knowledge, its actually fairly easy.

Getting GA4 Running

Guide to set up the GA4 API https://developers.google.com/analytics/devguides/reporting/core/v4/quickstart/service-py

Explore a free GA4 dataset https://developers.google.com/analytics/bigquery/web-ecommerce-demo-dataset

# Lets grab a sample of the data, and just a few columns for simplicity sake

q = """
SELECT event_timestamp, event_name, event_params                                                                
FROM  `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20201101' AND '20210131'
LIMIT 15
;
"""
sample_df = client.query(q).to_dataframe()
#grab one sample
sample_df.head(1)

	event_timestamp	event_name	event_params
0	1605594151663973	page_view	[{'key': 'page_referrer', 'value': {'string_va...

This data is filled with nested data here which is an issue if we want to do data-science or analysis

What if we want specific value inside “event_params” as its own column, which is currently nested inside event_params?

sample_df.iloc[0]['event_params'][:5]

array([{'key': 'ga_session_number', 'value': {'string_value': None, 'int_value': 2, 'float_value': None, 'double_value': None}},
       {'key': 'ga_session_id', 'value': {'string_value': None, 'int_value': 4389558959, 'float_value': None, 'double_value': None}},
       {'key': 'clean_event', 'value': {'string_value': 'gtm.js', 'int_value': None, 'float_value': None, 'double_value': None}},
       {'key': 'page_location', 'value': {'string_value': 'https://shop.googlemerchandisestore.com/Google+Redesign/Apparel/Google+Dino+Game+Tee', 'int_value': None, 'float_value': None, 'double_value': None}},
       {'key': 'engagement_time_msec', 'value': {'string_value': None, 'int_value': 40560, 'float_value': None, 'double_value': None}}],
      dtype=object)

Cross Join

This trick is given by google as its a common issue

Theoretically this is how it works, we join the main table every row of the nested table. This means for this one observation there would now be 5, one with each key Then we simply tell that we only want records with the key equal to ga_session_number

#It looks like this
querey = '''
SELECT
  event_timestamp,event_name, value.int_value as ga_session_id

FROM
    `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` AS T
    CROSS JOIN
      T.event_params

WHERE
  event_params.key = 'ga_session_id'
  AND _TABLE_SUFFIX BETWEEN '20201101' AND '20210131'
  LIMIT 500
  ;
'''
#Now lets look at the results
df_sample = client.query(querey).to_dataframe()
df_sample.head(10)

	event_timestamp	event_name	ga_session_id
0	1609079909474813	first_visit	6267689519
1	1609079913730045	user_engagement	6267689519
2	1609079909474813	session_start	6267689519
3	1609079909474813	page_view	6267689519
4	1609045476565920	first_visit	1433811680
5	1609045476565920	page_view	1433811680
6	1609046700919403	scroll	1433811680
7	1609045476565920	session_start	1433811680
8	1609031351701297	first_visit	6998346161
9	1609031351701297	page_view	6998346161

Look how nice this looks! We have pulled out a nested subclause as a column Naturally the next question becomes, what about pulling out multiple of these nested columns?

Theoretically we just want to keep making new tables and join them to our previous tables resulting in multiple exploded columns all in one row

Lets turn this into two functions, one for generating a single querey for exploding one column out, and another for joining that to another table!

def single_querey(start_date,end_date,val_type,field,sub_field):
    start_date = '\''+start_date+ '\''
    end_date = '\''+end_date+ '\''
    querey= """
    SELECT
      event_timestamp,event_name, value.{2}_value as {4}
    FROM

        `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` AS T
        CROSS JOIN
          T.{3}
    WHERE
      {3}.key = '{4}'
      AND _TABLE_SUFFIX BETWEEN {0} AND {1};
    """.format(start_date,end_date,val_type,field,sub_field)
    #print(querey)
    return(querey)

def join_queries(q1,q2,joined_col='T'):
    q1=q1.replace(';','')
    q2=q2.replace(';','')
    q_new = """
    SELECT T.*,{2}.{2}
     FROM 
     ( {0} ) AS T 
     LEFT JOIN 
     ( {1} ) AS {2} 
     ON T.event_timestamp = {2}.event_timestamp
     AND
     T.event_name = {2}.event_name;
    """.format(q1,q2,joined_col)
    return(q_new)

#Now lets use these functions to grab 3 exploded out columns, page_location, page_title and ga_session_id

start_date = '20201101'
end_date =  '20201202'
q1 = single_querey(start_date,end_date,'int','event_params','ga_session_id')
q2 = single_querey(start_date,end_date,'string','event_params','page_title')
q3 = single_querey(start_date,end_date,'string','event_params','page_location')
q_new = join_queries(q1,q2,joined_col = 'page_title')
q_new = join_queries(q_new,q3,joined_col = 'page_location')

%%time
df = client.query(q_new).to_dataframe()
df.head()

CPU times: total: 2.42 s
Wall time: 18.7 s

	event_timestamp	event_name	ga_session_id	page_title	page_location
0	1605498662409146	view_item	2200271228	Google Mural Socks	https://shop.googlemerchandisestore.com/Google...
1	1605497966809821	user_engagement	2200271228	#IamRemarkable \| Shop by Brand \| Google Mercha...	https://shop.googlemerchandisestore.com/Google...
2	1605498605857861	user_engagement	2200271228	Google Men's Puff Jacket Black	https://shop.googlemerchandisestore.com/Google...
3	1605489713558309	session_start	970858984	Water Bottles \| Drinkware \| Google Merchandise...	https://shop.googlemerchandisestore.com/Google...
4	1605535584700554	user_engagement	7103403152	Google Sherpa Zip Hoodie Charcoal	https://shop.googlemerchandisestore.com/Google...

Here we are able to run a very complex querey that would be a serious headache to write, automatically using python, and a pull a full month of data in less than half a minute.

#Heres what that querey looks like
print(q_new)

    SELECT T.*,page_location.page_location
     FROM 
     ( 
    SELECT T.*,page_title.page_title
     FROM 
     ( 
    SELECT
      event_timestamp,event_name, value.int_value as ga_session_id
    FROM

        `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` AS T
        CROSS JOIN
          T.event_params
    WHERE
      event_params.key = 'ga_session_id'
      AND _TABLE_SUFFIX BETWEEN '20201101' AND '20201202'
     ) AS T 
     LEFT JOIN 
     ( 
    SELECT
      event_timestamp,event_name, value.string_value as page_title
    FROM

        `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` AS T
        CROSS JOIN
          T.event_params
    WHERE
      event_params.key = 'page_title'
      AND _TABLE_SUFFIX BETWEEN '20201101' AND '20201202'
     ) AS page_title 
     ON T.event_timestamp = page_title.event_timestamp
     AND
     T.event_name = page_title.event_name
     ) AS T 
     LEFT JOIN 
     ( 
    SELECT
      event_timestamp,event_name, value.string_value as page_location
    FROM

        `bigquery-public-data.ga4_obfuscated_sample_ecommerce.events_*` AS T
        CROSS JOIN
          T.event_params
    WHERE
      event_params.key = 'page_location'
      AND _TABLE_SUFFIX BETWEEN '20201101' AND '20201202'
     ) AS page_location 
     ON T.event_timestamp = page_location.event_timestamp
     AND
     T.event_name = page_location.event_name;

Random Forests, An Artifact of good timing?

2022-08-23T00:00:00+00:00

From early on in my datascience career(2017) I have been a big fan and advocate of tree-based algorithms. The flexibility of input and target variable distributions, paired with astounding results made me come back to them again and again for a variety of problems. Everything from classification to NLP they are a great tool to get your project going.

The first algorithms in my data-science journey were random forests and elastic-net regression, then eventually GBMs, and then XGBoost, CatBoost etc.. Without fail the GBMs and eventually XGBoost would outperform the random forests and I repeatedly assumed that this individual problem just fit boosted algorithms better. However, after enough experience, I strated to think that perhaps random forests were simply an inferior algorithm and I wondered why they are so dominant in the field of datascience? Well perhaps its simply because they were around first.

The first algorithm for random decision forests was created in 1995 by Tin Kam Ho. The first GBMs proposed by Jerome Friedman, in his seminal paper in 1999 (updated in 2001).

Analytics

Is there any other evidence we can garner for this assertion? Google trends (take them with a grain of salt) lets us see there was a time when GBMs were almost non existent in searches while Random Forests were being searched quite regularly, and with time, for good reason, this is changing.

Pseudo Stratification for Skewed Continuous Response Variables

2021-10-10T00:00:00+00:00

High Level Idea

Most competent data scientists understand that if you have a categorical response variable that is extremely unbalanced, you should stratify your train/test split. This ensures that your test and train set are both representative of the overall dataset.

But what happens when you have a continuous response variable that is extremely skewed? Well in short you have the same problem, and without this trick, its quite likely you’ll generates a test set not representative of the overall dataset. To be more robust we could use Kolmogorov-Smirnov tests, but for this we will simply compare means of the distributions, as this is often an important metric in practice.

High Level Idea
Generate an uneven distribution and plot it
Evaluation of Train/Test Split Performance
Solution, Pseudo-Stratified Splitting!
Test Various Splits
Four Test Cases
Extreme case for stratified sampling importance
Medium Importance of Stratified Sampling 1
Medium Importance of Stratified Sampling 2
Low Importance of Stratified Sampling
Results

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import pandas as pd
from sklearn.model_selection import train_test_split

Generate an uneven distribution and plot it

This function generates a gamma distribution, gamma_shape being how skewed it is, and nsamples being how large our dataset is. Gamma is a perfect distribution for this, because very skewed datasets are often estimated quite well by gamma distributions. One example where I encountered this was insurance claims for vehicles, the vast majority are small issues like windshield damage, but can also be catostrophically expensive accidents.

def generate_distribution_and_plot(gamma_shape,gamma_scale,n_samples):    
    gamma_target=np.random.gamma(gamma_shape,gamma_scale,n_samples)
    X=np.random.rand(len(gamma_target))
    plt.figure(figsize=(8,5))
    plt.hist(gamma_target,bins=round(len(gamma_target)**.5))
    plt.xlabel('Sample Values',fontsize=12)
    plt.ylabel('Samples Per Bin',fontsize=12)
    plt.title('Gamma Distribution',fontsize=14)
    plt.show()
    return pd.DataFrame(data={'y':gamma_target,'X':X})

## EXAMPLE
df_example=generate_distribution_and_plot(.5,1,500)

Evaluation of Train/Test Split Performance

“grab_means_aftersplit” randomly splits the dataset into train and test splits, calculates their means, and returns their distance to the overall mean, as a percentage relative to the overall mean.

“Get_avg_diff” repeats this process 50 times, to give us an estimation of how far on average our train, and test sets are from the distribution mean. Note that as test is 20% and train is 80% of the dataset, test will be much further off on average.

Ignore the stratify argument for now.

def grab_means_aftersplit(df,stratify=None):
    if stratify==None:
        X_train, X_test, y_train, y_test = train_test_split(df['X'],df['y'],test_size=.2,stratify=None)
    else:
        X_train, X_test, y_train, y_test = train_test_split(df['X'],df['y'],test_size=.2,stratify=df[stratify])
    actual_mean=df['y'].mean()
    train_diff=abs(round(100-y_train.mean()/actual_mean*100,1))
    test_diff=abs(round(100-y_test.mean()/actual_mean*100,1))
    return train_diff,test_diff

def get_avg_diff(df,pseudo_stratify=None,verbose=False):
    train_diffs,test_diffs=[],[]
    for trial in range(50):
        train_diff,test_diff=grab_means_aftersplit(df,stratify=pseudo_stratify)
        train_diffs.append(train_diff)
        test_diffs.append(test_diff)
    train_mean_dev=round(np.mean(train_diffs),1)
    test_mean_dev=round(np.mean(test_diffs),1)
    if verbose==True:
        print('Absolute % Deviation From Mean (50 trials)')
        print('Train',train_mean_dev)
        print('Test',test_mean_dev)
    return train_mean_dev,test_mean_dev

##Example 
train_mean,test_mean=get_avg_diff(df_example,None,verbose=True)

Absolute % Deviation From Mean (50 trials)
Train 2.5
Test 10.0

Solution, Pseudo-Stratified Splitting!

The solution is quite simple in theory, we divide the dataset, in ascending order, into n_splits, (in this example 10), assign each sample a split group and then stratify our sampling based on this new column.

We can see on average, the stratified split is much closer to the overall distribution.

def add_splits(df_input,n_splits,split_name,verbose=False):
    sorted_indices=np.argsort(df_input.y) #This gives us list of indices that correspond to ascending values
    splits=np.array_split(sorted_indices, n_splits) #Now we split this into n sublists 
    #Print the means for every split and put the split number into the dataframe
    for i in range(n_splits):                    
        if verbose==True: #If we want to print intermediate info
            print('Split:',i,'Mean',round(df_input.y[splits[i]].mean(),2)) 
        df_input.loc[splits[i],split_name]=i
    return df_input

##Example
df_example=add_splits(df_example,n_splits=10,split_name='splits',verbose=True)
train_mean,test_mean=get_avg_diff(df_example,'splits',verbose=True)

Split: 0 Mean 0.0
Split: 1 Mean 0.02
Split: 2 Mean 0.05
Split: 3 Mean 0.11
Split: 4 Mean 0.18
Split: 5 Mean 0.28
Split: 6 Mean 0.44
Split: 7 Mean 0.72
Split: 8 Mean 1.13
Split: 9 Mean 2.26
Absolute % Deviation From Mean (50 trials)
Train 1.2
Test 4.9

Test Various Splits

Now it seems this did indeed help, the train and test samples are closer to the mean on average. But how many splits is correct, too few and we dont maximize performance, too many and we overfit with no added performance.

To test this, we feed in an array of number of splits we would like to test, get the average performance for each of them, and plot the results.

A good rule of thumb I’ve found for recommended number of splits, is to cap the samples at 10,000, (so if its 20,000 treat it as 10,000). And then take the square root of the test set. This rule is obviously problematic since its based only on sample size and not on skew which also very much effects the output, but in my experience works alright.

def test_various_splits(df,split_vals,recommended_splits=None):
    test_mean_devs,train_mean_devs=[],[]
    test_std_devs,train_std_devs=[],[]
    for n_splits in split_vals: #go through all splits that arnt 0
        if n_splits==0: 
            print('No Splits:')
            train_dev,test_dev=get_avg_diff(df,pseudo_stratify=None,verbose=True)
        elif n_splits == recommended_splits: 
            print()
            print('Recommended Split',str(recommended_splits)+':')
            train_dev,test_dev=get_avg_diff(df,pseudo_stratify='splits',verbose=True)
        else:
            df=add_splits(df,n_splits,'splits')
            train_dev,test_dev=get_avg_diff(df,pseudo_stratify='splits')
        train_mean_devs.append(train_dev)
        test_mean_devs.append(test_dev)
    plt.figure(figsize=(8,5))
    plt.plot(split_vals,train_mean_devs,label='train_mean_diff')
    plt.plot(split_vals,test_mean_devs,label='test_mean_diff')
    plt.legend()
    plt.ylabel('Absolute Difference from Mean',size=11)
    plt.xlabel('Number of Splits',size=11)
    plt.title('How many splits is too many?',size=14)
    if recommended_splits!=None:
        plt.axvline(recommended_splits,ls='--',color='grey')
        plt.text(recommended_splits+1,np.max(test_mean_devs)*.75,'Recommended Split',rotation=0)

Four Test Cases

Here we test the four combinations of low sample size or medium sample size, with low skew or high skew. Lower sample sizes, and higher skew make stratified sampling more important. Low sample sizes mean more volatility in sampling, and higher skew means more individual values that have disporportionate influence on the distributions mean and variance.
Due to the 50 trials at each split level, this method would be slow at high sample sizes, but it gives us insight into the relationship between split levels and performance, and perhaps the primitively calculated recommended splits generally does the job.

Extreme case for stratified sampling importance

Low samples, high skew

Stratified sampling is very important

#Create a gamma distribution, sample it
n_samples=1500
gamma_shape=.1
gamma_scale=1
print('Samples:',str(n_samples)+',','Gamma Shape:',gamma_shape)
df_samples=generate_distribution_and_plot(gamma_shape,gamma_scale,n_samples)
print()
recommended_splits=round(((n_samples*.2)**.5))
#print('Recommended Split:',recommended_splits)
test_various_splits(df_samples,np.arange(0,50,1),recommended_splits=recommended_splits)

Samples: 1500, Gamma Shape: 0.1

No Splits:
Absolute % Deviation From Mean (50 trials)
Train 4.1
Test 16.2

Recommended Split 17:
Absolute % Deviation From Mean (50 trials)
Train 1.7
Test 6.8

Medium Importance of Stratified Sampling 1

Low sample size, low skew

#Create a gamma distribution, sample it
n_samples=1500
gamma_shape=.5
gamma_scale=1
print('Samples:',str(n_samples)+',','Gamma Shape:',gamma_shape)
df_samples=generate_distribution_and_plot(gamma_shape,gamma_scale,n_samples)
print()
recommended_splits=round(((n_samples*.2)**.5))
test_various_splits(df_samples,np.arange(0,50,1),recommended_splits=recommended_splits)

Samples: 1500, Gamma Shape: 0.5

No Splits:
Absolute % Deviation From Mean (50 trials)
Train 1.5
Test 6.1

Recommended Split 17:
Absolute % Deviation From Mean (50 trials)
Train 0.6
Test 2.2

Medium Importance of Stratified Sampling 2

Medium Sample Size, high skew

#Create a gamma distribution, sample it
n_samples=15000
gamma_shape=.1
gamma_scale=1
print('Samples:',str(n_samples)+',','Gamma Shape:',gamma_shape)
df_samples=generate_distribution_and_plot(gamma_shape,gamma_scale,n_samples)
print()
recommended_splits=round(((np.min([n_samples,10000]))*.2)**.5)
test_various_splits(df_samples,np.arange(0,recommended_splits*3,3),recommended_splits=recommended_splits)

Samples: 15000, Gamma Shape: 0.1

No Splits:
Absolute % Deviation From Mean (50 trials)
Train 1.0
Test 4.2

Recommended Split 45:
Absolute % Deviation From Mean (50 trials)
Train 0.4
Test 1.6

Low Importance of Stratified Sampling

Medium sample size, low skew

#Create a gamma distribution, sample it
n_samples=15000
gamma_shape=.5
gamma_scale=1
print('Samples:',str(n_samples)+',','Gamma Shape:',gamma_shape)
df_samples=generate_distribution_and_plot(gamma_shape,gamma_scale,n_samples)
print()
recommended_splits=round(((np.min([n_samples,10000]))*.2)**.5)
test_various_splits(df_samples,np.arange(0,recommended_splits*3,3),recommended_splits=recommended_splits)

Samples: 15000, Gamma Shape: 0.5

No Splits:
Absolute % Deviation From Mean (50 trials)
Train 0.4
Test 1.8

Recommended Split 45:
Absolute % Deviation From Mean (50 trials)
Train 0.1
Test 0.4

Results

Sample Size	Gamma Shape	N Splits	Train Error No Strat	Test Error No Strat	Train Error With Strat	Test Error With Strat
1500	.1	17	4.1	16.2	1.7	6.8
1500	.5	17	1.5	6.1	0.6	2.2
15000	.1	45	1.0	4.2	0.4	1.6
15000	.5	45	0.4	1.8	0.1	0.4