1 Machine Learning Engineering

Documentação baseada no livro ‘Andriy Burkov - Machine Learning Engineering-True Positive Inc. (2020).pdf’

1.1 Business Problem

1.2 Goal Definition

1.3 Data Collection & Preparation

1.3.1 Data Augmentation

Objetivo: criar mais labeled examples.

1.3.1.1 Imagens

No caso de imagens, o procedimento consiste em aplicar operações simples como rotação, mudanças de contraste e perspectiva, adição de ruído entre outros para obter novas imagens no grupo de treino.
Outra técnica muito utilizada é o mixup. Essa técnica consiste em treinar o modelo numa combinação das imagens no training set.

1.3.1.2 Texto

É importante usar técnicas que preservem o contexto e a estrutura gramatical do texto.
Uma técnica é substituir um termo pelo seu sinônimo mais comum.
Uma técnica similar usa hipernômios no lugar de sinônimos. Ou seja, a palavra é substituída por outra de sentido mais abrangente. Exemplo: baleia -> mamífero.
Alternativamente, uma palavra em uma sentença pode ser substituída pelos seus k vizinhos mais próximos dando origem a k novas sentenças.
Outra maneira de aumentar uma base de textos é usando Back Translation. Essa técnica consiste em traduzir uma sentença para outro idioma e, seguida, fazer a tradução inversa. Se o texto obtido for diferente do original, ele pode ser adicionado à base de dados.

1.3.2 Dealing With Imbalanced Data

Um conjunto de dados é considerado desbalanceado se apresenta uma distribuição muito desigual de labels no training set.
Exemplo: um classificador que tem que distinguir entre transações genuínas e fraudulentas. As genuínas são muito mais frequentes.
Desbalanceamentos leves (60/40) não costumam ser problemáticos. No entanto, quando o desbalanceamento é muito grande (90/10), os algoritimos tradicionais de Machine Learning (que atribuem pesos iguais aos erros de previsão em ambas as classes) podem não funcionar bem.

1.3.2.1 Oversampling

O oversampling consiste em fazer múltiplas cópias dos exemplos da classe minoritária.
Outra possibilidade é criar exemplos sintéticos fazendo combinações dos atributos de outras entradas do banco de dados. As técnicas mais comuns são o SMOTE e o ADASYN.

1.3.2.2 Undersampling

O undersampling consiste em retirar exemplos da classe majoritária do training set.
O undersampling pode ser feito aleatoriamente ou podem se basear em alguma propriedade da classe majoritária. Exemplo do segundo caso: Tomek links.
Em alguns casos as técnicas de oversampling e undersampling podem ser combinadas para gerar uma base final balanceada.

1.3.3 Data Sampling Strategies

Nem sempre é prático trabalhar com o dataset inteiro. Em vez disso, o usuário pode utilizar amostras dos dados que conheam informação suficiente para o processo de modelagem.
Existem duas estratégias mais comuns de amostragem: probabilística e não probabilística.
Na amostragem probabilística todos os exemplos tem uma chance de serem selecionados; na não probabilística, os exemplos são selecionados a partir de uma sequência determinística de ações.

1.3.3.1 Simple Random Sampling

É o método mais direto. Os exemplos são escolhidos aleatoriamente com a mesma probabilidade.
Uma maneira simples de fazer amostragem aleatória consiste em atribuir números para os exemplos do dataset e sorteá-los ao acaso.
Uma desvantagem da amostragem aleatória é que o usuário pode não selecionar uma quantidade suficiente de exemplos que tenham uma determinada propriedade de interesse.

1.3.3.2 Systematic Sampling

Para implementar o systematic sampling (ou interval sampling) deve-se criar uma lista contendo todos os exemplos do dataset. Dessa lista, seleciona-se aleatoriamente o primeiro exemplo \(x_{start}\) dos primeiros \(k\) elementos da lista. Em seguida, selecionam-se todos os k-ésimos elementos a partir de \(x_{start}\). O valor \(k\) deve ser escolhido de modo que se obtenha uma amostra do tamanho desejado.
Uma vantagem dessa técnica é que ela é capaz de gerar amostras ao longo de todo o conjunto de dados. No entanto, ela não será adequada caso existam padrões repetitivos nos dados.

1.3.3.3 Stratified Sampling

Se os dados apresentam grupos como gênero, localização ou idade, o usuário deve selecionar exemplos de cada um desses grupos.
Na amostragem estratificada deve-se dividir o dataset em grupos e, então, selecionar exemplos de cada grupo aleatoriamente. O número de exemplos selecionados de cada grupo deve ser proporcional ao tamanho do estrato.
Pode ser difícil definir os estratos. Uma estratégia comum é definí-los com o auxílio de técnicas de clustering. Nesse caso, basta definir quantos clusters serão necessários.

1.3.4 Storing Data

Manter os dados seguros é um seguro para a organização.
When sensitive data or personally identifiable information (PII) is provided by customers or business partners, it must be stored in not just a safe but also a secure location.
It’s also recommended to limit access to read-only and add-only operations, by restricting write and erase operations to specific users.
If the data is collected on mobile devices, it might be necessary to store it on the mobile device until the owner connects to wifi. This data might need to be encrypted so that other applications cannot access it.

1.3.4.1 Data Formats

Data for machine learning can be stored in various formats.. The tidy data is usually stored as comma-separated values (CSV) or tab-separated values (TSV) files. In this case, all examples are stored in one file. Alternatively, collection of XML (Extensible Markup Language) files or JSON (JavaScript Object Notation) files can contain one example per file.

1.3.4.2 Data Storage Levels

Storage can be organized in different levels of abstraction: from the lowest level, the filesystem, to the highest level, such as data lake.
Filesystem is the foundational level of storage. The fundamental unit of data on that level is a file. A file can be text or binary, is not versioned, and can be easily erased or overwritten.
A local filesystem can be as simple as a locally mounted disk containing all the files needed for your machine learning project.
A distributed filesystem can be accessed over the network by multiple physical or virtual machines. Files in a distributed filesystem are stored and accessed over multiple machines in the network.
Object storage is an application programming interface (API) defined over a filesystem. Using an API, you can programmatically execute such operations on files as GET, PUT, or DELETE without worrying where the files are actually stored.
The fundamental unit of data in an object storage level is an object. Objects are usually binary: images, sound, or video files, and other data elements having a particular format.
The access to the data stored on the object storage level can often be done in parallel, but the access is not as fast as on the filesystem level.
Canonical examples of object storage are Amazon S3 and Google Cloud Storage (GCS).
The database level of data storage allows persistent, fast, and scalable storage of structured data with fast parallel access for both storage and retrieval.
The fundamental unit of data at this level is a row. A row has a unique ID and contains values in columns. In a relational database, rows are organized in tables.
Databases are not exceptionally well suited for storing binary data, though rather small binary objects can sometimes be stored in a column in the form of a blob.
The four most frequently used DBMS in the industry are Oracle, MySQL, Microsoft SQL Server, and PostgresSQL.
A data lake is a repository of data stored in its natural or raw format, usually in the form of object blobs or files. A data lake is typically an unstructured aggregation of data from multiple sources, including databases, logs, or intermediary data obtained as a result of expensive transformations of the original data

1.3.4.3 Data Versioning

If data is held and updated in multiple places, you might need to keep track of versions. Versioning the data is also needed if you frequently update the model by collecting more data, especially in an automated way
Data versioning can be implemented in several levels of complexity, from the most basic to the most elaborate.
data is unversioned: At this level, data may reside on a local filesystem, object storage, or in a database. The advantage of having unversioned data is the speed and simplicity of dealing with the data.
data is versioned as a snapshot at training time: At this level, data is versioned by storing, at training time, a snapshot of everything needed to train a model. Such an approach allows you to version deployed models and get back to past performance
both data and code are versioned as one asset. At this level of versioning, small data assets, such as dictionaries, gazetteers, and small datasets, are stored jointly with the code in a version control system, such as Git or Mercurial.
using or building a specialized data versioning solution. Data versioning software such as DVC and Pachyderm provide additional tools for data versioning.

1.3.4.4 Documentation and Metadata

While you are actively working on a machine learning project, you are often capable of remembering important details about the data. However, once the project goes to production and you switch to another project, this information will eventually become less detailed. Before you switch to another project, you should make sure that others can understand your data and use it properly.
If the data is self-explanatory, then you might leave it undocumented.
Documentation has to accompany any data asset that was used to train a model. This documentation has to contain the following details: (Consultar livro)

1.3.4.5 Data Lifecycle

Some data can be stored indefinitely. However, in some business contexts, you might be allowed to store some data for a specific time, and then you might have to erase it. If such restrictions apply to the data you work with, you have to make sure that a reliable alerting system is in place
For every sensitive data asset, a data lifecycle document has to describe the asset, the circle of persons who have access to that data asset, both during and after the project development. The document has to describe how long the data asset will be stored and whether it has to be explicitly destroyed.

1.3.5 Data Manipulation Best Practices

1.3.5.1 Reproducibility

Reproducibility should be an important concern in everything you do, including data collection and preparation.
Usually, the data collection and transformation activities consist of multiple stages. These include downloading data from web APIs or databases, replacing multiword expressions by unique tokens, removing stop-words and noise, cropping and unblurring images, imputation of missing values, and so on. Each step in this multistage process has to be implemented as a software script, such as Python or R script with their inputs and outputs

1.3.5.2 Data First, Algorithm Second

Remember that in the industry, contrary to academia, “data first, algorithm second,” so focus most of your effort and time on getting more data of wide variety and high quality, instead of trying to squeeze the maximum out of a learning algorithm

1.4 Feature Engineering

Feature engineering is a creative process where the analyst applies their imagination, intuition, and domain expertise.

1.4.1 Feature Engineering for Text

Bag-of-words
- Tokenize the texts
- Flavors
  - The binary-value model.
  - Counts of tokens.
  - Frequencies of tokens.
  - TF-IDF (term frequency-inverse document frequency).
bag-of-n-grams

1.4.2 Converting Categorical Features to Numbers

one-hot encoding
mean encoding (bin counting or feature calibration)
- Calculate the sample mean of the label using all examples where the feature has value \(z\). Each value \(z\) of the categorical feature is then replaced by that sample mean value.
For binary classification problem, in addition to sample mean, we can also use:
- the raw counts of the positive class for a given value of \(z\).
- the odds ratio, and
- the log-odds ratio.
Ordered categorical features, but not cyclical (ex.: “junior,” “mid-level,” “senior”):
- Use uniform numbers in the [0, 1] range, like 1/3 for “junior”, 2/3 for “mid-level” and 1 for “senior.” If some values should be farther apart, you can reflect that with different ratios. If “senior” should be farther from “mid-level” than “mid-level” from “junior,” you might use 1/5, 2/5, 1 for “junior,” “mid-level,” and “senior,” respectively. This is why domain knowledge is important.
When categorical features are cyclical, integer encoding does not work well. For example, try converting Monday through Sunday to the integers 1 through 7. The difference between Sunday and Saturday is 1, while the difference between Monday and Sunday is −6. However, our reasoning suggests the same difference of 1, because Monday is just one day past Sunday. Instead, use the sine-cosine transformation.

1.4.3 Feature Hashing or hashing trick

Feature Hashing or hashing trick converts text data, or categorical attributes with many values, into a feature vector of arbitrary dimensionality.
How it works:
- First you decide on the desired dimensionality of your feature vectors.
- Then, using a hash function, you first convert all values of your categorical attribute (or all tokens in your collection of documents) into a number, and
- then you convert this number into an index of your feature vector.
Commonly used hash functions are MurmurHash3, Jenkins, CityHash, and MD5.

1.4.4 Topic Modeling

Topic modeling is a family of techniques that uses unlabeled data, typically in the form of natural language text documents. The model learns to represent a document as a vector of topics.
Topic modeling algorithms are Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA),

1.4.5 Features for Time-Series

Check if the observations are evenly spaced over time. Otherwise, convert it into the classical time-series data.
If observations are irregular, such time-series data is called a point process or an event stream.
It’s usually possible to convert an event stream into the classical time-series data by aggregating observations. Examples of aggregation operators are COUNT and AVERAGE.
Shallow machine learning toolkit. To transform a time-series into training data in the form of feature vectors, two decisions must be made:
- how many of the consecutive observations are needed to make an accurate prediction (so-called prediction window), and
- how to convert a sequence of observations into a fixed-dimensionality feature vector.

Usually decisions are made based on the subject-matter expert’s knowledge, or by using a hyperparameter tuning technique.

However, some recipes work for many time-series data. Below is one such recipe:

chunk the entire time series into segments of length \(w\),
create a training example e from each segment \(s\),
for each \(e\), calculate various statistics on the observations in \(s\), such as:
- average (the mean or median during the last \(w\) periods);
- spread (e.g., standard deviation, median absolute deviation, or interquartile range of the values during the last \(w\) periods);
- outliers;
- growth;
- visual (e.g., how different the curve of the values is from a known visual image, such as a hat, or head and shoulders) ???

In the modern neural-network era, analysts most often prefer to train deep neural networks. Long short-term memory (LSTM), convolutional neural network (CNN), and Transformer are popular choices of architecture for a time-series model.

1.4.6 Use Your Creativity

Stacking Features

In the movie title classification problem, concatenate each example, joining the feature vectors of the left context, the extraction, and the right context. We obtain the final feature vector that represents the entire example.

Stacking Individual Features

All additional features, as long as they are numerical, can be concatenated to the feature vector.

1.4.7 Properties of Good Features

High Predictive Power
Fast Computability
- A sparse vector is a vector whose values in most dimensions are zero. If your dataset is small and the texts are short, the learning algorithm will have a hard time seeing patterns in sparse vectors because they contain little information compared to their size.
- A less informative feature computed in a fraction of a millisecond is often preferred to a feature with a high predictive power that takes seconds to compute.
Reliability
Uncorrelatedness
Other Properties
- An essential property of a good feature is that the distribution of its values in the training set is similar to the distribution it will receive in production.
- You could consider engineering cyclical features like “hour of the day,” “day of the week,” “month of the year.”. For the prediction problems in which time seasonality has predictive power, having such features can be useful.
- Features that you design should be unitary, easy to understand, and maintain. A feature like “length divided by weight” is not unitary, as it’s composed of two unitary features. Some learning algorithms may benefit from combining features. However, it’s preferable to do this in a dedicated stage in the model training pipeline. This will be seen later.

1.4.8 Feature Selection

If we could estimate the importance of features, we would keep only the most important ones. That would allow us to save time, fit more examples in memory, and improve the model’s quality. Below, we consider some feature selection techniques.

Cutting the Long Tail
- Typically, if a feature contains information (e.g., a non-zero value) only for a handful of examples, such a feature could be removed from the feature vector.
- The decision on a threshold for defining the long tail is somewhat subjective. You can set it as a hyperparameter for your problem and discover the optimal value experimentally.
- Whether to cut the long tail, and where to do it, is debatable. In classification problems with many classes, the difference between some classes can be very subtle. Even features whose values are rarely non-zero may become important. However, removing long-tail features often results in faster learning and a better model.
Boruta (“Boruta - A System for Feature Selection” paper)
- Boruta iteratively trains random forest models and runs statistical tests to identify features as important and unimportant.
- Boruta worked well for many Kaggle competitions; therefore, you can consider it a universally applicable tool for feature selection.
- One thing worth noting, though, before using Boruta in production: Boruta is a heuristic. There are no theoretical guarantees for its performance.
- If you want to be sure that Boruta doesn’t harm, run it multiple times and make sure that the feature selection is stable (i.e., consistent across multiple Boruta applications to your data).
- If the feature selection is not stable, make sure that the number of trees in the random forest is large enough to generate stable results.

1.4.9 L1-Regularization

Regularization is an umbrella term for a range of techniques that improve the generalization of the model. Generalization, in turn, is the model’s ability to correctly predict the label for unseen examples.

L1 penalizes the model for being too complex.

In practice, L1 regularization produces a model that has most of its parameters equal to zero. Therefore, L1 implicitly performs feature selection by deciding which features are essential for prediction, and which ones are not.

1.4.10 Synthesizing Features

Feature Discretization

Reasons to discretize a real-valued numerical feature:
- some feature selection techniques only apply to categorical features.
- a successful discretization adds useful information to the learning algorithm when the training dataset is relatively small.
- discretization can lead to improved predictive accuracy.
- it is also simpler for a human to interpret a model’s prediction if it is based on discrete groups of values, such as age groups or salary ranges.
Binning, also known as bucketing, is a technique that allows transforming a numerical feature into a categorical one by replacing numerical values in a specific range by a constant categorical value.
- uniform binning
- k-means-based binning, and
- quantile-based binning.

In all three cases, you should decide how many bins you want to have.

Most modern machine learning algorithm implementations require numerical features. The bins must be transformed back to numerical values by using a technique like one-hot encoding.

Synthesizing Features from Relational Data

If you want to increase the predictive power of your feature vectors, or when your training set is rather small, you can synthesize additional features that would help in predictions. There are two typical ways to synthesize additional features:

from the data, or
from other features.

Synthesizing Features from the Data

synthesize one or more additional features with clustering using algorithms like the k-means clustering.
Then add \(k\) additional features to your feature vectors. The additional feature \(D + j\), where \(j = 1, . . . , k\), will be binary and equal to 1 if the corresponding feature vector belongs to cluster \(j\).

Synthesizing Features from Other Features

Three typical simple transformations that apply to a numerical feature \(j\) in example \(i\) are:
- discretization of the feature
- squaring the feature
- computing the sample mean and the standard deviation of feature \(j\) from \(k\)-nearest neighbors of the example \(i\) found by using some metric like Euclidean distance or cosine similarity.
- Transformations that apply to a pair of numerical features are simple arithmetic operators: +, −, ×, and ÷ (a technique also known as feature-crossing).

1.4.11 Learning Features from Data

Learning features from data is especially effective when we can get access to large collections of relevant labeled or unlabeled data, such as text corpora or collections of images from the Web.

Word Embeddings

Word embeddings are feature vectors that represent words.
Similar words have similar feature vectors, where similarity is given by a certain measure, such as cosine similarity.
Once you have a collection of word embeddings for some language, you can use them to represent individual words in sentences or documents written in that language, instead of using one-hot encoding.
One problem with word embeddings trained using word2vec is that the set of word embeddings is fixed, and you cannot use the model for out-of-vocabulary words, that is, the words that weren’t present in the corpus used to train word embeddings. There are other architectures of neural networks that allow obtaining embeddings for any word, including out-of-vocabulary words. One such architecture, often used in practice, is fastText. It was invented at Facebook, and the code is available in open source.

Document Embeddings

A popular way of obtaining an embedding for a sentence or an entire document is to use the doc2vec neural network architecture, also invented at Google and available in open source. The architecture of doc2vec is very similar to word2vec. The only major difference is that now there are two embedding vectors, one for the document ID and one for the word.

Embeddings of Anything

Choosing Embedding Dimensionality

\(d = \sqrt(D)\)
- \(d\) = the embedding dimensionality
- \(D\) = the “number of categories.”

Dimensionality Reduction

often results in increased learning speed and better generalization.
Principal Component Analysis (PCA):
- the oldest of the techniques.
- the fastest option.
- use PCA as a step preceding your model training, and find the optimal value of the reduced dimensionality experimentally as part of the hyperparameter tuning process.
- PCA’s most significant drawback is that the data must fit in memory entirely for the algorithm to work.
- Incremental PCA allows running the algorithm on batches of the dataset, loading in memory one batch at a time, but is an order of magnitude slower than PCA.
Dimensionality Reduction for Visualization:
- If visualization is your goal.
- Uniform Manifold Approximation and Projection (UMAP) algorithm, or an autoencoder.
- They can be specifically programmed to produce 2D or 3D feature vectors.
- UMAP requires all data to be in memory, while autoencoder can be trained in batches.
- t-SNE

1.4.12 Data Leakage in Feature Engineering

Data leakage during feature engineering can happen in several situations, including feature discretization and scaling.
Problem:
- Use the entire dataset to calculate the ranges of each bin or the feature scaling factors. Then you split the dataset into training, validation, and test sets. If you proceed like that, the values of features in the training data will, in part, be obtained by using the examples that belong to the holdout sets.
Solution:
- First, split the entire dataset into training and holdout sets, and only do feature engineering on the training data. This also applies when you use mean encoding to transform a categorical feature to a number: split the data first and then compute the sample mean of the label, based on the training data only.

1.4.13 Storing and Documenting Features

Schema File
- names of features
- type (categorical, numerical)
- the fraction of examples that are expected to have that feature present
- minimum and maximum values
- sample mean and variance
- whether it allows zeros
- whether it allows undefined values.

1.4.14 Feature Store

Challenges faced by Large distributed organizations:
- Features not being reused
- Feature definitions vary
- Computationally intensive features
- Inconsistency between training and serving
- Feature expiration is unknown
A feature store is a central vault for storing documented, curated, and access-controlled features within an organization. Each feature is described by four elements:
- 1. name: is a string that uniquely identifies the feature, for example: “aver- age_session_length” or “document_length.”
- 1. description: is a natural language textual description of the object’s property it represents, for example, “The average length of the session for a user.” or “The number of words in the document.”
- 1. metadata:
  - why the feature was added to the model,
  - how it contributes to generalization,
  - the person’s name in the organization responsible for maintaining the feature’s data source,
  - the input type (e.g., numerical, string, image), the output type (e.g., numerical scalar, categorical, numerical vector),
  - whether the feature store must cache the value of the feature, and if yes, for how long.
  - A feature can also be marked as available online and offline, or just for offline processing. Features available for online processing must be implemented in such a way that their value can be either:
    - 1. read fast from a cache or a value store or
    - 1. computed in real-time.
  - Features that can be computed in real-time include, for example, squaring the input number, determining the shape of the word, or doing a search in the organization’s intranet. Setor censitário.
- 1. definition. The definition of the feature is the versioned code, such as Python or R. It will be executed in a runtime environment and applied to the input to compute the feature value.
A feature store allows data engineers to insert features. In turn, data analysts and machine learning engineers use an API to get feature values which they deem relevant. A feature store can provide features for a single online input. Or, the analyst working on a model offline may want to convert the training data into a collection of feature vectors, and will send to the feature store a batch of inputs.
For reproducibility, feature values in a feature store are versioned. With feature value versioning, the data analyst is able to rebuild the model with the same feature values as those used to train the previous model version. After the feature value for a given input is updated, the previous value is not erased. Rather, it is saved with a timestamp indicating when that value was generated. Furthermore, a feature \(j\) used by model \(m_B\) can itself be the output of some model \(m_A\) . Once model \(m_A\) changes, it is important to keep its older versions: model \(m_B\) still might expect as input the outputs generated by an older version of \(m_A\) .

1.4.15 Feature Engineering Best Practices

Generate Many Simple Features
Reuse Legacy Systems
- When replacing an old, non-machine-learning-based algorithm with a statistical model, use the output of the old algorithm as a feature for the new model.
Use IDs as Features when Needed…
Reduce the Cardinality When Possible
- Use categorical features with many values (more than a dozen) only when you want the model to have different “modes” of behavior that depend on that categorical feature.
  - Typical examples of this are:
    - zip code (postal code) or country. You might consider using the categorical feature “Country” if you want the model to behave differently in Russia versus the United States, for otherwise similar inputs. (Often, what you want your model to do and what the data dictates are two very different things. Even if you think that the model must make similar predictions independently of the country, in reality, you might get poor model performance because the distribution of labels in the training data is different for different countries.)
- If you have a categorical feature with many values, but you do not need a model that has several modes depending on that feature, try to reduce the cardinality (i.e., the number of distinct values) of that feature.
- Other techniques to reduce the cardinality of features:
  - Group similar values
  - Group the long tail
  - Remove the feature
- The reduction of a feature’s granularity should be made with care. Categorical features often have functional dependencies with other categorical features, and their predictive power often comes from their combinations. Take state and city as an example. If we decide to group or remove some values in the state feature, we might inadvertently destroy the information that would allow the model to distinguish one “Springfield” from another.
Use Counts with Caution
- Some counts remain roughly in the same bounds over time.
- The same caution must be applied when you group feature values in bins based on how common those values are in the dataset. Infrequent values today may become more frequent over time, as more data is added. It is considered a best practice to re-evaluate the model and the features from time to time.
Make Feature Selection When Necessary
- The reasons could be:
  - the need to have an explainable model (so you keep the most significant predictors),
  - strict hardware requirements, such as RAM, hard drive space, or
  - short time available to experiment and/or rebuild the model in production, you expect a significant distribution shift between two model trainings.
  - If you decide to do feature selection, start with Boruta.
Test the Code Carefully
- Unit tests should cover each feature extractor.
- Check that each feature is generated correctly using as many inputs as possible.
- For each boolean feature, check that it is true when it should be true and is false when it should be false.
- Check numerical features for a reasonable value range.
- Check for NaNs (Not-a-Number values), nulls, zeros, and empty values.
- A broken extractor for one feature can result in arbitrarily poor performance of the model.
- Feature extractors are the first place to look for a problem if the model’s behavior is strange.
- Each feature has to be tested for speed, memory consumption, and compatibility with the production environment. What works reasonably well in your local environment may perform poorly when deployed in production.
- Once the model is deployed in the production environment, and each time it is loaded, you must rerun feature extractor tests. If a feature consumes some external resources like a database or an API, these resources might be unavailable on a specific production runtime instance.
- The feature extractor has to throw an exception and die if any resource during feature extraction is unavailable.
- Avoid silent failures that may remain unnoticed for a long time with model performance degrading or becoming completely wrong.
- It is also recommended to perform regular runs of feature extractors on a fixed test data to make sure that the feature value distribution remains the same.
Keep Code, Model, and Data in Sync
- The version of the feature extraction code must be in sync with the model’s version and the data used to build it. The three have to be deployed or rolled back at the same time.
- Each time the model is loaded in production, it’s useful to check that the three elements are in sync (that is, their versions are the same).
Isolate Feature Extraction Code
- The feature extraction code must be independent of the remaining code that supports the model.
- It should be possible to update the code responsible for each feature without affecting other features, the data processing pipeline, or the way the model is called. The only exception is when many features are generated in bulk, like in one-hot encoding and bag-of-words.
Serialize Together Model and Feature Extractor
- When possible, jointly serialize (pickle in Python, RDS in R) the model and the feature extractor object that was used when the model was built. In the production environment, deserialize both and use them.
- When possible, avoid having several versions of the feature extraction code. If your production environment doesn’t let you deserialize both the model and the feature extraction code, use the same feature extraction code when you train the model and serve it. Even a tiny difference between the code a data scientist used to train the model, and the optimized code the IT team might have written for the production environment, may result in significant prediction error.
- Once the production code for feature extraction is ready, use it to retrain the model.
- Always completely retrain the model after any change in the feature extraction code.
Log the Values of Features
- Log the feature values extracted in production for a random sample of online examples.
- When you work on a new version of the model, these values will be useful to control the quality of the training data. They will allow you to compare and ensure that the feature values logged in the production environment are the same as those you observed in the training data.

1.4.16 Feature Engineering and Selection: A Practical Approach for Predictive Models

1.4.17 The Bias-Variance Trade-Oﬀ

On the Bias-Variance Tradeoff: Textbooks Need an Update

1.4.18 Feature selection

The main concern during feature selection is overfitting. This is especially true when the number of data points in the training set is small relative to the number of predictors.
Feature selection should not be used as a formal method of determining feature significance. More traditional inferential statistical approaches are a better solution for appraising the contribution of a predictor to the underlying model or to the data set.

1.4.19 Chapter 2: Predicting Risk of Ischemic Stroke

Model development pipeline
- split the original data set.
  - Validar a distribuição de todas as variáveis, não apenas da variável resposta.
- preprocess
  - feature individual distribution
  - missingness within each predictor.
    - Many models cannot tolerate any missing values. Therefore we must take action to eliminate missingness to build a variety of models.
  - potentially unusual values within predictors
  - relationships between predictors
  - the relationship between each predictor and the response
  - mean centered and scaled to unit variance.
  - skewness is often due to the underlying distribution of the data. The distribution, instead, is where we should focus our attention.
    - A simple log-transformation, or more complex Box-Cox or Yeo-Johnson transformation (Section 6.1), can be used to place the data on a scale where the distribution is approximately symmetric, thus removing the appearance of outliers in the data. This kind of transformation makes sense for measurements that increase exponentially.
    - remove predictors that are highly correlated with other predictors (\(r^2\) > 0.9). The correlation threshold is arbitrary and may need to be raised or lowered depending on the problem and the models to be used.
- Exploration (explore potential predictive relationships between individual predictors and the response and between pairs of predictors and the response.)
  - Use 10-fold cross-validation technique to train models to answer the question “which of the predictors have simple associations with the outcome?”. When we want to compare two models (\(M_1\) and \(M_2\)), the following procedure will be used:
  - To illustrate this algorithm, two logistic regression models were considered. The simple model, analogous to the statistical “null model” contains only an intercept term while the model complex model has a single term for an individual predictor from the risk set. The figure below orders the risk predictors from most significant to least significant in terms of improvement in ROC.
  - Similarly, relationships between the continuous predictors and outcome can be explored. As with the risk predictors, the predictive performance of the intercept-only logistic regression model is compared to the model with each of the imaging predictors.
  - Univariate associations between continuous predictors and outcome. The p-value of the improvement in ROC for each predictor over the intercept-only logistic regression model is listed in the top center of each facet.
  - ROC curve for a continous predictor as a stand alone predictor of outcome based on the training data.
  - There are more exploratory steps can be taken to identify other relevant and useful constructions of predictors that improve a model’s ability to predict. In this case, the stroke data in its original form does not contain direct representations of interactions between predictors. Pairwise interactions between predictors are prime candidates for exploration and may contain valuable predictive relationships with the response.
    - For numeric predictors, the interactions are simply generated by multiplying the values of each predictor.
    - For each interaction term, the same resampling algorithm was used to quantify the cross-validated ROC from a model with only the two main effects and a model with the main effects and the interaction term. The improvement in ROC as well as a p-value of the interaction model versus the main effects model was calculated.

1.5 Model Training

1.5.1 Validate Schema Conformity

1.5.2 Define an Achievable Performance Level

Defining an achievable performance level is a crucial step. It gives you an idea of when to stop trying to improve the model.

1.5.3 Choose a Performance Metric

There’s no single best metric you can use for every project. You will choose based on your data and the problem.
It is recommended to choose one, and only one, performance metric before you start working on the model. Then, compare different models and track the overall progress by using this one metric.

1.5.4 Choose the Right Baseline

Before you start working on a predictive model, it is important to establish baseline performance on your problem.
A baseline doesn’t have to be the result of any learning algorithm. It can be a rule-based or heuristic algorithm, a simple statistic applied to the training data, or something else.
The two most commonly used baseline algorithms are:
- random prediction, and
- zero rule.
The random prediction algorithm makes a prediction by randomly choosing a label from the collection of labels assigned to the training examples.
In classification, the zero rule algorithm strategy is to always predict the class most common in the training set, independently of the input value. The strategy for regression is to predict the sample average of the target values observed in the training data.