Data Science Stack Exchange Community Digest

Top new questions this week:

In sklearn tfidf what is the difference between term frequecy and document frequency

Looking at the sklearn tfidf page: scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html and trying to understand the difference between term frequency …

nlp scikit-learn tfidf

asked by james pow Score of 2 answered by ru.mp Score of 2

Transformer XL – understanding paper's illustration

If I understand correctly, the Key hidden layer in the Transformer XL is of size 2L * d, where …

deep-learning nlp transformer attention-mechanism

asked by AMT Score of 2 answered by Nicolas Martin Score of 1

How to use categorical data in forecasting with Prophet?

I’m trying to create a model to predict the number of players on a video game at a certain time and was trying to figure out how to integrate categorical data into my forecasting problem. So far, my …

machine-learning python categorical-data forecasting prophet

asked by CrazyRageMonkey Score of 2 answered by Nicolas Martin Score of 0

NER – What advantage does IO Format have over BIO Format

In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or “O” in case the word …

machine-learning nlp transformer bert named-entity-recognition

asked by Damm Joe Score of 2 answered by Erwan Score of 2

Difference in proportion between labelled sample and the population

I’m working on a project to predict bots from legit users from social medias. The data that I collected has about 5% of bots for 95% of legit users. The problem is as I labelled my data, I was more …

machine-learning dataset ensemble-modeling labeling

asked by Marc Score of 2 answered by Nicolas Martin Score of 2

Can using model evaluation metrics to choose a model cause data leakage?

I was reading a blog post about improving machine-learning model train/validate/test splits. Towards the end was this remark: I say we should be more creative in the way we test machine learning …

machine-learning data-leakage model-evaluations

asked by Davis Score of 1 answered by Erwan Score of 0

Topic Modeling – n-grams or 1,2,3,…n-grams?

Do people use n-grams or 1,2,3,…n-grams in both matrix factorisation and generative models in Topic Modeling? I’ve been trying to understand the basics of Topic Modeling and came to know that there …

nlp topic-model lda document-term-matrix

asked by rahuladwani Score of 1 answered by Brian Spiering Score of 1

Greatest hits from previous weeks:

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string …

python dataframe csv data-formats

asked by cappy0704 Score of 8 answered by cappy0704 Score of 2

What does “linear in parameters” mean?

The model of linear regression is linear in parameters. What does this actually mean?

regression linear-regression

asked by Albert Gao Score of 16 answered by ayhan Score of 18

Perform k-means clustering over multiple columns

I am trying to perform k-means clustering on multiple columns. My data set is composed of 4 numerical columns and 1 categorical column. I already researched previous questions but the answers are not …

python clustering pandas

asked by Lola Score of 13 answered by Simon Larsson Score of 19

DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20

I was watching Machine Learning A- Z from SuperDataScience but when I was doing below code sample: …

machine-learning python scikit-learn

asked by Navid Score of 10 answered by xYan Score of 11

What is the difference between bootstrapping and cross-validation?

I used to apply K-fold cross-validation for robust evaluation of my machine learning models. But I’m aware of the existence of the bootstrapping method for this purpose as well. However, I cannot see …

cross-validation evaluation

asked by Fredrik Score of 50 answered by cbeleites unhappy with SX Score of 54

AttributeError: 'numpy.ndarray' object has no attribute 'columns'

…

scikit-learn pandas numpy

asked by Balu Score of 2 answered by Djib2011 Score of 3

Calculation and Visualization of Correlation Matrix with Pandas

I have a pandas data frame with several entries, and I want to calculate the correlation between the income of some type of stores. There are a number of stores with income data, classification of …

python statistics visualization pandas

asked by gdlm Score of 38 answered by AN6U5 Score of 28

Can you answer these questions?

Handling missing value in column having textual data

I have been working on a supervised ML use case where dataset has Numerical (Price), Categorical(Category) and Textual data(Description) as features. Description feature has about 30% missing values. …

data-cleaning text-classification

asked by shobhit kumar Score of 1 answered by Erwan Score of 0

Why, in a classification task, with $n$ input variables, we can obtain all $2^n$ classification functions for each possible set of missing inputs?

I don’t get why, for a classification task with missing values, With $n$ input variables, we can obtain all $2^n$ different classification functions needed for each possible set of missing inputs,. …

classification

asked by Revolucion for Monica Score of 1

What does precision-recall curve and ROC curve tell us abouth threshold invariance

Consider a binary classification problem. Intuitively, a value for the area under the curve (for both curves) very close to 1, shows that the curve is almost L-shaped. Thus, this means that the value …

classification evaluation metric binary-classification roc

asked by liakoyras Score of 1