Data Science Stack Exchange Community Digest

Top new questions this week:

1% of data for training 99% of data for testing

I got feedback from a reviewer. It is really important for me to answer to this question. I would appreciate of any help. it was mentioned that 1% of the data was used for training while 99% was used …

machine-learning

asked by Ahmad Turani Score of 11 answered by Erwan Score of 19

How can I observe my CSV files better?

I’m running a lot of experiments that give their output as CSV files. An experiment might be running for hours, with a new line being added to the CSV every 10 seconds. Right now I’m opening these CSV …

data plotting csv

asked by Ram Rachum Score of 6 answered by griko Score of 1

Algorithm to determine whether the first row in CSV is likely to be a header row or a data row

I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a …

statistics word-embeddings similarity csv

asked by David Ferris Score of 2 answered by Nemo_the_scientist Score of 1

why CNN model can't learn well the peak from data

here I have two different datasets. dataset1 is force plate data and dataset2 is plantar pressure data. dataset1 has shape (2050,2) and dataset2 has shape(2050,89). before doing the training I have …

deep-learning regression cnn

asked by stack offer Score of 1 answered by Nicolas Martin Score of 0

How to convert an ONNX model with dynamic input shape to Tensorflow model

I am trying to convert an ONNX model with a dynamic input shape to TensorFlow format using the onnx_tf package. I am using TensorFlow 2.11.0, ONNX 1.13.0, and onnx_tf 1.10.0. The input to the model …

python tensorflow data-science-model

asked by Valeria Mordoh Score of 1

Diffusion Model without Text Prompt?

I hope I understand diffusion models well enough that this isn’t too dumb of a question but… I’d like to (I think) augment a diffusion model to: At training time accept CT scan images, then apply an …

deep-learning computer-vision gan

asked by ncf031 Score of 1

Test score higher than train score

I implemented a Gaussian Naive Bayes classifier and I got a test score (99,99%) higher than the train score (96,87%) Is this normal or does it mean that my model is underfitting ? Thank you.

machine-learning classification training naive-bayes-classifier test

asked by biihu Score of 1 answered by Brian Spiering Score of 0

Greatest hits from previous weeks:

What is the difference between “equivariant to translation” and “invariant to translation”

I’m having trouble understanding the difference between equivariant to translation and invariant to translation. In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio)…

neural-network deep-learning convolution

asked by Aamir Score of 77 answered by Laurent Duval Score of 80

How can I check the correlation between features and target variable?

I am trying to build a Regression model and I am looking for a way to check whether there’s any correlation between features and target variables? This is my …

machine-learning scikit-learn regression linear-regression

asked by user_6396 Score of 30 answered by JahKnows Score of 23

Adding Features To Time Series Model LSTM

have been reading up a bit on LSTM’s and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to …

machine-learning neural-network deep-learning time-series

asked by Rjay155 Score of 68 answered by Adam Sypniewski Score of 56

Why should the data be shuffled for machine learning tasks

In machine learning tasks it is common to shuffle data and normalize it. The purpose of normalization is clear (for having same range of feature values). But, after struggling a lot, I did not find …

machine-learning neural-network deep-learning

asked by Green Falcon Score of 61 answered by Josh Score of 43

Mapping column values of one DataFrame to another DataFrame using a key with different header names

I have two data frames df1 and df2 which look something like this. …

python pandas dataframe

asked by Danny Score of 10 answered by Stephen Witkowski Score of 8

How to include labels in sns heatmap

I got this matrix …

python visualization numpy seaborn

asked by Srihari Score of 17 answered by Rudra Mohan Score of 20

What does “baseline” mean in the context of machine learning?

What does “baseline” mean in the context of machine learning and data science? Someone wrote me: Hint: An appropriate baseline will give an RMSE of approximately 200. I don’t get this. Does he …

machine-learning regression predictive-modeling terminology

asked by Meiiso Score of 28 answered by Carl Rynegardh Score of 31

Can you answer these questions?

Python Library Trend time series multivariate

Our csv contains 36 columns 1 date time column collected every 30 mins 3 variables (count,latency,Totaltime) x 10 Features(user io, serverio ,concurrency ..etc ) Of different data points from the …

outlier multivariate-distribution

asked by trent Score of 1 answered by Nicolas Martin Score of 0

How many samples in dataset are required to fine-tune BERT for binary classification?

I’m trying to fine-tune a BERT-based model for a binary classification task (data is in English). The dataset I’m working with is quite small (~500 samples, out of which 80% are currently used for …

nlp dataset bert finetuning

asked by Occasus Score of 1

How to properly compute spam message score as a combination of fixed features and probability from a naive bayes classifier?

I am building a learning spam/ham email classifier as an assignment. It’s not supposed to be a good general classifier, but one that can learn on a small set of labeled emails of a user (approx. 650 – …

naive-bayes-classifier

asked by David Korčák Score of 1 answered by Mohith7548 Score of 0