previous thread >>16187525This is one of the boards newest generals. Fairly high activity due to edge lords trying to be funny but instead spreading facts about the absolute state of our world.Intro stats is fairly easy, intermediate stats come the programming and we have already have several battles about what language is the best in the thread. Nobody uses SAS funnily enough, SPSS has had some people trying to joust the edgelords who are into R and C++, while the stata children are silent as usual.Come one, come all. State your dumb questions, /pol/tardy or not. Some fairly useful and funny math is showcased in this thread.
>>16256996I actually chuckled at the meme. Good job OP.Anyone worked with GIS and spatial statistics for countries?
>>16256996I always thought the Bell curve was named after Alexander Graham Bell
>>16256996>no racial crime statistics in the OP.Yeah, I can tell this thread will die prematurely again very soon.
>>16257157The previous one lasted for weeks and yielded fruitful discussions and meming
>>16256996you should have had Poisson distribution for high N
>>16256996but we have /qm/ - quantum mechanics general already
>>16256996I heard from a actuarial science major that only us physics majors would say "Gaussian" and everyone else said "normal distribution". Is that accurate?
>>16257607Gaussian is just standard normal, a specific normal distribution, but not every normal is gaussian
>>16256996>edgelords who are into Rmy people. Never noticed this thread before, but I love pointing out to my development friends that whatever they're doing can be done faster in R.
>>16257607I actually think it is kind of accurate.
>>16257726Honestly I learnt stats and data analysis in R. I could start using other shit but why would I? I am mostly doing one-offs anyways on the data so R is perfect. Only drawback on R is that it's portability is shit. Python is far better on this.
frequentists aren't real statisticians, only bayesianists and likelihoodists are.
>>16257726>I love pointing out to my development friends that whatever they're doing can be done faster in RR isn't really known for speed, but I'll agree with your assessment if you're including developer time. Being able to do data processing and shell scripting in a single language is a killer feature, and the great support for visualization gives it the edge over Perl.In theory you could script in Python as well (I had to do this once on a machine where R was unavailable), but I found it a nuisance to have to import modules for everything from regex to plotting to array manipulation, each of which forces you to constantly worry about the mutability/immutability of your data structures (arguably reasonable for programmers, but a nuisance for scripters who just want to get things done). R is content to get out of the way and let you do what you want, and I'll still do most of my exploratory data analysis (read: mucking around) in R before hiving off the slower parts to Python if necessary (not very often actually).
>>16257607I'm an EE and we use "Gaussian" all the time. Especially when we are assuming zero mean Gaussian additive errors.
>>16257607Neuro here, we use Gaussian all day every day
>>16257901The fuck is a "likelihoodist?" Aren't most frequentist methods variations on maximum likelihood approaches?
>>16257726R is love, R is life
>>16259330
>>16257996Eh, just use pandas and numpy. Data frames are immutable and pass by value so you can do things like feed into into a function or something like the below and retain both data frames.df_1 = df.groupby()......It's worth investing in object oriented programming as well ad Python is great if you're lazy and you don't want to deal with RAII or header files
I had probability and statistics this year, found out I really like it but I don't understand it that well even though I passedWhat are some books I could read to get started from the beginning? Is there a chart or something?I was thinking about going with Blitzstein and Hwangs intro to probability, is it good?
>>16259615Probability and Statistical Inference by Tanis and Hogg is really good imo. If you don't know calculus and linear algebra, I would recommend getting a life sciences or business statistics book that strips it.If you know real analysis, then I would recommend you consider going straight to measure theory. It's nice having a rigorous, ground up approach without having to worry about things like, "what is a set function", "what is a valid event? What is a valid event space? What is a probability space? How can the continuous distributions on the real numbers by countable additive like a discrete distribution?"
>>16256996I have a question for you guys. In my law school, the only 3 grades are P, H, and HH. The bottom 50% get Ps, the next 35% get an H, and the top 15% of the class get an HH. In first year, there are 7 courses, and 216 students taking the same courses. What percentage of the class got Ps in all 7 courses?
>>16257607I say "bell curve," but apparently that's a racist dogwhistle or something cuz google refuses to give results with that term
>>16259890Are the percentages fixed or do they differ from course to course? If they are fixed, then the worst half of the class probably gets P. The thing is though, you need more info course by course. So I would say the question is incomplete.
>>16260093Every course has this distribution. What further information do you need to answer this question?
>>16260101If every course has that distribution, your question is answered. However, to get how many who got 7 Ps and not a single other grade, you need more info, like a line item for each student with their grade transcript. I would have solved this via a simple SQL command desu, which is how this is solved, just filter on all who have 7Ps.
>>16260101This question has a hidden assumption about the relationship between a student's grade in one class vs. another. It seems to imply they are equally likely, meaning if student M has a P in class 1, the distribution of their potential grades in class 2 is conditionally independent from this information. To properly answer this in its most general form, you'd need to know the pairwise conditional distributions for p(G_i|G_j) for all combinations of (i,j).
>>16260225>If every course has that distribution, your question is answered.No it isn't, because a person who gets a P in one class isn't guaranteed to get a P in every single class. A median student probably has mostly Ps but an H or two. I'm asking what percentage of the class manages to get a P 7 times, in other words end up in the bottom 50 percent in every single class
>>16260225>>16260238>>16260367If the likelihood of a given student receiving a P in any course is 50%, then:(50%)^7 = 0.00781250.0078125 * 100 = 0.78125So, approximately 0.78125% of the class could potentially receive a P in all 7 classes.
My undergraduate statistics program doesn't require real analysis, but I have the option to take it as an elective. Should I? I'm a brainlet btw
Suppose I need to maximize the mutual information of two parametric random variables. I want to formulate this as a convex optimization. What family of the two random variables satisfies the requirement that their mutual information is convex everywhere and they have most possible representation power?
>>16257607Statisticians say normal.
>>16260374This is exactly what I was talking about where it has a hidden uniformity assumption.The course having a share of 50% P grades tells you nothing about how they are distributed (which you are assuming to be uniform). To go from student to grade (rather than from class to relative share of grades) you need to know how the grades are distributed within the class, not just their relative apportionment.When you say (.5)^7 for the probability of a student having all P grades, you are making an assumption that the P grades are distriubed uniformly among the students within class. That assumption is likely unfounded, as a student who gets an HH in one class is not equally likely to get a P in another. You'd likely need some sort of Markov or graphical model (which are basically the same thing but different ways of representing the same idea) to capture the distribution of the grades among the students within the class and between classes.
>>16260484Real analysis is helpful for graduate level probability/statistics because it gives you the basics you need for set operations and measure theory. Probability in grad level programs is basically an application of measure theory (in the definition of your probability space and probability measures), functional analysis (in the definition of the relative density functions relating one measurable space to another) and Fourier analysis (in the characteristic function representation of a probability distribution/density and the moment generating function if it has one, as well as the power spectral density of a stochastic process for repeated sampling). Real analysis definitely won't hurt to take, but don't be surprised if it takes a little bit for it to be useful as it doesn't start becoming relevant until measure theory (which is a later part of real analysis).
How do I start becoming a bayesianist? I want to escape this frequentist lifestyle.
>>16260785You just have to believe. That and probably read some Bayesian statistics textbook or something. If you're a proper statistician The Bayesian Choice or Gelman are both good choices (though I much prefer the former over the latter). If you aren't a statistician, probably read a modern ML book (like Bishop's Pattern Recognition and Machine Learning or Theodoridis's Machine Learning a Bayesian and Optimization Perspective) as they basically are all just Bayesian stats at the end of the day.
>>16260788Meme wording of my last post aside, I'm joining a master's program and already have had a supervisor assigned, who seems to have overestimated my calibre, I have undergrad level cs knowledge and my supervisor wants me to study and understand bayesian deep learning and bayesian machine learning in general before I join. I have like a month to join. I have read a couple machine learning books by now but I don't think I've read anything especially bayesian.
>>16260782I was thinking this as well. To just assume uniformity is most probably not a good representation of the sample.
>>16257607It actually depends on geography. A lot of European mathematicians say "Gaussian."
>>16260782You could just assume normal distribution
my hot take: statistics is 90% philosophy. the math is quite easy. understanding what the fuck is going on, the hidden underlying assumptions is more important than plugging equations.>>16260793>undergrad level cs knowledgeEurope? they're not know for having the most strict criteria for grad school.
>>16260945Normal distributions are for continuous random variables. There's no such thing as a "normal distribution" for the probability of an individual student to have one of three discrete grades (conditioned on the number of those grades in the overall class).
>>16261095But wouldn't the total class averages with all 7 courses be shaped as a bell curve? Like if you assigned P a value of 1, H a value of 2, and HH a value of 3, in the middle of the graphs would be students with a score of around 8 to 10, and they'd be the bulk of the class.
>>16260793Ah, okay. Yeah so if you read through any of the three following books you will get a basic background in the Bayesian approach to ML.- Theodoridis, Machine Learning A Bayesian and Optimization Perspective - Bishop, Pattern Recognition and Machine Learning- Murphy, Probabilistic Machine Learning An Introduction While all three of those books do offer a "general coverage" of ML, they do so very much from a Bayesian perspective. Q: What does a Bayesian perspective mean in this case?A: We assume that the quantities themselves we seek to estimate are also randomly distributed in some fashion, and we impose a prior structure onto them in order to better pose the problem. This shows up in Bayesian interpretations of frequentist methods as well (e.g., LASSO is simply a Laplacian prior on the weights for your "frequentist" linear regression scheme, effectively turning it into a maximim a posteriori problem rather than a constrained likelihood problem). Bayesian ML essentially just views "the truth" of whatever you are doing (classifying, regression, clustering etc.) as a process with some intrinsic probabilistic structure which reflects our uncertainties.
>>16261102You are confusing the intuitive notion of a "bell curve" with an actual proper Gaussian/normal which is continuous. If you were to make an estimate for the average grade across the 7 courses via your numerical structure, it would approach a bell shape but it wouldn't be a normal distribution because it isn't possible for it to be supported beyond the upper and lower bounds. You couldn't, for example, have a value of -30 as your grade, but a normal on the reals must have a non-zero probability of x < -30. You could get something bell shaped via that approach, but it would likely be a beta distribution because of the constrained support (minimum 7 course score is 3, maximum 7 course score is 21) not a normal distribution.
>>16261102But more to the point, and aside from what I've said in >>16261113 you are implicitly imposing a structure. If you assume the grades are uniformly distributed, the distribution of their sum across the 7 courses for an individual student will approach something bell-shaped on the support {3,4,...,21} if we assume that the grades are uniformly distributed within a class and the classes are independent (i.e., student A is equally likely to get a P as an HH, and student A is equally likely to get a P in class 1 as in class 2). This is an imposition of a prior distribution on the grade to student pairings. It is an uninformative prior, but it's a prior nonetheless. It could very well be the case that there is some conditional dependency structure. Students who get P's in class 1 are probably very likely to get P's in classes 2-7. Students who get HH's in class 1 might be very unlikely to get P's in all of the others. This is information that needs to be specified in order to give a real answer to the question, and it is not possible to answer it without imposing a structure onto how grade-student pairings function.
>>16261102According to the central limit theorem you are not incorrect, however, it would not be a "normal bell curve", it would just look like one and you could approximate it as one, because you are using discrete variables.
what's the best way to estimate mutual information of two random distributions whose you don't know the parametric form but you can sample a (limited) number of samples from their join distribution and individual distribution?
anyone from here who can help me with this?>>16258416>>16258691
>>16261549techniques from measure theory
>>16261885what techniques?
>>16261883EE PhD who has written a few papers on statistical learning in the context of signal processing here.There's basically two completely different approaches to AI/ML that have very little interaction with each other because neither party wants to actually learn what the other is doing. There's the CS approach (which falls genuinely under what used to be called AI) which focuses heavily on algorithms and treats everything like a variation on a search strategy. This approach is great if you want a ML based AI for a chatbot or something because it allows for a clear logical decision framework and provides a lot of structure for data management/processing. It is, however, absolutely hot trash for the classical ML tasks of classification, regression and clustering because CS people only seem to know how to think in terms of discrete graphs and don't know a God damn thing about probability theory on real random variables. This is pretty important in many fields where the discrete thing your computer is doing is operating on a sampled version of some analog signal (e.g., computer vision, image recognition from a real camera, auditory speech processing and really any kind of classification/regression tasks where the variables of interest exist in meat space and are thus continuous). The other approach is the "engineering" or "applied math" approach (which is more my background). We are much more into the probability theory side of things, and as a result an engineeringor mathematics/stats oriented ML textbook will cover very deeply the probabilistic models, what they mean, how to apply them and how to vary them. What they won't cover with really any depth are the algorithmic and data structures aspect of things. This leads to interesting things like "data science" graduates using decision trees and having no clue how the fuck a graphical model works or how their model will scale as their featureset grows wide relative to your dataset (in general, not well).
>>16261883Also, to answer the question about the prereqs that I didn't really answer here >>16261951Unless you actually want to be a data structures/algo-man, I'd avoid the CS side of things if you can and focus on the probabilistic modeling. There's more interesting things there, and also graphical models and state machines are a fucking retarded way to look at statistical inference unless you are specifically working on something like a stationary Markov/autoregressive model. You'll need to get comfortable with linear algebra from a mechanical side of things. You don't really need linear systems (though they help quite a lot) but you should understand everything done in the linear algebra portions of a linear control theory book like Modern Control Theory by Brogan. A ton of modern ML methods are based on adaptive filtering techniques, so learning the basics of that stuff could help you quite a lot as well as teach you the mechanics of linear algebra for probabilistic systems. Kevin Murphy's Probabilistic Machine Learning Vol 1 (free draft copy on his website and some code there too if I'm remembering correctly) is specifically written to be friendly to undergrads and covers an absolute shit ton of the fundamentals without ever being too challenging.
thanks first and foremost for your time to type this long response m8.There's a boat load of areas that you mentioned, such as control theory, that I've no idea about. Nevertheless, I shall use this reference as I keep completing the bare minimum things.>>16261959>You'll need to get comfortable with linear algebra from a mechanical side of things.you mean as to how they're applied in EE, specifically control theory, rather than focussing on abstract proofs ? Speaking of EE, do I now need to study circuits, physics for EE, and stuff of that kind b4 hopping on to control theory or linear systems? What about signal processing?>kevin murphy:probablistic machine learning intro>undergrad friendlywell, the preface does say it aims to be as self contained, but just skimming through the univariate probability section is frightening given it dives straight into probability distributions and employs all kinds of fancy math notation. Would it be advisable to actually complete calc 1-3, probability(sheldon ross first course in probability?), strang's and/or axler's linear algebra, some other good text on intro to stats, before trying my luck with this text?>>16261951>two approaches of AI/ML>discrete algo way and applied math wayis it even possible & useful to think along the lines of unifying these two approaches? Would it help trying learn both the ways? I'll be learning DS & algo as part of my course requirement and other areas of computer science like databases.
>>16262075wtf is a pseudoneet?
>>16262715barely studying, which is a bad habit I've started eliminating
>>16262724So you are enrolled and paying for an education you are not taking advantage of? Sounds like financial planning in Murrifattland.
>>16262725>So you are enrolled and paying for an education you are not taking advantage of?the classes have not even began yet.
taking my stat101 final in an hour...
>>16257098what do you need to know?
>>16262967I take any tips. I think it's an interesting field when combined with demographics, various political policy suggestions and sensitivity analysis.
>>16262075You don't really need circuits and don't really need a lot of control theory either, but linear system theory will be helpful. Modern Control Theory by Brogan does rely on some amount of assumptions, but really you don't need it. I'd say all of what you need is in the "linear algebra" part of the book. Chapter 3 through 8 of that book will have some examples based on circuits, but you definitely do not need to have an indepth understanding of electrical engineering to read and understand them. The concept of state is every important to time-series models regardless of field and this book covers this well. Mostly what you need is the ability to be comfortable with functional transformations of matrix and vector quantities and eigenvalues based analysis of linear algebraic systems. Almost everything in the world of "linear regression" is just some projection matrix like you'd see in Ch. 3 of Brogan. I've worked in EE doing this for almost 6 years now and have never once done anything researchwise involving a circuit in any meaningful capacity. > Is it even possible to think along the lines of unifying these two approaches.In some sense Bayesian mixture approaches do exactly this by having either a mixed discrete set of weights for individual component distributions (which can be either continuous or discrete) or having a graph like in interacting model mixture approaches to information fusion.
>>16256996>probability and statisticsDo you guys think that 1 out of 2 is magically 33% or 66% if you add a convoluted word problem to it?
>>16263072Yes, we do, because we know how to read and understand that observing one of the three choices to be wrong doesn't change that there were still 3 choices to begin with.
>>16263068>what you need is in the "linear algebra" part of the book. Chapter 3 through 8 of that book will have some examples based on circuits, but you definitely do not need to have an indepth understanding of electrical engineering to read and understand them. ok, but just in case, high school or freshman level electricity & magnetism would suffice right?>Mostly what you need is the ability to be comfortable with functional transformations of matrix and vector quantities and eigenvalues based analysis of linear algebraic systems. Almost everything in the world of "linear regression" is just some projection matrix like you'd see in Ch. 3 of Brogan.do you think any addtional prereq is necessary for this apart from the prereq mentioned in the preface that states only a first course in LA is required(again ,would something like G strang suffice for this?)?I must apologize for pestering you on the minutia of details that will obviously seem dumb to you m8. I ask all of this because I will never be able to take these hard core math classes and I would be forced to learn all of this on my own. Since I'm quite new to the enterprise of hardcore math, I feel a bit lost and have to use the type of material covered in the standard books I'd asked about in previous posts such as sheldon ross' intro to probability, as a reference point .Btw, were you well acquainted with LA, prob stats and calculus & analysis(?) before you used kevin murphy's probabilistic machine learning?
>>16263106You don't need to apologize over small things. Yes, high school EM is fine. If you want to go through the basics of a linear circuits text it wouldn't hurt but it isn't strictly necessary. If you really want to understand the circuits and mechanics part of things you could look at Ch. 2 of Nise's Control Systems Engineering textbook but I don't think it is actually necessary for your purposes. What matters for your ML stuff is understanding how linear models work, not how to develop a linear model for a circuit or some Newtonian dynamic system. I don't think any additional prereq is required beyond very basic familiarity with an applied linear algebra textbook like Strang or Lay. Brogan is pretty hand-holdy in that regard and each chapter ends with a ton of worked out "illustrative problems."Similarly, Murphy's book has a decent number of examples and worked out problems as well that are more directly related to linear algebra. I know the notation might look scary at first, but just give yourself some patience. If you need to, spend some time with a reference book on undergraduate probability theory itself separate from the ML book. There's a ton of these that are all pretty good, but Sheldon Ross's undergrad level "Introduction to Probability and Statistics for Engineers and Scientists" is free and fairly alright for the undergrad level prereqs Murphy's book assumes some familiarity with. If you feel lost in Murphy's univariate probability chapter, try skimming for the relevant section in Ross. Overall message: patience with yourself will pay off in the end. I know it's hard not to freak out about these things, but you need to find the ability to forgive yourself for "being stupid" or else you'll never get to "be smart" because you won't get out of your own way.
>>16263354*more directly related to ML, not linear algebra. Oops.
Statistics is not real. If your conclusion hangs on the assumption "as the number of samples approaches infinity" then your conclusion is fucking illegitimate. Nobody can count to infinity.
>>16263374You are retarded and should probably actually spend some time trying to learn how "convergence" as a concept works before spouting your nonsense.
>>16263374Bell curves only exist because they were looking for a pattern and found a loose-fitting one. Bell curves are a human mind-biased representation of reality, and do not actually have any meaningful application in the real world.
>>16263397> Bell-curves are a human mind-based representation of reality, and do not actually have any meaningful application in the real world.The central limit theorem would be true as a consequence of repeated convolution regardless of whether we were looking for a trend or even had ever discovered it. Nothing about the central limit theorem requires anything to do with the human mind. It's just math and integrals.
>>16263417>repeatedOnce again, this requires a supertask of repeating something an infinite number of times. Which, newsflash, cannot be done. Will never even be close to possibly being accomplished. Not by the end of human history, not even before the heat death of the universe.Go on and tell me that the central limit theorem will not assume some bimodal, trimodal, hell, some starfish-shaped structure when you take 10^10^10^10^10 samples. Tell me and show me. The latter cannot be done so the former can never be true. You can go ahead and assume it's true, and then I can go ahead and assume you are a dumbass.
>>16256996https://youtu.be/Q2Ve--JatAs?si=RNXzXQ8jaHO42IOT
>>16263074lol, fool.
>>16263434Again, you should actually spend a few moments trying to learn about how convergence works within real analysis rather than just going off of your ignorant and baseless assumptions. The whole point of asymptotic/limit based approaches is that you don't need to actually infinitely sample or infinitely repeat operations if the fundamentals that generate the sequence remain stationary. Literally the point of asymptotic analysis is that we can't infinitely sample so we need to know what the "truth" would converge to if we could as a means of testing how good or bad our statistical outcome is when we terminate the operation in finite time. I understand that you're completely retarded and probably entirely ignorant of real analysis (and probably a finitist from your response) but if you actually take a few minutes to ask "but what if I'm wrong" you might genuinely learn something. >>16263581Nice rebuttal retard.
>>16263354
>>16257607Every actuary is someone who liked math as a kid but had no idea what JOB you get with a math degree because they had a poor background without access to people with an elite education. Physics majors are people who want most people to think they’re good at math because most people will never meet a math major in their life.
>>16260484In the same way that Calculus is the bridge from early high school math to college math, real analysis is the bridge from early college math to graduate level math.
Night train bump
>>16264146Literally me. What other job are you even supposed to get with a math degree? Anyway im an actuary in my 20s making more than enough money to live in nyc so I cant complain
>>16265245With a math degree you can be go into any line of work that is primarily quantitative and doesn’t require a license, so that translates to a lot of tech jobs and finance jobs plus some engineering adjacent jobs and if you really want to swing into social science jobs they’re always happy to have you.
>>16265245I've got some question for you actuary anon.>do you folks incorporate machine learning? >what does your industry think on the current AI/ML hype?>Is there an overproduction of actuaries with very little job openings?>how much papers does one need to clear in the IFoA exams to get that first actuary related job that pays decently?
>>16265193ty anon
>>16263397Holy midwits in this thread.You have it backwards.Bell curves actully ONLY apply to biology driven things.This is why when Wall Street used things a like bell curves on investment productions they got blown the fuck out, Becuase investment results are not normally distributed.Genetics are, (well most things genetic) when you limit to a shared inheritance, and control everything else. This goes for simple stuff like height and complex stuff like IQ.It one 100% does not apply everywhere, this is a huge misunderstanding
>>16265692Bell curves apply to more than living things. One one side of the spectrum, robotics and process, the curves are narrowed down to equipment tolerances - see metrics like Cpk. On the other end of the spectrum, like the stock market, the bell curve is ginormous with little sample data. If you can't quantify the underlying commonality, then there is no way to pull the distribution out of the noise.Take a simple example baseball. When a ball is hit into the outfield it has a preferred side of the field it goes on. If you were to ignore left and right hand batters, it would look like a multi-modal distribution depending on the ratio of batters.So now to create the analogy with investment. 1. Every investor is a batter2. The ball is sum market forces and strategies.3. The investor swings at the ball when he opens a position.4. Investment results would be where the ball ends up over a number of at bats or time frame.It is very obvious why "investment results" would look like noise. The investor has thousands of different ways to enter the market, within each different financial instrument. Then there are thousands of different strategies analogous to batter posture / handedness. Unlike baseball, where the ball and the bat are of fixed proportions, results of a hit are proportional to a whole host of factors including batter risk tolerance, bankroll, etc. So when we are trying to gauge investment results across a whole group of hitters, we have to normalize all of this into a coherent game where ball distance actually means something. Basically, you fell for financial voodoo. They are incompetent in pretty much every regard and they want you to believe its because it is magic out there, when it is really criminal enterprise.
>>16265436>do you folks incorporate machine learning? Not in my company, but it is part of the curriculum now. There's a lot of talk about it and Im certain that some companies do use it. ML really is a subset of linear modeling in general and similar principles apply.>what does your industry think on the current AI/ML hype?Overall tilts to the "it's a bit overhyped" side.>Is there an overproduction of actuaries with very little job openings?Not in my experience. I applied to less than 100 jobs, got 5 interviews and 3 offers in about 6 weeks of job hunting.>how much papers does one need to clear in the IFoA exams to get that first actuary related job that pays decently?YMMV with IFOA, but im CAS and it took me 3 exams. Really the first two are the "prerequisites".
>>16265854how come this field isn't inundated with like software engineering or data science?>YMMV with IFOA??
>>16265866>withpajeets and other immigrants?
>>16265866Your mileage may vary
>>16265866Because actuaries are protected by a professional organisation aka a white collar union
>>16265903maybe, but it may also be because a lot of people may not have heard about it before. Things may soon change anon. I say this because this enterprise has a well structured exam system, something which asians are deeply fond of.
>>16261549Principal component analysis.
>>16265815> the bell curve is ginormous with little sample data. If you can't quantify the underlying commonality, then there is no way to pull the distribution out of the noiseWrong. It simply doesn’t matter how much data you have, Becuase future results are not based or even estimate-able on prior performances. There is no connection between past and future. Bell curves Simply.DoNotApply.To investments,I didn’t read the rest of your post, the central limit theorem and its use cases is a deeply misunderstood thing.It apples to genetics Becuase genetics are past on, and (you are right not just biology) to tolerances and such, where things are being reproduced with QC.It simply doesn’t’t apply to most of human life thoughl
Are there any good texts on adaptive experiments? I really find them fascinating as a family of experimental methods. >>16260785It's super easy, just realize they're fundamentally the same except with Bayesian inference you are assuming a part or more of your model is fundamentally random and not just the data. Or, if you want it from an effect standpoint, if you have some amount of information going into an analysis, even if it's just bounds, Bayesian methods let you abuse that information. In truth, you can and should know when to use either. It's literally just switching between stochastic vs. deterministic models and abusing prior information vs going in blind.
>>16266972Yes they do. Lmao, the most basic financial model is a gaussian random walk. Gaussian as in BELL CURVE. Which is also just Brownian motion of particles. The gaussian pops up all over the place. It's not the be all end all, but a lot of it's properties are fundamentally special and derived from the fact that it's generating PDF is e^(-x^2) which has a crap ton of nice features.
>>16266995You don’t understand what you think you understand, It only fits a Gaussian till it doesn’t, it’s not predictive in the way you think it appears to be
>>16267336I agree with this, we can model past performance according to this, but future performance cannot and will not be gaussian. It would be preposterous to think this.
Hello /psg/! have an undergrad degree in a non-statistics field but really wanna learn and study stats.Should I just youtube/read some books or would getting a second undergrad degree or masters be the better step?
>>16267424Definitely don't get a second undergrad degree. That's a massive waste of time. Stats is barely a thing until the master's level anyways, so I'd say just learn some basics (and if you need to, spend some time studying the mathematics needed if you aren't strong in that).
>>16257622nobody says this
>>16266995>Lmao, the most basic financial model is a gaussian random walkIt's not used in practice
>>16267815NTA, but I'm not sure if I believe you. Black-Scholles equations are the basis of the modern theory of investment, and most Black-Scholes models are literally driven by a geometric autocorrelation Gaussian random walk.
>>16259545What is disc rot?
>>16267653In some EU countries you only need bachelors even for pretty good jobs.
I just received an offer to pursue an undergraduate degree in Stats and Operations Research at my college for a heavily subsidized price, it's going to cost me about $10k to complete the degree. I already have a computer science and economics degree, but i found the econ side quite lacking, I really enjoyed taking econometrics, stats and discrete structures. Do you guys think it's worth taking this offer or should i just go apply for jobs?
>>16268122Theory is one thing, practice is another. Pricing options doesn't work like this anymore. Tails are skewed so badly you can't accurately model them with a random walk.
>>16268779What rank is the uni and do you have connections in stats/operations research?
>>16266023Redpill me on PCA
>>16268958Can you explain more? I'm definitely not a pricing/options kind of person.Is it tailed because the distribution is non-stationary? Conditional dependencies? Insufficient sampling relative to the conditional entropy rate of the process? It is interesting to me to hear about these practicalities because I am very much in an academic bubble.
>>16270075NTA, but do you know about eigenvalues/eigenvectors? The principle components/features are just the eigenvectors of the fundamental covariance matrix for the generating distribution. PCA just looks to identify and separate these eigenvectors using observational data as evidence.
>>16270075Given samples from a multivariate distribution, it aims to find a standardised linear combination of the variables that best represents the data. So, a simple example of standardised linear combination is taking sample mean of one variable, but that would ignore all the other variables. We instead want to find a weighted combination of the different variables that best represents the data. One way to do this, is just find the linear combination that minimises the Euclidean distance between the points and the combinations. This is done by the 4th figure. Alternatively, we can find the one which has maximal variance (since we want to include that variables which most distinguishes between the observation). Coincidentally, at least for multivariate normal, this leads to the same thing. The second principal component does the same thing but with the constraint that it must be uncorrelated to the previous combination. And then, third and so on and so forth. So, in practice, if you have a data with maximum, minimum and average temperature of day (obviously very correlated). The first principal component would most likely be the average, and the second would be the difference in maximum and minimum i.e., the range.
>>16268968top 100 world i don't have any connections.
>>16270293It's risky but if you get very high grades, try to network your ass off and leverage what you have at your current job, the pivot can be done.
>>16270094nta but this is a pretty good video on fat tails: https://www.youtube.com/watch?v=IsHNb-4IxDY
>>16268739Lots of my bio friends got employed as a data scientist. They only took statistics as a minor subject
>>16270094it's because a random walk just doesn't describe how people actually behave in a market, it describes a model which is too simplehere's an example: if you've got high volatility in the pricing then it tends to be the case that you get people who are more willing to buy at the tail end of your distribution for various reasons (they will benefit strongly from a move in one direction/more cautious of a loss than a gain/something like this)so in practice a lot of people kind of abandon that when pricing options, and certain options market makers basically consider the whole stochastic calculus method of pricing options to be worthless and do it entirely based on statistical models instead, and it works very well for them
>>16272617>high volatilityyou can probably guess but this is just finspeak for "large variance"
>>16272617You ever read "An Engine Not a Camera"? It's an interesting history of options pricing. He argues that the development of BS did more to shape the way options are traded than it did to dispassionately describe the way options behave. Cool book.Now if only I could somehow break me off a piece of that by getting a job
>>16272618>higher moments always exist, they're just sometimes infinite
>>16256996Reminder that American stats programs have fueled AI genocide and to divest in Israel
If I start flipping a coin and.flip it one time every one trillion years for all eternity, is the probability that I will eventually land on heads 1:1 or .9...:1?
>>16272568If you have domain knowledge in CS, economics or business it's even easier.
>>16273517AI what now? And on who?
>>16270075It actually kind of sucks. People like it because it captures maximum variance along the first components but that only characterizes the distribution projected onto the component well if that distribution is gaussian. Mean squared error isn't always the best thing to reduce. You want to capture the signal you want to capture, not simply reduce mse. You have to ask, "what is that signal and how do I get it out"?
>>16276847PCA works on any elliptical distribution.
>>16276847PCA is also misused a lot
>>16268607kek
is poker a good way to get into probability? it could be a fun way to apply it
>>16272617can you recommend any resources to learn about statistical market making?
>>16280487Yeah, card counting is so fun it's illegal.
>>16280487update: it wasn't
hello, this is a paper I'm reading: https://franknielsen.github.io/entropy-22-01100-v2.pdf
>>16256996Night train bump
>>16282142He also protects you from true enemy evil.
