Best big data training institute in India

by Simran Aggarwal your beauty our passion

What Is Big Data?

IT guys and welcome back to this class Bazy machine learning in Python part. In this lecture, we going to build on our mathematical tools in preparation for Bazy an AB testing way back in an earlier section I talked about the Best big data training institute in India. I said that the frequencies paradigm is that when we measure parameters like the mean are like the click-through rate we are doing point estimates we sum up all the results divide by and that's the estimate. This was a problem because it didn't take into account how certain we were about those estimates. However, the solution was we could use the Central Limit Theorem to show that the confidence interval is approximately Yassine. And from there we can get an upper and lower bound on the percent confidence interval.

So the frequencies paradigm calculates the likelihood maximizes the likelihood with respect to the parameter in question. And that's how we get the maximum likelihood estimate for that parameter in busy statistics. We treat the parameter as a random variable. So it has its own distribution and the shape of that distribution will tell us how confident we are of any value for that parameter. In particular, we apply base rules to help us with this. We say that the distribution of the parameter given the data is equal to the probability of the big data Hadoop data given the parameter times the prior probability of the parameter divided by the probability of the data. Recall that we just think of the denominator as a normalizing constant since it doesn't depend on the parameter theta to remind you the three key pieces of the Bosnian equation all have a name.

Best Big Data training institute in India

The posterior which tells us our new belief about the parameter after collecting data the prior which reflects our old beliefs about the parameter and the likelihood which tells us how likely it is. We saw the data we collected given our old belief about the parameter. So you might be thinking well that's pretty easy. It's just multiplication and division but not so fast. We're not multiplying and dividing numbers here. These are all big data training agency, in particular, you'll recall that the thing on the bottom is the integral of the thing on the top overall settings of the parameter. Generally speaking this integral is either hard or impossible to solve. One solution to that is to use sampling methods like MC MC but we won't cover those in this course since we're going to arrive at a much more elegant solution.

The answer to our problem is conjugate Pryors. What does this mean? Well, it turns out that for a specific likelihood distribution and for a specific prior distribution when you solve Bayes equation the posterior distribution will be the same type of distribution as the prior. So let's do it an example of big data training. We know that our likelihood for a quick theory is Bernoulli. We also know that theta must be a parameter between and because the probability of getting a click Well if you browse through a list of probability distributions that can range from zero to one you might arrive at the beta distribution.

In particular, the PFA beta is fit to the power of a minus one time one minus theta to the power of B minus divided by the beta function of AB and the beta function can be defined in terms of the Gamma function and the gamma function is a generalized form of the factorial function. But it works for any real number now let's combine the likelihood and the prior using Bayes Rule and see if we can solve for the posterior so we use the proportionality symbol because we're not considering the normalization constant.

We can combine like terms. So everything that has stayed on the base goes together and everything that has one minus on the basic goes the other. And then we realize this has the same shape as the beta distribution. Remember that the normalization constant doesn't matter because it does not depend on theta in particular we can see that p of say that given X is also a beta distribution but the parameters are a prime and B prime were a prime and B prime can be defined in terms of all the X's we've collected or we can phrase it in terms of our Click through rate problem. One interesting fact about the beta distribution is that the mean is equal to eight divided by A-plus B which is exactly what the maximum likelihood estimate of the click-through rate would have been.

However, this solution is much more elegant because this distribution doesn't require an approximation. Another interesting fact arises when we analyze the variance of the data. So what's interesting about this well we know that the more data we collect the bigger a and b will become. Therefore as a and b get bigger the variance gets smaller. So this automatically follows the same behavior we observe with the approximated confidence interval which is that the variance shrinks as we get more data. One question you might have is well how do we choose the original in B.

Future Of Big Data Training

It turns out if you set a equals and B equals the beta distribution is equal to the uniform distribution. This makes a lot of sense because if you don't know anything about big data training theory prior to your experiment then all possible Klavier rates are equally probable. We call this a non-informative prior one important fact is that as we collect a large amount of data the influence of the parameters becomes negligibly small. So let's summarize what we just did.

We first recapped what it means to be frequentist and what it means to be busy in which we treat the parameter we are measuring as a random variable and give it a probability distribution. We identified a set of probability distributions that Bernoulli and the beta such that when you combine them using Bayes Rule the prior and the posterior are of the same type of probability distribution. We call these conjugate Pryors. If you're interested big data training Wikipedia has an article on conjugate Pryors that lists out all the likelihood prior pairs that go together.
I found it to be a very useful reference when doing Bazy and statistics. Lastly, we showed that the beta distribution naturally has the same mean as the maximum likelihood theory and that the variance automatically shrinks as you collect more data. Just like with the frequentist confidence interval.