- Wonder Woman 1984
- Matematika dan arsitektur
- Information gain ratio
- Information gain (decision tree)
- Kullback–Leibler divergence
- Omega ratio
- Decision tree learning
- C4.5 algorithm
- List of statistics articles
- Golden ratio
- Outline of machine learning
- Decibel
- Information gain ratio - Wikipedia
- Information Gain, Gain Ratio and Gini Index - Tung M Phung's Blog
- Decision Trees. Part 3: Gain Ratio | by om pramod | Medium
- Information Gain, Gini Index, Entropy and Gain Ratio in Decision …
- Understanding Information Gain in Decision Trees: A Complete …
- Information Gain and Mutual Information for Machine Learning
- How to calculate Gain Ratio – Data and Machine by viswateja
- A Simple Explanation of Information Gain and Entropy
- Information Gain Versus Gain Ratio: A Study of Split Method …
- Entropy, information gain, and Gini impurity(Decision tree splitting ...
information gain ratio
Information gain ratio GudangMovies21 Rebahinxxi LK21
In decision tree learning, information gain ratio is a ratio of information gain to the intrinsic information. It was proposed by Ross Quinlan, to reduce a bias towards multi-valued attributes by taking the number and size of branches into account when choosing
an attribute.
Information gain is also known as mutual information.
Information gain calculation
Information gain is the reduction in entropy produced from partitioning a set with attributes
a
{\displaystyle a}
and finding the optimal candidate that produces the highest value:
IG
(
T
,
a
)
=
H
(
T
)
−
H
(
T
|
a
)
,
{\displaystyle {\text{IG}}(T,a)=\mathrm {H} {(T)}-\mathrm {H} {(T|a)},}
where
T
{\displaystyle T}
is a random variable and
H
(
T
|
a
)
{\displaystyle \mathrm {H} {(T|a)}}
is the entropy of
T
{\displaystyle T}
given the value of attribute
a
{\displaystyle a}
.
The information gain is equal to the total entropy for an attribute if for each of the attribute values a unique classification can be made for the result attribute. In this case the relative entropies subtracted from the total entropy are 0.
Split information calculation
The split information value for a test is defined as follows:
SplitInformation
(
X
)
=
−
∑
i
=
1
n
N
(
x
i
)
N
(
x
)
∗
log
2
N
(
x
i
)
N
(
x
)
{\displaystyle {\text{SplitInformation}}(X)=-\sum _{i=1}^{n}{{\frac {\mathrm {N} (x_{i})}{\mathrm {N} (x)}}*\log {_{2}}{\frac {\mathrm {N} (x_{i})}{\mathrm {N} (x)}}}}
where
X
{\displaystyle X}
is a discrete random variable with possible values
x
1
,
x
2
,
.
.
.
,
x
i
{\displaystyle {x_{1},x_{2},...,x_{i}}}
and
N
(
x
i
)
{\displaystyle N(x_{i})}
being the number of times that
x
i
{\displaystyle x_{i}}
occurs divided by the total count of events
N
(
x
)
{\displaystyle N(x)}
where
x
{\displaystyle x}
is the set of events.
The split information value is a positive number that describes the potential worth of splitting a branch from a node. This in turn is the intrinsic value that the random variable possesses and will be used to remove the bias in the information gain ratio calculation.
Information gain ratio calculation
The information gain ratio is the ratio between the information gain and the split information value:
IGR
(
T
,
a
)
=
IG
(
T
,
a
)
/
SplitInformation
(
T
)
{\displaystyle {\text{IGR}}(T,a)={\text{IG}}(T,a)/{\text{SplitInformation}}(T)}
IGR
(
T
,
a
)
=
−
∑
i
=
1
n
P
(
T
)
log
P
(
T
)
−
(
−
∑
i
=
1
n
P
(
T
|
a
)
log
P
(
T
|
a
)
)
−
∑
i
=
1
n
N
(
t
i
)
N
(
t
)
∗
log
2
N
(
t
i
)
N
(
t
)
{\displaystyle {\text{IGR}}(T,a)={\frac {-\sum _{i=1}^{n}{\mathrm {P} (T)\log \mathrm {P} (T)}-(-\sum _{i=1}^{n}{\mathrm {P} (T|a)\log \mathrm {P} (T|a)})}{-\sum _{i=1}^{n}{{\frac {\mathrm {N} (t_{i})}{\mathrm {N} (t)}}*\log {_{2}}{\frac {\mathrm {N} (t_{i})}{\mathrm {N} (t)}}}}}}
Example
Using weather data published by Fordham University, the table was created below:
Using the table above, one can find the entropy, information gain, split information, and information gain ratio for each variable (outlook, temperature, humidity, and wind). These calculations are shown in the tables below:
Using the above tables, one can deduce that Outlook has the highest information gain ratio. Next, one must find the statistics for the sub-groups of the Outlook variable (sunny, overcast, and rainy), for this example one will only build the sunny branch (as shown in the table below):
One can find the following statistics for the other variables (temperature, humidity, and wind) to see which have the greatest effect on the sunny element of the outlook variable:
Humidity was found to have the highest information gain ratio. One will repeat the same steps as before and find the statistics for the events of the Humidity variable (high and normal):
Since the play values are either all "No" or "Yes", the information gain ratio value will be equal to 1. Also, now that one has reached the end of the variable chain with Wind being the last variable left, they can build an entire root to leaf node branch line of a decision tree.
Once finished with reaching this leaf node, one would follow the same procedure for the rest of the elements that have yet to be split in the decision tree. This set of data was relatively small, however, if a larger set was used, the advantages of using the information gain ratio as the splitting factor of a decision tree can be seen more.
Advantages
Information gain ratio biases the decision tree against considering attributes with a large number of distinct values.
For example, suppose that we are building a decision tree for some data describing a business's customers. Information gain ratio is used to decide which of the attributes are the most relevant. These will be tested near the root of the tree. One of the input attributes might be the customer's telephone number. This attribute has a high information gain, because it uniquely identifies each customer. Due to its high amount of distinct values, this will not be chosen to be tested near the root.
Disadvantages
Although information gain ratio solves the key problem of information gain, it creates another problem. If one is considering an amount of attributes that have a high number of distinct values, these will never be above one that has a lower number of distinct values.
Difference from information gain
Information gain's shortcoming is created by not providing a numerical difference between attributes with high distinct values from those that have less.
Example: Suppose that we are building a decision tree for some data describing a business's customers. Information gain is often used to decide which of the attributes are the most relevant, so they can be tested near the root of the tree. One of the input attributes might be the customer's credit card number. This attribute has a high information gain, because it uniquely identifies each customer, but we do not want to include it in the decision tree: deciding how to treat a customer based on their credit card number is unlikely to generalize to customers we haven't seen before.
Information gain ratio's strength is that it has a bias towards the attributes with the lower number of distinct values.
Below is a table describing the differences of information gain and information gain ratio when put in certain scenarios.
See also
Information gain in decision trees
Entropy (information theory)
References
Kata Kunci Pencarian: information gain ratio
information gain ratio
Daftar Isi
Information gain ratio - Wikipedia
In decision tree learning, information gain ratio is a ratio of information gain to the intrinsic information. It was proposed by Ross Quinlan, [1] to reduce a bias towards multi-valued attributes by taking the number and size of branches into account when choosing an attribute. [2]
Information Gain, Gain Ratio and Gini Index - Tung M Phung's Blog
Information Gain, Gain Ratio and Gini Index are the three fundamental criteria to measure the quality of a split in Decision Tree. In this blog post, we attempt to clarify the above-mentioned terms, understand how they work and compose a guideline on when to use which.
Decision Trees. Part 3: Gain Ratio | by om pramod | Medium
Jan 29, 2023 · Gain Ratio is an alternative to Information Gain that is used to select the attribute for splitting in a decision tree. It is used to overcome the problem of bias towards the...
Information Gain, Gini Index, Entropy and Gain Ratio in Decision …
What is Gain Ratio? Proposed by John Ross Quinlan, Gain Ratio or Uncertainty Coefficient is used to normalize the information gain of an attribute against how much entropy that attribute has. Formula of gini ratio is given by . Gain Ratio=Information Gain/Entropy
Understanding Information Gain in Decision Trees: A Complete …
Sep 29, 2024 · Information gain solves this by helping the model choose the right attribute for each split, ensuring it stays on the optimal path. In this blog, I’m going to take you through everything you...
Information Gain and Mutual Information for Machine Learning
Apr 15, 2024 · Information Gain (IG) is a measure used in decision trees to quantify the effectiveness of a feature in splitting the dataset into classes. It calculates the reduction in entropy (uncertainty) of the target variable (class labels) when a particular feature is known.
How to calculate Gain Ratio – Data and Machine by viswateja
Jul 10, 2018 · Gain Ratio is modification of information gain that reduces its bias. Gain ratio overcomes the problem with information gain by taking into account the number of branches that would result before making the split.It corrects information gain…
A Simple Explanation of Information Gain and Entropy
Jun 7, 2019 · Information Gain is calculated for a split by subtracting the weighted entropies of each branch from the original entropy. When training a Decision Tree using these metrics, the best split is chosen by maximizing Information Gain.
Information Gain Versus Gain Ratio: A Study of Split Method …
Quinlan's ID3, an early decision tree learner, initially used the information gain split metho d. But Quinlan disco v ered that information gain sho w ed unfair fa v oritism to ard attributes with man y outcomes. Consequen tly , gain ratio later b e- came the default split metho d.
Entropy, information gain, and Gini impurity(Decision tree splitting ...
Dec 6, 2022 · Information gain is the difference between before and after a split on a given attribute. It measures how much information a feature provides about a target. Constructing a decision tree is solely about finding a feature that returns the highest information gain.