假設檢定
在統計分析中,假設檢定是一個強大的工具,用來判斷結果的統計量是否充分,以及結果是否隨機。MLlib目前支持Pearson卡方檢定() 來檢定適配度和獨立性。輸入數據類型決定了是否產生適配度或獨立性,適配度檢定需要Vector輸入類型,而獨立性檢定需要一個Matrix矩陣輸入。
MLlib也支持RDD[LabeledPoint]輸入類型,然後使用卡方獨立性檢定來進行特徵選擇。
Statistics提供了進行Pearson卡方檢定的方法。下面示例演示了怎樣運行和解釋假設定。
import org.apache.spark.SparkContext
import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.stat.Statistics._
val sc: SparkContext = ...
val vec: Vector = ... // a vector composed of the frequencies of events
// compute the goodness of fit. If a second vector to test against is not supplied as a parameter,
// the test runs against a uniform distribution.
val goodnessOfFitTestResult = Statistics.chiSqTest(vec)
println(goodnessOfFitTestResult) // summary of the test including the p-value, degrees of freedom,
// test statistic, the method used, and the null hypothesis.
val mat: Matrix = ... // a contingency matrix
// conduct Pearson's independence test on the input contingency matrix
val independenceTestResult = Statistics.chiSqTest(mat)
println(independenceTestResult) // summary of the test including the p-value, degrees of freedom...
val obs: RDD[LabeledPoint] = ... // (feature, label) pairs.
// The contingency table is constructed from the raw (feature, label) pairs and used to conduct
// the independence test. Returns an array containing the ChiSquaredTestResult for every feature
// against the label.
val featureTestResults: Array[ChiSqTestResult] = Statistics.chiSqTest(obs)
var i = 1
featureTestResults.foreach { result =>
println(s"Column $i:\n$result")
i += 1
} // summary of the test