roman10
Side Note: First Draft on Apr 1 2011.
This article is a follow up of the Part 2 Producing the Trimap.
4. Color Likelihood Propagation
The Trimap based on Iprob divides the video frame into foreground (F), background (B) and unknown (U) regions. MRF can be applied to classify the pixels in U.
In the MRF energy equation, all other terms can be calculated based on previous computation except for L(pi|F) and L(pi|B), which are the color likelihood of some reliable feature points in the video frame.
The color likelihood propagation technique first will identify reliable pixels from a frame based on feature selection and motion between two adjacent frames (itself and next frame). Once these pixels are identified, their color likelihood are calculated.
The calculation is based on foreground and background distribution, meaning the segmentation result is already known. Therefore we use previous segmentaed frame to estimate the color likelihood of current frame. In other words, a pixel p’s likelihood in current frame is assumed to be the same as the likelihood of its corresponding pixel q (based on optical flow to find the correspondence) in the previous frame.
The calculation is as below,
Let’s use foreground as an example. First, the mean color Fi and covariance of the foreground color distribution is calculated as above. where Wi is a regularization constant. wq is spatial Gaussian coefficient with σ = 8. Cq is the color vector for a pixel q. Ni is the known foreground colors within q’s neighborhood.
Then the foreground color likelihood of q is computed as,
This L(qi|F) is assumed to be the same as L(pi|F) in the current frame, which is what we’re looking for.
Based on background color distribution, L(pi|B) can also be calculated.
But how to select reliable feature and do optical flow in the first place?
Good Features to Track is a research paper talking about a technique of selecting reliable features to track in feature-based vision system. It is used to select good features.
Lucas-Kanade Optical Flow is a technique to compute motion between two frames.
These two techniques are not covered here, and there’s tons of material online. Fortunately, OpenCV library has a good implementation of these two techniques. And one can apply them by making very simple function calls.
This color likelihood is different from the part 1 color likelihood in the sense it based on local color distribution, as it’s computed against a pixel’s neighborhood Ni, where the color likelihood of part 1 is based on color histogram, which indicates the global distribution of colors.
5. To be Continued on Implementation
With this final piece, MRF can be applied to solve the energy equation. This completes the theory for Real-Time Bilayer Segmentation. Part 4 will start with implementation, it’s done with Matlab, OpenCV C++ libraries, and MRF C++ libraries.
Please download the implementation source code here.
If you enjoyed this article, please consider sharing it!
what I was looking for, thanks
I am building example.cpp on windows7 with VS2010. and the following are the error I get. What I am doing wrong? Please help me.
1>—— Build started: Project: MRF_TEST2, Configuration: Debug Win32 ——
1>Build started 23/02/2012 4:33:34 PM.
1>InitializeBuildStatus:
1> Touching “Debug\MRF_TEST2.unsuccessfulbuild”.
1>ClCompile:
1> All outputs are up-to-date.
1> MRF_TEST2.cpp
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall BPS::BPS(int,int,int,class EnergyFunction *)” (??0BPS@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall TRWS::TRWS(int,int,int,class EnergyFunction *)” (??0TRWS@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall MaxProdBP::MaxProdBP(int,int,int,class EnergyFunction *)” (??0MaxProdBP@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall Swap::Swap(int,int,int,class EnergyFunction *)” (??0Swap@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: void __thiscall MRF::optimize(int,float &)” (?optimize@MRF@@QAEXHAAM@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: double __thiscall MRF::totalEnergy(void)” (?totalEnergy@MRF@@QAENXZ) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: void __thiscall MRF::initialize(void)” (?initialize@MRF@@QAEXXZ) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall ICM::ICM(int,int,int,class EnergyFunction *)” (??0ICM@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function _wmain
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: __thiscall GCoptimization::GCoptimization(int,int,int,class EnergyFunction *)” (??0GCoptimization@@QAE@HHHPAVEnergyFunction@@@Z) referenced in function “public: __thiscall Expansion::Expansion(int,int,int,class EnergyFunction *)” (??0Expansion@@QAE@HHHPAVEnergyFunction@@@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual double __thiscall GCoptimization::dataEnergy(void)” (?dataEnergy@GCoptimization@@UAENXZ)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual double __thiscall GCoptimization::smoothnessEnergy(void)” (?smoothnessEnergy@GCoptimization@@UAENXZ)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual void __thiscall GCoptimization::setNeighbors(int,int,double)” (?setNeighbors@GCoptimization@@UAEXHHN@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall Expansion::optimizeAlg(int)” (?optimizeAlg@Expansion@@MAEXH@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual void __thiscall GCoptimization::clearAnswer(void)” (?clearAnswer@GCoptimization@@UAEXXZ)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual void __thiscall GCoptimization::setParameters(int,void *)” (?setParameters@GCoptimization@@UAEXHPAX@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “public: virtual char __thiscall MRF::checkEnergy(void)” (?checkEnergy@MRF@@UAEDXZ)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setData(double *)” (?setData@GCoptimization@@MAEXPAN@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setData(double (__cdecl*)(int,int))” (?setData@GCoptimization@@MAEXP6ANHH@Z@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setSmoothness(int,double,double)” (?setSmoothness@GCoptimization@@MAEXHNN@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setSmoothness(double (__cdecl*)(int,int,int,int))” (?setSmoothness@GCoptimization@@MAEXP6ANHHHH@Z@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setSmoothness(double *)” (?setSmoothness@GCoptimization@@MAEXPAN@Z)
1>MRF_TEST2.obj : error LNK2001: unresolved external symbol “protected: virtual void __thiscall GCoptimization::setCues(double *,double *)” (?setCues@GCoptimization@@MAEXPAN0@Z)
1>MRF_TEST2.obj : error LNK2019: unresolved external symbol “public: virtual __thiscall GCoptimization::~GCoptimization(void)” (??1GCoptimization@@UAE@XZ) referenced in function “public: virtual __thiscall Expansion::~Expansion(void)” (??1Expansion@@UAE@XZ)
1>D:\MRF_TEST2\Debug\MRF_TEST2.exe : fatal error LNK1120: 23 unresolved externals
1>
1>Build FAILED.
1>
1>Time Elapsed 00:00:00.99
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========