Android LayerDrawable and Drawable.Callback

LayerDrawable is a Drawable that manages an array of Drawables, where each Drawable is a layer of it. It can be used to compose fancy visual effects. However, used incorrectly, it can also introduce difficult to catch bugs. This post discusses a bug that can be caused by the Callback mechanism of Drawable.

Chain of Callbacks

In LayerDrawable, every layer/drawable registers the LayerDrawable as its Drawable.callback. This allows the layer to inform the LayerDrawable when it needs to redraw. As shown in the invalidateSelf method from Drawable.java, the DrawableCallback.invalidateDrawable is called to inform the client (the LayerDrawable, in this case) the drawable needs to redraw.

public void invalidateSelf() {
    final Callback callback = getCallback();
    if (callback != null) {
        callback.invalidateDrawable(this);
    }
}

 

Also, a View registers itself as Drawable.callback, so when the drawable needs to redraw, the View can be informed and invalidated. So if we set background of a View to be LayeredDrawable, we have a chain of DrawableCallbacks. This is illustrated in the figure below.

1.jpg

Figure 1. Chain of Callbacks when Setting LayerDrawable as View’s Background

When the Drawable needs to redraw, it calls its callback.invalidateDrawable, which in turns calls LayerDrawable’s callback.invalidateDrawable, which causes the View to redraw the corresponding region if necessary.

View Breaks Callback of Old Background Drawable

So far all work well, but the setBackgroundDrawable of View.java has the following code snippet.

public void setBackgroundDrawable(Drawable background) {
    ...

    /*
     * Regardless of whether we're setting a new background or not, we want
     * to clear the previous drawable.
     */
    if (mBackground != null) {
        mBackground.setCallback(null);
        unscheduleDrawable(mBackground);
    }

    …
    if (background != null) {
        background.setCallback(this);
    }
    ...
}

This means setting a new background will break the old background drawable’s callback by setting it to null, regardless whether the callback is still set to the View.

The Bug

With all the knowledge above, we can “fabricate” a bug by follow the steps below.

  1. Set a Drawable A as a background of a View V. The A.callback is set to V.

  2. Create a LayerDrawable L with A as one layer. Now A.callback is set to L.

  3. Now set another Drawable (or just null) as background of V. V will set callback of its old background (A in our example) to null, which breaks the link between A and L.

  4. Updates A won’t trigger L to update.

The solution to this issue is to update the background of V before creating L. This is illustrated in the code example here.

In the example, we tried animating the actionbar background. The background consists of two layers, the first one is a green ColorDrawable, and second a bitmap drawable. We want to animate the alpha value of the bitmap drawable, so it appears the image is faded in.

The code contains two methods, one works and the other doesn’t due to the bug we described. Each method can be triggered by a button. Below we extract the key part of the code, one can refer to the link for more details.


Button btn1 = (Button) findViewById(R.id.button1);
btn1.setOnClickListener(new View.OnClickListener() {
  @Override
  public void onClick(View v) {
    // 1. This sets launcherIconDrawable.callback to actionBar
    actionBar.setBackgroundDrawable(launcherIconDrawable);
    animateActionBarWorking();
  }
});
Button btn2 = (Button) findViewById(R.id.button2);
btn2.setOnClickListener(new View.OnClickListener() {
  @Override
  public void onClick(View v) {
    // 1. This sets launcherIconDrawable.callback to actionBar
    actionBar.setBackgroundDrawable(launcherIconDrawable);
    animateActionBarNotWorking();
  }
});

private void animateActionBarNotWorking() {
	Drawable[] layers = new Drawable[] { colorLayer, launcherIconDrawable };
	// 2. This sets launcherIconDrawable.callback to layerDrawable
	LayerDrawable layerDrawable = new LayerDrawable(layers);
	// 3. This sets launcherIconDrawable.callback to null
	actionBar.setBackgroundDrawable(layerDrawable);
	ValueAnimator valueAnimator = ValueAnimator.ofInt(0, 255);
	valueAnimator.setDuration(1000);
	valueAnimator.addUpdateListener(new ValueAnimator.AnimatorUpdateListener() {
	  @Override
	  public void onAnimationUpdate(ValueAnimator animation) {
		// 4. Updates launcherIconDrawable will not trigger action bar background to update
		// as launcherIconDrawable.callback is null
		launcherIconDrawable.setAlpha((Integer) animation.getAnimatedValue());
	  }
	});
	valueAnimator.start();
}

private void animateActionBarWorking() {
	actionBar.setBackgroundDrawable(null);
	animateActionBarNotWorking();
}

 

References:

  1. LayerDrawable: http://developer.android.com/reference/android/graphics/drawable/LayerDrawable.html

Support Vector Machine Concept: VC Dimension

[latexpage]

Given a set of n samples $x_1, x_2, …, x_n$, we want to label them as either -1 or 1. In total, there are $2^n$ possible label combinations.

A class of learning machines H can be used to label the samples. If for each label combination, we can always find a learning machine $h in H$ that labels it correctly, we then say that H shatters n points.

VC (Vapnik-Chervonekis) dimension is then defined as the maximum number of points that can be shattered by H, which measures the capacity of the hypothesis class H.

Note that VC(H) = 4 does not mean H can shatter any 4 points in the hyperplane, as long as there’s 4 points can be shattered by H, it’s good enough. And it also infers that for any > 4 (say 5) points in the hyperplane, it cannot be shattered by H.

Now consider a simple example to find the VC dimension of a class of hypothesis H. H is defined as straight line in a two dimensional hyperplane.

We can find the following 3 points a, b and c where all $2^3$ possible label combinations can be classified correctly by H.

1

As mentioned before, it’s possible to find 3 points that H cannot shatter. For example, we just place a, b and c in the same line. There will be combinations that a straight line cannot label them correctly. But this doesn’t affect the VC dimension as it’s defined as the maximum number of points can be shattered.

One can try separate 4 points using a straight line, but no matter where we place those 4 points, there will be combinations that cannot be separated.

 

Naive Bayes

Bayes’ Theorem

Let’s start from Bayes’ theorem, also referred as Bayes’ law or Bayes’ rule.

P(A|B) = P(B, A) / P(B)

= P(B|A) * P(A) / P(B)

= P(B|A) * P(A) / (P(B|A) * P(A) + P(B|^A) * P(^A))

P(A): prior probability. It’s the probability event A happens.

P(^A): the probability that event A not happen.

P(B): evidence, or background. The probability of event B happens.

P(B|A), P(B|^A): conditional probability, or likelihood. The probability of event B happens given A happened or not happened respectively.

P(A|B): posterior probability. The probability of A happens taking into account B for and against A.

Naive Bayes

When used for classification, Bayes’ Theorem can be expressed as below,

P(C|F1, F2, … Fn) = P(C, F1, F2, …, Fn) / P(F1, F2, … , Fn)

= P(F1, F2, … Fn|C) * P(C) / P(F1, F2, …, Fn)

C is some class/label we can classify a sample into, and F1, F2, … Fn represents features of the sample data.

P(F1, F2, …, Fn) doesn’t depend on C and are normally given or can be calculated based on probability of each feature. It’s effectively a constant and can be ignored for classification purpose.

The numerator can be expressed as following,

P(C, F1, F2 … , Fn)

= P(C) * P(F1, F2, … , Fn|C)

= P(C) * P(F1 | C) * P(F2, F3, … Fn | F1, C)

= P(C) * P(F1 | C) * P(F2 | F1, C) * P(F3, … Fn | F1, F2, C)

= P(C) * P(F1 | C) * P(F2 | F1, C) * P(F3 | F1, F2, C) * …  * P(Fn | F1, F2, …, Fn-1, C)

In Naive Bayes, all features are assumed to be independent. Thus Fi is independent from every other feature Fj where j != i. Therefore we have

P(F2 | F1, C) = P(F2 | C)

P(F3 | F1, F2, C) = P(F3 | C)

P(Fn | F1, F2, … Fn-1, C) = P(Fn | C)

Thus,

P(C, F1, F2 … , Fn) = P(C) * P(F1 | C) * P(F2 | C) * P(F3 | C), …, P(Fn | C)

For example, two authors A and B like to use words “love”, “life” and “money”. The probability of these words appears in A’s article is 0.1, 0.1 and 0.8, and in B’s as 0.5, 0.3 and 0.2. Now we have the phrase “love life”, which one of the author is more likely to have written that?

Without any information, there’s 50% percent probability for either A or B. Assuming the words are independent features, we can use Naive Bayes.

P(A | love, life) = P(A) * P(love | A) * P(life | A) / P(love, life) = 0.5 * 0.1 * 0.1 / P(love, life)

P(B | love, life) = P(B) * P(love | B) * P(life | B) / P(love, life) = 0.5 * 0.5 * 0.3 / P(love, life)

Clearly, it’s more likely that the phrase “love life” is written by author B. Note that P(love, life) is independent from the authors and just a scaling factor.

References:

  1. Bayes’ theorem: http://en.wikipedia.org/wiki/Bayes%27_theorem
  2. Naive Bayes classifier: http://en.wikipedia.org/wiki/Naive_Bayes_classifier
  3. Udacity, Intro to Machine Learning. Lesson 1. Naive Bayes.

Performance Metrics for Binary Classification

Binary classification classifies samples as either 0 (negative) or 1(positive). Depending on which class/label the sample data belongs to and its predication result, each sample fits in a cell of the following table.

predicted | actual positive negative
positive true positive (TP) false positive (FP)
negative false negative (FN) true negative (TN)

We can then calculates different metrics, which measures the classification results from different angles.

True Positive Rate (Sensitivity, hit rate, recall): Out of all the positive samples, what fraction is actually detected as positive.

TPR = TP / (TP + FN)

True Negative Rate (Specificity): Out of all the negative samples, what fraction is actually detected as negative.

TNR = TN / (TN + FP)

Positive Predictive Value (Precision): Out of all samples predicted as positive, what fraction is actually positive.

PPV=TP / (TP + FP)

Negative Predictive Value: Out of all samples predicted as negative, what fraction is actually negative.

NPV = TN / (TN + FN)

False Positive Rate (Fall out): Out of all negative samples, what fraction is detected as positive by mistake.

FPR = FP / (FP + TN) = 1 – TNR

False Discovery Rate: Out of all samples predicted as positive, what fraction is actually negative.

FDR = FP / (FP + TP) = 1 – PPV

Accuracy: Out of all samples, what fraction is predicted correctly. That is, positive samples are predicted as positive, negative samples are predicted as negative.

Accuracy = (TP + TN) / (P + N)

The table below gives a nice overall view of the metrics mentioned above,

predicted | actual

positive negative

positive

true positive (TP) false positive (FP)

PPV = TP / (TP + FP)

negative

false negative (FN) true negative (TN)

NPV = TN / (FN + TN)

TPR = TP / (TP + FN) TNR = TN / (FP  + TN)

Accuracy =

(TP + TN) / (P + N)

F1 score: the harmonic mean of precision and recall.

F1 =2 * TPR*PPV / (TPR + PPV) = 2TP / (2TP + FP + FN)

Matthews correlation coefficient: It takes account true and false positives and negatives, and is regarded as balanced measure. It can be used for cases where the number of samples at different classes vary drastically.

MCC = (TP * TN – FP * FN) / √(TP + FP)(TP + FN)(TN + FP)(TN + FN)

MCC returns a value between -1 to 1, where -1 indicates total disagreement between prediction and actual facts, 0 means no better than random guess, and 1 indicates perfect prediction.

References:

  1. Matthews correlation coefficient: http://en.wikipedia.org/wiki/Matthews_correlation_coefficient
  2. Sensitivity and specificity: http://en.wikipedia.org/wiki/Sensitivity_and_specificity

Ghost Object Causing Crash in Java

A Bad Code Style

Java’s GC has made our life easier, but it also makes things more implicit and sometimes easier to make mistakes. Recently I encountered a bad code style which is easy to introduce bugs.

Imagining you have a class with a reference field and a method. The method creates a new object and assign it to the reference. Every time you call the method, a new object is created and assigned to the reference. The code is essentially like below,


class A {
    SomeClass B;
    void initB() {
         B = new SomeClass();
         …… do something with B … …
    }
}

Just call initB many times, this pattern creates a lot of objects without referencing to them. Since we’re not referencing to them, GC will do us the favor of garbage collecting them. However, things can easily go wrong.

When we do something with B after creation of the object, the object can be referred at somewhere else (e.g.: B is a task, and do something adds it to a system task queue). Now we call initB many times, we’ll end up with lots of objects somewhere and only one referred by B. We may not have control over the non-referenced objects any more. This results in garbage.

Things can go worse. If SomeClass defines some callback method, which is triggered later by some signal. We may ends up having callback method from many instances of SomeClass.

An Example in Android

Below is an example in Android.


package com.example.androidtest;

import android.os.Bundle;
import android.os.Handler;
import android.app.Activity;
import android.util.Log;

public class MainActivity extends Activity {

  private static final String TAG = "MainActivity";
  
  private Handler handler;
  private Runnable runnable;
  private Integer value;
  
  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    value = 10;
    startRunnable();
    startRunnable();
    removeRunnable();
  }
  
  private void startRunnable() {
    handler = new Handler();
    runnable = new Runnable() {
      @Override
      public void run() {
        Log.d(TAG, value.toString());
      }
    }; 
    handler.postDelayed(runnable, 2000);
  }
  
  private void removeRunnable() {
    handler.removeCallbacks(runnable);
    value = null;
  }
  
}

 

The startRunnable method creates a new Handler and Runnable objects, assigns them to the class reference field. We called this method twice, leaving a Handler and Runnable unreferenced by us.

However, these objects are still referenced by some Android system components. The postDelayed method essentially adds the runnable to a message queue associated with the current thread (The actual details are more complicated than this but you get the idea).

We then call removeRunnable to remove the runnable associated with handler, and assign null to value.

However, this doesn’t remove the ghost runnable associated with the ghost handler. And the run() method of the ghost runnable will be triggered, which will cause a crash when it tries to access value.toString().

The Fix

This coding style is bad and easy to introduce bugs. We essentially only want a single instance of Handler and Runnable at any time. If we have to create a new instance, we should ensure the old one is discarded properly, and leaving them referenced by system components is not.

If the system provides some callback functions that are guaranteed to be called once and we can create the objects inside those functions.

If we need to create objects inside some random methods, we can either reuse the old object, or properly reset the old object before creating new one. The code below shows how to do that to fix the example above.


package com.example.androidtest;

import android.app.Activity;
import android.os.Bundle;
import android.os.Handler;
import android.util.Log;

public class MainActivityNew extends Activity {

  private static final String TAG = "MainActivityNew";
  
  private Handler handler;
  private Runnable runnable;
  private Integer value;
  
  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    value = 10;
    startRunnable();
    startRunnable();
    removeRunnable();
  }
  
  private void startRunnable() {
    if (handler == null) {
      handler = new Handler();
    } else {
      handler.removeCallbacks(runnable);
    }
    runnable = new Runnable() {
      @Override
      public void run() {
        Log.d(TAG, value.toString());
      }
    }; 
    handler.postDelayed(runnable, 2000);
  }
  
  private void removeRunnable() {
    handler.removeCallbacks(runnable);
    value = null;
  }
  
}

 

In startRunnable method, if a Handler instance is available, we’ll just reuse it; If there’re potentially old Runnable associated with handler, we’ll remove them.

Logistic Regression

This is some notes taken when I summarize the things learned after taking Andrew Ng’s machine learning course at coursera.

Introduction

Linear regression predicts continuous values. At times, we need to categorize things. Logistic regression is a probabilistic statistical classification model does that.

We will examine how logistic regression classify things to two categories (either 0 or 1) first, and then how it is used for multiple categories.

The logistic regression model can be described by the following logistic/sigmoid function below,

1

h(x) an be interpreted as the estimated probability that y = 1 on input x.

If theta’X >= 0, h(x) >= 0.5, we predict output y = 1

If theta’X < 0, h(x) < 0.5, we predict output y = 0

theta’X essentially describes the decision boundary.  Note that we can use  other values instead of 0.5 as the cutoff point if it is more suitable.

Cost Function

The cost function for logistic regression is defined as below,

3

The cost is further defined as,

5

We can merge the functions, and the cost function eventually becomes

4

With regularization, the cost function becomes,

6Note that j starts from 1 as a convention.

Gradient Descent

The gradient descent of logistic regression is identical to linear regression, except that h(x(i)) is different.

Multi-class Classification: One-vs-All

We can use one-vs-all technique to apply logistic regression to multi-class classification. The idea is to train a logistic regression classifier for each class i to predict the probability that y = i. Then we pick the category that has the maximum probability for an input.

Linear Regression

This is some notes taken when I summarize the things learned after taking Andrew Ng’s machine learning course at coursera.

Introduction

Regression is a technique to model relationships among variables. Typically, there’s one dependent variable y and one or many independent variables. This relationship is usually expressed as a regression function.

Linear regression, as the name suggests, models the relationship using a linear regression function. Depending on how many independent variables we have, we have simple linear regression with one independent variable and multivariate linear regression with more than one independent variables.

The hypothesis of linear regression can be described by the following equation,

linear_regression

The X are called features, and theta are the parameters. Given a set of training samples, we’ll need to choose theta to fit the training examples.

To measure how well we fit the training examples, we define the cost function of linear regression as below,

linear_regression_cost_function

m represents the number of training samples, h(x) is the predicted value and y is the sample output value. The cost function measures the average square error of all samples and then divide by 2.

This is essentially an optimization problem where we need to choose parameter theta such that the cost defined by the cost function is minimized.

Over-fitting and Regularization

Fitting the regression parameters minimize the error for training samples, however we can run into the problem of trying too hard such that the regression function doesn’t generalize well. i.e.: The hypothesis produce high error for input outside of the training set. This problem is known as overfitting.

Two commonly used techniques to address overfitting is reducing number of features and regularization.

Regularization adds an additional term to the cost function to penalize having large theta value, which tends to produce much more smooth curves.

linear_regression_cf_regularization

Note that by convention, the regularization term exclude j=0 case, which is theta 0.

Given the hypothesis and its cost function, there’re many ways to fit the parameter theta (i.e., solve the optimization problem), including conjugate gradient, BFGS, L-BFGS etc. The most commonly used technique is Gradient Descent.

Gradient Descent

The idea of gradient descent is to start at some random values, evaluate the cost. And keep iterating on theta value based on the function below to reduce the cost until we reach a minimal.

linear_regression_gradient_descent

The alpha is called the learning rate. It can be proven that if choose a sufficiently small alpha value, the cost will converge at some minimum. However, we don’t want alpha value to be too small in practice because it will take longer time. Typically, we try out a range of alpha values (0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1) and plot the cost to see how fast it converges.

For linear regression with regularization, the above equation is essentially the following,gd

The second term can easily be rewritten as,

gd2

Feature Scaling and Mean Normalization

When we do gradient descent, the values for different features normally differ in scale. For example, feature A may have value in the range of [1, 10], feature B varies from [-10000, 10000].

It’s good to have the feature values have similar scales and centered around 0 (i.e.: have approximately mean of 0).

The former can be achieved using feature scaling, just divide every value of that feature by a number such that the range is approximately [-1, 1]. The latter is accomplished using mean normalization (This doesn’t apply to X0). We can usually use (X – mean) to achieve this.

Numerical Analysis

Besides using optimization algorithms to fit theta iteratively, it turns out we can also compute the theta values numerically.

Without regularization, the numerical equation is as below,

linear_regression_numerical_analysis

While this method doesn’t need to choose learning rate and iterate, it is more computationally expensive as n get large because of the matrix multiplication and inverse. In addition, the inverse may not even exist. This is typically due to redundant features (some features are not linearly independent) or too many features too few samples.

With regularization, the numerical solution is the following,

linear_regression_numerical_analysis_2

Note that inverse part will exist even if the equation without regularization is not invertible.

OneClick Movie Maker Released

Here it is, OneClick Movie Maker is released. We can download it from Google Play at: https://play.google.com/store/apps/details?id=roman10.chocolate

The idea of this app is to make video editing easy with just a single click. Well, maybe multiple clicks eventually. The screenshot below illustrate the idea.

screenshot_phone_3

Figure 1. OneClick Movie Maker Video Effects

As shown above, we can apply a video effects/filters to videos by a single click. The video will starts playing with the effects automatically. Once we’re sure which effect/filter to apply, we click the top right button to trigger the video processing.

The processing is completely asynchronous. Actually, it runs on a remote process on its own. This means one can continue to play with the app without worrying about the video processing, which can take a long time if the video is big.

The app can handle multiple video editing easily. We can apply filters to as many videos as we want without waiting for the previous ones to finish. These video processing will be queued and finished one by one. Better still, the app provides an interface for one to manage the tasks pending, as shown below.

screenshot_phone_3

Figure 2. OneClick Video Maker: manages current tasks

Once the videos are processed, they’ll appear in processed page and videos page, where we can play and share the video.

I’ve made a demo video about the app. Hopefully it gives everyone a brief idea of how the app works.

 

The app is still in beta stage, I’ll add more functions, including adding frame borders, more effects/filters etc. Hope you’ll like it. 😉 The app can be downloaded from Google play OneClick Movie Maker.

How to Build ffmpeg with NDK r9

This is a updated post for a previous post, where we built ffmpeg 0.8 with Android NDK r5 and r6. This post will give instructions of how to build ffmpeg 2.0.1 with Android NDK r9.

0. Download Android NDK

The latest version of Android NDK can be downloaded at Android NDK website. At the time of writing, the newest version is NDK r9. Note that the website provides both current and legacy toolchains. We only need the current toolchain to compile ffmpeg.

After download NDK, simply decompress the archive. Note that we’ll use $NDK to represent the root path of the decompressed NDK.

1. Download ffmpeg source code

FFMPEG source code can be downloaded from the ffmpeg website. The latest stable release is 2.0.1. Download the source code and decompress it to $NDK/sources folder. We’ll discuss about the reason for doing this later.

2. Update configure file

Open ffmpeg-2.0.1/configure file with a text editor, and locate the following lines.

SLIBNAME_WITH_MAJOR='$(SLIBNAME).$(LIBMAJOR)'

LIB_INSTALL_EXTRA_CMD='$$(RANLIB) "$(LIBDIR)/$(LIBNAME)"'

SLIB_INSTALL_NAME='$(SLIBNAME_WITH_VERSION)'

SLIB_INSTALL_LINKS='$(SLIBNAME_WITH_MAJOR) $(SLIBNAME)'

This cause ffmpeg shared libraries to be compiled to libavcodec.so.<version> (e.g. libavcodec.so.55), which is not compatible with Android build system. Therefore we’ll need to replace the above lines with the following lines.

SLIBNAME_WITH_MAJOR='$(SLIBPREF)$(FULLNAME)-$(LIBMAJOR)$(SLIBSUF)'

LIB_INSTALL_EXTRA_CMD='$$(RANLIB) "$(LIBDIR)/$(LIBNAME)"'

SLIB_INSTALL_NAME='$(SLIBNAME_WITH_MAJOR)'

SLIB_INSTALL_LINKS='$(SLIBNAME)'

3. Build ffmpeg

Copy the following text to a text editor and save it as build_android.sh.

#!/bin/bash

NDK=$HOME/Desktop/adt/android-ndk-r9

SYSROOT=$NDK/platforms/android-9/arch-arm/

TOOLCHAIN=$NDK/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64

function build_one

{

./configure 

    --prefix=$PREFIX 

    --enable-shared 

    --disable-static 

    --disable-doc 

    --disable-ffmpeg 

    --disable-ffplay 

    --disable-ffprobe 

    --disable-ffserver 

    --disable-avdevice 

    --disable-doc 

    --disable-symver 

    --cross-prefix=$TOOLCHAIN/bin/arm-linux-androideabi- 

    --target-os=linux 

    --arch=arm 

    --enable-cross-compile 

    --sysroot=$SYSROOT 

    --extra-cflags="-Os -fpic $ADDI_CFLAGS" 

    --extra-ldflags="$ADDI_LDFLAGS" 

    $ADDITIONAL_CONFIGURE_FLAG

make clean

make

make install

}

CPU=arm

PREFIX=$(pwd)/android/$CPU 

ADDI_CFLAGS="-marm"

build_one

We disabled static library and enabled shared library. Note that the build script is not optimized for a particular CPU. One should refer to ffmpeg documentation for detailed information about available configure options.

Once the file is saved, make sure the script is executable by the command below,

sudo chmod +x build_android.sh

Then execute the script by the command,

./build_android.sh

4. Build Output

The build can take a while to finish depending on your computer speed. Once it’s done, you should be able to find a folder $NDK/sources/ffmpeg-2.0.1/android, which contains arm/lib and arm/include folders.

The arm/lib folder contains the shared libraries, while arm/include folder contains the header files for libavcodec, libavformat, libavfilter, libavutil, libswscale etc.

Note that the arm/lib folder contains both the library files (e.g.: libavcodec-55.so) and symbolic links (e.g.: libavcodec.so) to them. We can remove the symbolic links to avoid confusion.

5. Make ffmpeg Libraries available for Your Projects

Now we’ve compiled the ffmpeg libraries and ready to use them. Android NDK allows us to reuse a compiled module through the import-module build command.

The reason we built our ffmpeg source code under $NDK/sources folder is that NDK build system will search for directories under this path for external modules automatically. To declare the ffmpeg libraries as reusable modules, we’ll need to add a file named $NDK/sources/ffmpeg-2.0.1/android/arm/Android.mk with the following content,

LOCAL_PATH:= $(call my-dir)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libavcodec

LOCAL_SRC_FILES:= lib/libavcodec-55.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libavformat

LOCAL_SRC_FILES:= lib/libavformat-55.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libswscale

LOCAL_SRC_FILES:= lib/libswscale-2.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libavutil

LOCAL_SRC_FILES:= lib/libavutil-52.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libavfilter

LOCAL_SRC_FILES:= lib/libavfilter-3.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

 

include $(CLEAR_VARS)

LOCAL_MODULE:= libwsresample

LOCAL_SRC_FILES:= lib/libswresample-0.so

LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include

include $(PREBUILT_SHARED_LIBRARY)

Below is an example of how we can use the libraries in a Android project’s jni/Android.mk file,

LOCAL_PATH := $(call my-dir)

 

include $(CLEAR_VARS)

 

LOCAL_MODULE    := tutorial03

LOCAL_SRC_FILES := tutorial03.c

LOCAL_LDLIBS := -llog -ljnigraphics -lz -landroid

LOCAL_SHARED_LIBRARIES := libavformat libavcodec libswscale libavutil

 

include $(BUILD_SHARED_LIBRARY)

$(call import-module,ffmpeg-2.0.1/android/arm)

Note that we called import-module with the relative path to $NDK/sources for the build system to locate the reusable ffmpeg libraries.

For real examples to how to use the ffmpeg libraries in Android app, please refer to my github repo of android-ffmpeg-tutorial.

Android Animation Playback: Inline Animation with TextView

TextView allows us to attach and detach markup objects to a range of text using Spannable. We can use this to implement inline animation within TextView.

0. AnimatedImageSpan

We implement a AnimatedImageSpan, which extends DynamicDrawableSpan. The object will accept a set of image frames and keep track of the frame to display. A handler and a runnable are implemented to update the frame to display.

Note that AnimatedImageSpan doesn’t update the screen by itself. It calls AnimatedImageUpdateHandler.updateFrame to refresh the entire TextView, which subsequently redraw the AnimatedImageSpan with new image.

public class AnimatedImageSpan extends DynamicDrawableSpan {

    private AnimationAssetsSet mGifAssets;

    private int mCurrentFrameIdx;

    private Context mContext;

    

    private SimpleImageMemCache mImageCache;

    private AnimatedImageUpdateHandler mImageUpdater;

    

    private final Handler handler = new Handler();

    

    public AnimatedImageSpan(Context context) {

        mContext = context;

    }

    

    public void setImageCache(SimpleImageMemCache pImageCache) {

        mImageCache = pImageCache;

    }

    

    public void setAnimationAssets(AnimationAssetsSet pGifAssets) {

        mGifAssets = pGifAssets;

    }

    

    private Runnable mRunnable;

    private int mPlaybackTimes;

    private boolean mPlaying;

    public void playGif(final AnimationSettings pGifSettings, AnimatedImageUpdateHandler pListener) {

        mPlaying = true;

        mImageUpdater = pListener;

        mPlaybackTimes = 0;

        mRunnable = new Runnable() {

            public void run() {

                mCurrentFrameIdx = (mCurrentFrameIdx + 1)%mGifAssets.getNumOfFrames();

//                Logger.d(this, "current frame " + mCurrentFrameIdx);

                handler.postDelayed(this, pGifSettings.mDelay);

                if (null != mImageUpdater) {

//                    Logger.d(this, "update frame using listener " + mImageUpdater.getId());

                    mImageUpdater.updateFrame();

                }

                if (mCurrentFrameIdx == mGifAssets.getNumOfFrames() - 1) {

                    if (pGifSettings.mPlaybackTimes == 0) {

                        //repeat forever

                    } else {

                        mPlaybackTimes++;

                        if (mPlaybackTimes == pGifSettings.mPlaybackTimes) {

                            stopRendering();

                        }

                    }

                }

            }

        };

        handler.post(mRunnable);

    }

    

    public boolean isPlaying() {

        return mPlaying;

    }

    

    public void stopRendering() {

        handler.removeCallbacks(mRunnable);

        mPlaying = false;

    }

    

    @Override

    public Drawable getDrawable() {

        Bitmap bitmap = mImageCache.loadBitmap(mContext, mGifAssets.getGifFramePath(mCurrentFrameIdx));

        BitmapDrawable drawable = new BitmapDrawable(mContext.getResources(), bitmap);

        int width = drawable.getIntrinsicWidth();

        int height = drawable.getIntrinsicHeight();

        drawable.setBounds(0, 0, width > 0 ? width : 0, height > 0 ? height : 0);

        return drawable;

    }

 

    @Override

    public void draw(Canvas canvas, CharSequence text, int start, int end, float x, int top, int y, int bottom, Paint paint) {

//        Logger.d(this, "draw " + mCurrentFrameIdx);

        Drawable b = getDrawable();

        canvas.save();

 

        int transY = bottom - b.getBounds().bottom;

        if (mVerticalAlignment == ALIGN_BASELINE) {

            transY -= paint.getFontMetricsInt().descent;

        }

 

        canvas.translate(x, transY);

        b.draw(canvas);

        canvas.restore();

    }

}

1. Detect Click Events

We want the animation to start playing when the area is clicked, therefore detecting clicking is necessary. We’ll extend ClickableSpan as below.

private static class AnimationClickableSpan extends ClickableSpan {

        AnimatedImageSpan mAnimatedImage;

        AnimationSettings mSettings;

        AnimatedImageUpdateHandler mHandler;

        AnimationClickableSpan(MyTextView pView, AnimatedImageSpan pSpan, AnimationSettings pSettings) {

            mAnimatedImage = pSpan;

            mSettings = pSettings;

            mHandler = new AnimatedImageUpdateHandler(pView);

        }

        

        @Override

        public void onClick(View widget) {

            MyTextView view = (MyTextView) widget;

            if (mAnimatedImage.isPlaying()) {

                mAnimatedImage.stopRendering();

            } else {

                mAnimatedImage.playGif(mSettings, mHandler);

            }

        }

    }

When the click event is detected, we start the animation playback, if it’s clicked while the animation is playing, we stop the playback.

2. MyTextView

We glue everything together in the MyTextView as below.

public class MyTextView extends TextView {

    private SpannableStringBuilder mSb = new SpannableStringBuilder();

    private String dummyText = "dummy " + System.currentTimeMillis();

    private Context mContext;

    private SimpleImageMemCache mImageCache;

    private ArrayList<AnimatedImageSpan> mAnimatedImages = new ArrayList<AnimatedImageSpan>();

 

    public MyTextView(Context context) {

        super(context);

        mContext = context;

    }

    

    public MyTextView(Context context, AttributeSet attrs) {

        super(context, attrs);

        mContext = context;

    }

    

    public MyTextView(Context context, AttributeSet attrs, int defStyle) {

        super(context, attrs, defStyle);

        mContext = context;

    }

    

    public void setImageCache(SimpleImageMemCache pImageCache) {

        mImageCache = pImageCache;

    }

 

    @Override

    protected void onDetachedFromWindow() {

        super.onDetachedFromWindow();

        Log.d(this.getClass().getName(), "onDetachedFromWindow ");

        for (AnimatedImageSpan ais : mAnimatedImages) {

            Log.d(this.getClass().getName(), "animation playing " + ais.isPlaying());

            if (ais.isPlaying()) {

                ais.stopRendering();

            }

        }

        mAnimatedImages.clear();

        mSb.clearSpans();

        mSb.clear();

    }

    

    public void appendText(String pStr) {

        mSb.append(pStr);

    }

    

    public void appendAnimation(AnimationAssetsSet pAsset, AnimationSettings pSettings) {

        mSb.append(dummyText);

        AnimatedImageSpan ais = new AnimatedImageSpan(mContext);

        ais.setImageCache(mImageCache);

        ais.setAnimationAssets(pAsset);

        mSb.setSpan(ais, mSb.length() - dummyText.length(), mSb.length(), Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);

        AnimationClickableSpan clickSpan = new AnimationClickableSpan(this, ais, pSettings);

        mSb.setSpan(clickSpan, mSb.length() - dummyText.length(), mSb.length(), Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);

        mAnimatedImages.add(ais);

    }

    

    public void finishAddingContent() {

        this.setText(mSb);

        this.setMovementMethod(LinkMovementMethod.getInstance());

    }

    

    private static class AnimationClickableSpan extends ClickableSpan {

        AnimatedImageSpan mAnimatedImage;

        AnimationSettings mSettings;

        AnimatedImageUpdateHandler mHandler;

        AnimationClickableSpan(MyTextView pView, AnimatedImageSpan pSpan, AnimationSettings pSettings) {

            mAnimatedImage = pSpan;

            mSettings = pSettings;

            mHandler = new AnimatedImageUpdateHandler(pView);

        }

        

        @Override

        public void onClick(View widget) {

            MyTextView view = (MyTextView) widget;

            if (mAnimatedImage.isPlaying()) {

                mAnimatedImage.stopRendering();

            } else {

                mAnimatedImage.playGif(mSettings, mHandler);

            }

        }

    }

}

3. The Complete Source Code

The complete source code can found at github repo.