How to Port ffmpeg (the Program) to Android–Ideas and Thoughts

Why ffmpeg for Android and The App

I was working on a project that requires transcoding of video files and then play these files on Android device. The definite tool to do this is ffmpeg. I can do it on Desktop and transfer the files to Android, but why not just transcode on Android itself?

Then I searched for ffmpeg and transcoding tools on Android, but cannot find any. And since I’ve built ffmpeg libraries for Android before, I believe it is also doable to port the ffmpeg command line utility to Android. So I started doing it.

Below is the screen capture of the Android app (I released it as ffmpeg for Android Beta),

devicedevice1

Figure 1. Screenshots of ffmpeg for Android

Currently it only supports armv7 processor, the one that my Nexus One device uses.

How did I Port it?

The main design goal is to change as few lines of code as possible for ffmpeg.c file. To meet this goal, I decoupled the UI and ffmpeg processing into two processes.

Running ffmpeg on a separate processes has a few advantages,

  • ffmpeg uses a lot of exit function, which will terminate the process. Use a separate process for ffmpeg will ensure the termination of ffmpeg won’t affect the front UI.
  • ffmpeg is processing intensive, running it on a separate process will not affect the responsiveness of front UI.

There’s inter-process communication problem to solve (IPC) so the two processes can communicate with each other. The UI will be able to start and stop ffmpeg, and ffmpeg will send output message to UI.

From UI to ffmpeg is simple, I implement a service that will call ffmpeg. And we start the service using Intent. It’s the same for stop command.

From ffmpeg to UI, there’re a few choices. pipe, socket, etc. But since it requires only a very simple one way communication channel (ffmpeg write, UI read), I just use text files. ffmpeg dump all outputs to text files, and UI read it every second or two (you can also use file observer and update the UI only when the text files changes).

What are the Changes for ffmpeg.c

With the design above, the change for ffmpeg.c is simple.

  • Change the main(int argc, char **argv) function prototype to a java JNI interface prototype that Android java code can call.
  • Change the output to stdout to the text files we created.
  • Comment out a few lines that cause the compilation error.

A little More Thoughts

I’m thinking this method can be used to port lots of other applications to Android too. Two processes, running the background processing utility as a service, change the main method to an native interface so Java code can call, use intents and text files for communication, etc.

If you would like to try out this app, go to https://market.android.com/developer?pub=roman10. Note that it’s for armv7 processor only.

A Quick Note on Error Recovery for Streaming Applications

There’re generally 3 ways to recover error caused by Internet transmission, including Retransmission, Redundant Data, and Error Concealment.

Retransmission

Retransmission is easy to understand. Just resend the packet that is lost or has error. It has the following pros and cons,

Pros:

  • Retransmission resends the entire packet, so the data error/lost is repaired accurately.
  • It has low overhead when there’s enough bandwidth, as it doesn’t computation.

Cons:

  • Retransmission increases the delay, as the sender needs to wait for some signal from receiver side about the packets needs to be retransmitted, or an timeout event has occurred. Also the retransmission itself take some time.
  • Retransmission takes bandwidth. If there’s congestion at the network, retransmission can make the congestion worse.

In video streaming application context, there’re other aspects that matters, and retransmission can be selective based on a few factors,

  • Important/Urgent packets are retransmitted first. For example, I frame packets are more important than B, P frame packets.
  • Packets are only retransmitted when there’s enough time. For example, if a packet has missed its play time, there’s no need to retransmit the packet.

Redundant Data

Redundant Data is well known as Forward Error Correction, a technique that sends additional information which can be used to recover lost/error packet at the receiver side.

Pros:

  • Bandwidth requirement doesn’t change when there’s loss, so it won’t make the congestion worse if there’s one.
  • Receiver doesn’t needs sender’s action to recover.
  • Delay is better if the computation is fast. This is also related to the dependency of the FEC, as the recovery may requires waiting of several packets to arrive first.

Const:

  • It has constant overhead in terms of network bandwidth. Even when there’s no lost/error, redundant error still has to be sent.
  • In terms of burst data loss, this method could fail.

To improve FEC against burst data loss, there’s two methods can be used,

  1. Arrange packets in multiple dimensions.
  2. Interleaving, rearrange the packet order when doing FEC.

Interleaving itself has some pros and cons

Pros:

  • It transfer burst data loss to random data loss, as the consecutive packets in the network transmission are not consecutive packets in FEC.
  • It doesn’t add any overhead to bandwidth or computation, only rearrangement is done.

Cons:

  • It could cause delay, as the recovery needs to wait for packets in a longer distance in network transmission.

Error Concealment

Error concealment refers to the technique of trying to coneal the error instead of recover it accurately.

Pros:

  • no overhead in terms of bandwidth, no matter there’s a congestion or not.
  • Delay can be small if the concealment computation is fast.

Cons:

  • It may not always give a good result
  • The computation can be complicated if complex concealment algorithms are used.

The concealment techniques include splice, noise substitution, repetition, interpolation, regeneration etc.

Android PhoneStateListener Illustration–with Auto Logging Calls to Google Calendar Example

android.telephony.PhoneStateListener class is provided to monitor various state changes related to phone function of your Android device.

In order to use it, one needs to complete the following steps.

0. Extends the PhoneStateListener listener class, and override the methods for the state you want to receive updates.

1. Pass your customized PhoneStateListener instance to TelephonyManager.listen() function with a flag to indicate which states you want to monitor.

I’ll use a calltrack implementation (auto logging call logs to google calendar, used in my app Advanced Phone Log) to illustrate it. You can download the entire sample app source code at the end of this post. The UI of  the sample app and a sample log to Calendar is as below,

sc0sc1

The idea of the calltrack app is  to use a broadcast receiver to receive the Android system broadcast for a call state change. There’re three call states, CALL_STATE_IDLE, CALL_STATE_OFFHOOK and CALL_STATE_RINGING.

Based on the call state transitions, we can decide whether there’s call ended. In other words, if call state changes from CALL_STATE_OFFHOOK to CALL_STATE_IDLE, we know there’s a outgoing call; if it’s from CALL_STATE_RINGING to CALL_STATE_IDLE, we know it’s a incoming call or miss call.

Once we detected a call ends, we know there’ll be a new call log generated in Android system contacts database. We read the phone log and use the method illustrated in Android Tutorial – Programming with Calendar to create a new event to google calendar.

Below illustrates the implementation details.

0. Extend the PhoneStateListener.

The code is as below,

class MyPhoneStateListener extends PhoneStateListener {

        public void onCallStateChanged(int state,String incomingNumber){

            AdvancedPhoneLogRecord mQueryRecord;

            switch(state) {

              case TelephonyManager.CALL_STATE_IDLE:

                  if (mLastState != TelephonyManager.CALL_STATE_IDLE) {

                      AdvancedPhoneLogRecord lQueryRecord = null;

                      int lSize = mQueryRecords.size();

                      for (int i = 0; i < lSize; ++i) {

                          if (mQueryRecords.get(i).aplr_number.compareTo(incomingNumber)==0) {

                              //Log.e("query-all-match", mQueryRecords.get(i).aplr_number + ":" + mQueryRecords.get(i).aplr_time.toString());

                              lQueryRecord = mQueryRecords.get(i);

                              mRecord = getLastCallLog(mQueryRecords.get(i));

                              break;

                          }

                      }

                      List<AdvancedPhoneLogRecord> lQueryRecords = new ArrayList<AdvancedPhoneLogRecord>();

                      if (lQueryRecord!=null) {

                          for (int i = 0; i < lSize; ++i) {

                              if (mQueryRecords.get(i).aplr_number.compareTo(lQueryRecord.aplr_number)!=0) {

                                  lQueryRecords.add(mQueryRecords.get(i));

                              } 

                          }

                          mQueryRecords = lQueryRecords;

                          lQueryRecords.clear();

                      }

//                      for (int i = 0; i < mQueryRecords.size(); ++i) {

//                          Log.e("query-all----", mQueryRecords.get(i).aplr_number + ":" + mQueryRecords.get(i).aplr_time.toString());

//                      }

                      if (CalltrackSettingsStatic.getTrackEnable(mContext) != 0) {

                          switch (mRecord.aplr_type) {

                          case android.provider.CallLog.Calls.INCOMING_TYPE:

                              if (CalltrackSettingsStatic.getTrackInEnable(mContext) != 0) {

                                  addEvent(mRecord);

                              }

                              break;

                          case android.provider.CallLog.Calls.OUTGOING_TYPE:

                              if (CalltrackSettingsStatic.getTrackOutEnable(mContext) != 0) {

                                  addEvent(mRecord);

                              }

                              break;

                          case android.provider.CallLog.Calls.MISSED_TYPE:

                              if (CalltrackSettingsStatic.getTrackMissEnable(mContext) != 0) {

                                  addEvent(mRecord);

                              }

                              break;

                          }

                      }

                      Log.v("PhoneIntentReceiver", mRecord.aplr_name + ":" + mRecord.aplr_number + ":" + mRecord.aplr_dataId + ":" + mRecord.aplr_duration + ":" + mRecord.aplr_id + ":" + mRecord.aplr_type);

                  }

                  mLastState = TelephonyManager.CALL_STATE_IDLE;

                  break;

              case TelephonyManager.CALL_STATE_OFFHOOK:

                  mLastState = TelephonyManager.CALL_STATE_OFFHOOK;

                  mQueryRecord = new AdvancedPhoneLogRecord();

                  mQueryRecord.aplr_number = incomingNumber;

                  mQueryRecord.aplr_time = new Date(System.currentTimeMillis());

                  mQueryRecords.add(mQueryRecord);

                  break;

              case TelephonyManager.CALL_STATE_RINGING:

                  mLastState = TelephonyManager.CALL_STATE_RINGING;

                  mQueryRecord = new AdvancedPhoneLogRecord();

                  mQueryRecord.aplr_number = incomingNumber;

                  mQueryRecord.aplr_time = new Date(System.currentTimeMillis());

                  mQueryRecords.add(mQueryRecord);

                  break;

             }

        } 

   }

When a CALL_STATE_RINGING or CALL_STATE_OFFHOOK even is detected, we know that there’s new call starts (outgoing, incoming or miss call). So we create a new Query and remember the state.

Note that sometimes the broadcast receiver receives mutliple broadcast events when there’s a single state change, and sometimes there’re incoming calls when you’re in a call. So we use a queue to store all the queries.

When a CALL_STATE_IDLE event is detected, we know a call ends. We go through all the query records in the queue, and get the one that has the same number the current call state change number. We call a method getLastCallLog (details in source code) to retrieve the call log from the system database and then clear all entries in the queue with this number.

Next step is just to log the call to calendar based on user settings.

One thing note-worthy is that getLastCallLog queries the system database multiple times to get the latest call log as the Android system takes some time to update its database.

1. Register your customized PhoneStateListener

The code is simple,

MyPhoneStateListener phoneListener=new MyPhoneStateListener();

TelephonyManager telephony = (TelephonyManager) context.getSystemService(Context.TELEPHONY_SERVICE);

telephony.listen(phoneListener,PhoneStateListener.LISTEN_CALL_STATE);

2. Register Broadcast Receiver

Add lines below to AndroidManifest.xml,

<receiver android:name=".PhoneIntentReceiver" android:process=":remote">

    <intent-filter>

        <action android:name="android.intent.action.PHONE_STATE" />

    </intent-filter>

</receiver>

Also don’t forget the permissions,

<uses-permission android:name="android.permission.READ_PHONE_STATE"></uses-permission>

<uses-permission android:name="android.permission.READ_CONTACTS"></uses-permission>

<uses-permission android:name="android.permission.WRITE_CALENDAR"></uses-permission>

<uses-permission android:name="android.permission.READ_CALENDAR"></uses-permission>

More Notes

Android also supports lots of other phone state change detection, including,

Flags

Event Handler you should overwrite

LISTEN_CALL_FORWARDING_INDICATOR

onCallForwardingIndicatorChanged

LISTEN_CALL_STATE

onCallStateChanged

LISTEN_CELL_LOCATION

onCellLocationChanged

LISTEN_DATA_ACTIVITY

onDataActivity

LISTEN_DATA_CONNECTION_STATE

onDataConnectionStateChanged

LISTEN_MESSAGE_WAITING_INDICATOR

onMessageWaitingIndicatorChanged

LISTEN_NONE

 

LISTEN_SERVICE_STATE

onServiceStateChanged

LISTEN_SIGNAL_STRENGTH

onSignalStrengthChanged(int asu)

LISTEN_SIGNAL_STRNEGTHS

onSignalStrengthChanged(SignalStrength signalStrength)

Just like using the OnCallStateChanged can write a calltrack function, you can use the event handlers in the above list to write very useful apps.

Here is the download link for the the sample calltrack app code.

Android Tutorial–Customized Radio Button List

Radio buttons are almost always used to represent single choice from a list of items. Android is no exception to that.

Android provides two ways to use radio buttons. At a higher level, it allows one to use radio button list in ListActivity by setting the choice mode, as indicated in the code below,

public class List10 extends ListActivity {

    @Override

    public void onCreate(Bundle savedInstanceState) {

        super.onCreate(savedInstanceState);

        setListAdapter(new ArrayAdapter<String>(this, android.R.layout.simple_list_item_single_choice, CONTENT));

        final ListView listView = getListView();

        listView.setItemsCanFocus(false);

        listView.setChoiceMode(ListView.CHOICE_MODE_SINGLE);

    }

    private static final String[] CONTENT = new String[] {

        "Action", "Adventure", "Animation", "Children", "Comedy", "Documentary", "Drama",

        "Foreign", "History", "Independent", "Romance", "Sci-Fi", "Television", "Thriller"

    };

}

The effect is as below,

sc0

At a lower level, one can create radio button list by using RadioGroup and RadioButton. One can find the detailed code in the RadioGroup1.java source file downloadable at the end of this post. The effect of this method is as below,

sc1

However, the lower level default radio button only allows one to place the text on the right of the button, and the higher level method is also difficult to customize. But one can always build a radio button list from scratch.

The method is to place a radio button and a text view in a linear layout to form a single item and then use it as the element to build list view. The effect is as below,sc2

The complete code can be downloaded from here.