Accessing screenshots from Android's Recent Apps screen
source link: https://worthdoingbadly.com/androidrecents/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Accessing screenshots from Android's Recent Apps screen
May 10, 2018
I learned to call Android’s hidden ActivityManager APIs from the ADB command line to access the screenshots of Recent Apps, so I can build a custom app switcher.
Introduction
Google presented Android P’s new navigation bar at Google I/O, and I was impressed by the animations and the integration with the homescreen. I wanted to try replicating the navigation UI to see how it’s made. I started by making a basic horizontal carousel showing my recent apps:
Figure 1: my task switcher prototype, after one day’s work
Figure 2: Android’s current task switcher, for reference
To make this interface, I needed the phone’s list of tasks and screenshots of each task. This is much more challenging than it sounds: I had to learn how Android apps talk to the system at the lowest levels.
Why this is hard
Getting the list of current apps used to be a simple ActivityManager.getRecentTasks call. However, apps started abusing it, so in Android 5.0, this API was hidden behind a new permission, android.permission.GET_DETAILED_TASKS
. This permission is only granted to system applications, so my application can’t get it. However, the ADB shell can access it.
For a prototype, I can simply tether my phone to a computer, and ask the ADB shell to send the tasks to my app. Thus, I need to make an ADB command line app that can:
- access the current list of running apps
- get the screenshot of each app
- export this data to a normal Android app
Running from adb
Normally, Android applications are started from Android’s graphical user interface. However, in the adb shell, there’s only a command line, and the entry point is good old public static void main
. No Context
, no Activity
- how do I run any code that talks to Android?
I know a command line tool on Android is possible, since the Substratum theme manager also uses a command line tool started from ADB. How did they do it?
I looked at the existing utilities on Android: one commonly used command is am
, used to start activities from the command line when debugging. The executable, /system/bin/am
, is actually a simple shell script:
base=/system
export CLASSPATH=$base/framework/am.jar
exec app_process $base/bin com.android.commands.am.Am "$@"
which sets a CLASSPATH pointing to the Java code of the tool, then runs app_process
with the working directory and the main class of the tool. I can do the same by setting the CLASSPATH to my APK and running my main class.
To autodetect the APK path, I used the pm path
command:
$ pm path net.zhuoweizhang.pill
package:/data/app/net.zhuoweizhang.pill-1/base.apk
Using a sed
command, I removed the leading package:
from the path before storing it in CLASSPATH
, giving a final command line of
CLASSPATH="$(pm path net.zhuoweizhang.pill|sed -e s/^package//)" app_process /sdcard net.zhuoweizhang.pill.PillServer
Oddly, Instant Run causes pm path
to show multiple packages: I had to disable Instant Run to make this work.
Talking to the Android system
Now that I’m running Java code from the ADB command line, how do I talk to the Android system? There’s no Context
, so I can’t just run Context.getSystemService(ACTIVITY_MANAGER)
to get an ActivityManager to get the list of tasks.
I once again turn to the am
utility. The Java code for am
shows how it accesses the ActivityManager:
private IActivityManager mAm;
mAm = ActivityManager.getService();
Note that it accesses an IActivityManager, not the regular ActivityManager - which needs a Contextnote 1. As it turns out, ActivityManager is just a wrapper around IActivityManager: all ActivityManager methods eventually call the equivalent IActivityManager method.
Therefore, if I use IActivityManager, I can talk to Android from a command line app, without a Context
!
The list of IActivityManager’s exported methods is, of course, defined in its AIDL file, just like a regular Android Service.
Getting the recent apps images
Let’s see how Android’s existing Recent Apps screen gets its images. I know - from looking at the Android log - that the Recent Apps screen is implemented in SystemUI:
$ logcat|grep Recent
I ActivityManager: START u0 {flg=0x10804000 cmp=com.android.systemui/.recents.RecentsActivity} from uid 10027
Let’s take a look at RecentsActivity’s source: TaskViewThumbnail
sounds relevant. It sets the app screenshot when it receives a TaskSnapshotChangedEvent
. Looking for this class brings us to RecentsImpl
, which sends the TaskSnapshotChangedEvent
from the onTaskSnapshotChanged
method of a TaskStackListener
. This listener is registered on the SystemServicesProxy
class. Looking through this class, I found many relevant methods.
public List<ActivityManager.RecentTaskInfo> getRecentTasks(int numLatestTasks, int userId,
boolean includeFrontMostExcludedTask, ArraySet<Integer> quietProfileIds) {
if (mAm == null) return null;
// snip
List<ActivityManager.RecentTaskInfo> tasks = null;
try {
tasks = mAm.getRecentTasksForUser(numTasksToQuery, flags, userId);
} catch (Exception e) {
Log.e(TAG, "Failed to get recent tasks", e);
}
Sounds like getRecentTasksForUser lets us find the recent apps. This is called on the ActivityManager
, not the IActivityManager
, so let’s find the method in ActivityManager:
public List<RecentTaskInfo> getRecentTasksForUser(int maxNum, int flags, int userId)
throws SecurityException {
try {
return getService().getRecentTasks(maxNum,
flags, userId).getList();
} catch (RemoteException e) {
throw e.rethrowFromSystemServer();
}
}
The IActivityManager equivalent is just getRecentTasks
. I can call it like this:
List<ActivityManager.RecentTaskInfo> tasks = iam.getRecentTasks(25, 0, 0)
to get the last 25 tasks for user 0 (the main user).
/**
* Returns a task thumbnail from the activity manager
*/
public @NonNull ThumbnailData getThumbnail(int taskId, boolean reducedResolution) {
if (mAm == null) {
return new ThumbnailData();
}
final ThumbnailData thumbnailData;
if (ActivityManager.ENABLE_TASK_SNAPSHOTS) {
ActivityManager.TaskSnapshot snapshot = null;
try {
snapshot = ActivityManager.getService().getTaskSnapshot(taskId, reducedResolution);
} catch (RemoteException e) {
Log.w(TAG, "Failed to retrieve snapshot", e);
}
if (snapshot != null) {
thumbnailData = ThumbnailData.createFromTaskSnapshot(snapshot);
} else {
return new ThumbnailData();
}
Looks like thumbnails are accessed through the getTaskSnapshot
method on IActivityManager. Let’s look at how ThumbnailData processes the returned snapshot:
public static ThumbnailData createFromTaskSnapshot(TaskSnapshot snapshot) {
ThumbnailData out = new ThumbnailData();
out.thumbnail = Bitmap.createHardwareBitmap(snapshot.getSnapshot());
Following this method’s example, I can turn the TaskSnapshot into a Bitmap easily. To get a JPEG of an app’s screenshot, all I have to do is take the persistentId
from the task information, and run:
ActivityManager.TaskSnapshot thumbnail = iam.getTaskSnapshot(id, false);
GraphicsBuffer graphicBuffer = thumbnail.getSnapshot();
Bitmap bmp = Bitmap.createHardwareBitmap(graphicBuffer);
bmp.compress(Bitmap.CompressFormat.JPEG, 80, os);
Just what is a GraphicsBuffer? Android Developer explains that it’s a graphic that can be shared across processes without copying.
Now I have all the data I need, but how do I send it to the main application, running as a different UID?
Sending the information across
The usual methods of inter-process communication on Android is, of course, through Intents (Activity launch, Broadcast Intent) or through a Service. Unfortunately, I can’t use a Service since a Context is needed to register one. I did try using a Broadcast Intent, since I wanted to try passing the GraphicsBuffer directly to my app without converting it to a JPEG: it didn’t work. It turns out Intents can’t serialize file descriptors, which is used by GraphicsBuffers to share memory between processes.
Instead, I decided to design for prototyping, not security. I wanted to load these images into an ImageView, and there are many libraries that help load images into ImageView from HTTP.
Therefore, I decided to simply create a local HTTP server. Sure, it’s insecure (allows any app to access the screen), but for a prototype, this is fine. (Do not use this in a real app).
I used the well-known NanoHTTPD library, which is a single file HTTP server that can be easily integrated into any app. I made two endpoints:
- The root page,
GET /
, calls thegetRecentTasksForUser
method and returns the tasks in JSON format. - The thumbnail endpoint,
GET /thumbs/(id)
, calls thegetTaskSnapshot
method and returns a JPEG of the desired task.
Originally, I only had one endpoint, which sent the images along with the tasks; however, it turns out converting a GraphicsBuffer to a Bitmap takes almost half a second each, and it takes several seconds to get the list of apps. They were broken out into a separate endpoint to allow the main app to load the thumbnails on demand.
The app itself: learning RecyclerView and Glide
Now that the list of tasks is available, I just need to show them in an app. I chose to use a RecyclerView to display the list of apps.
To download data from the local server, I used Square’s okhttp3 to simplify getting the JSON. To load the images into the ImageViews, I used Bumptech/Google’s Glide library, which made loading images absolutely pain-free. I try to minimize the number of libraries I use in apps, but these libraries are well worth their size.
After a tiny bit of styling, we’re seeing the list of apps!
The code so far can be found at https://github.com/zhuowei/PillAppSwitcher.
What I learned
- How Android’s Recent Apps screen actually works
- Accessing Activity Manager methods on Android from the command line
- What you can’t do on Android (registering a Service from a command line app, sending an Intent with file descriptors passed in)
- Using Glide to load images in a RecyclerView
Future steps
Next, I’ll work on making an actual task switcher - that’ll be the subject of an upcoming post.
Note 1: Context
Why can’t I just make a Context, then? A Context needs an ApplicationThread, which I can’t make from a command line app. I can go more in-depth on this: let me know if how an Android app starts up interests you.
Recommend
-
5
Alexander ZeitlerAccessing local domains in local dev environment from Android emulatorPublished on Friday, September 18, 2020Let's consider you're using local domains in deve...
-
4
Accessing Localhost Server From Ionic App Running on a Mobile Device (iOS/Android)By Josh Morony | Last Updated: October 20, 2020Ever been in a situati...
-
4
Looking for a Chrome extension to take screenshots of a page for different screen sizes Apr 21 ・1 min read...
-
2
"Flagship killer" was apparently the codename of the last update — OnePlus admits to throttling 300 popular apps with recent update Chrome performance tanks 85-75 percent in some tests.
-
5
📌 Why not use ‘ActivityManager.getRunningTasks’?As in most accepted answers on SO, you are most likely to use the Activ...
-
6
Android 13’s new launcher search lets you pin recent queries to your home screen By Manuel Vonau Published 2 hours ago
-
4
Android 13’s Restricted setting feature will block malicious apps from accessing your notifications Google is introducing a change with
-
3
PopFrameElevate screen recordings & screenshotsFree OptionsStyle screen recordings and screenshots with device frames & curated backgrounds,...
-
2
Programmatically Accessing App Settings from Logic Apps Expressions Programmatically Accessing App Settings from Logic Apps Expressions
-
4
Google to Ban Financial Lending Apps From Accessing User Photos, Contacts The move is a bid to stop predatory loan apps from harassing and intimidating borrowers into paying outstanding debts,...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK