Understanding External Storage Features and Implementation Across Various Android Versions

Introduction

External storage, as an important component often encountered in development, has undergone numerous significant changes in various Android versions. I have also been puzzled by the many strange paths for such simple external storage: /sdcard, /mnt/sdacrd, /storage/extSdCard, /mnt/shell/emulated/0, /storage/emulated/0, /mnt/shell/runtime/default/emulated/0... In fact, these represent the maturity and release of various technologies: emulated external storage, multi-user support, runtime permissions, and more.

External Storage Features in Various Android Versions

1. Android 4.0

● Supports emulated external storage (implemented through FUSE)

● Introduced primary external storage and secondary external storage (no exposed interfaces)

● Supports MTP (Media Transfer Protocol) and PTP (Picture Transfer Protocol)

2. Android 4.1

● In the developer options, there is a toggle switch that enforces apps to declare read permissions before they can perform read operations.

3. Android 4.2

● Supports multiple users, each with their own independent external storage

4. Android 4.4

● Read operations require READ_EXTERNAL_STORAGE permission

● Apps can read and write to the external storage app directory (/sdcard/Android/<pkg>/) without declaring permissions

● Added Context.getExternalFilesDirs() interface, which allows access to the app's files paths under primary external storage and other secondary external storage

● Introduced Storage Access Framework (SAF)

5. Android 6.0

● External storage supports dynamic permission management

● Adoptable Storage feature

6. Android 7.0

● Introduced scoped directory access

Additional point:

If the app's minSdkVersion and targetSdkVersion are set to <=3, the system will grant the READ_EXTERNAL_STORAGE permission by default.

Explanation of Some Features

1. Emulated External Storage

a. Necessity

● FAT32 is a Microsoft patent, which may involve licensing and legal issues (related articles);

● Allows customization of Android's own external storage access rules;

● Lays the groundwork for multi-user support;

b. Implementation Principle

The system daemon /system/bin/sdcard uses FUSE to implement a simulated FAT-like format SD card file system, which is what we often refer to as the built-in SD card. (For detailed code, please refer to: /xref/system/core/sdcard/sdcard.c)

Filesystem in Userspace (FUSE) is a software interface for Unix-like computer operating systems that allows unprivileged users to create their own file systems without editing kernel code. Currently, Linux supports this through a kernel module.

The general process of the sdcard daemon simulating external storage (using Android 4.0 as an example):

● First, the /data/media directory is designated for emulated external storage. The owner and group of this path are generally set to media_rw, ensuring that only the sdcard program or root process can access the directory.

● After the sdcard daemon starts, open the /dev/fuse device.

● Mount the fuse file system in the /mnt/sdcard directory.

● Start a new thread, handle file system events within the thread, and write the results back.

After going through a series of steps, the 'sdcard' process creates a FUSE (Filesystem in Userspace) filesystem on the '/mnt/sdcard' path. Any access requests to '/mnt/sdcard' are converted into events, which are then handled by the 'sdcard' daemon process and mapped to the '/data/media' directory for actual file operations.

For example, when an application creates a file "/mnt/sdcard/a", it actually creates the file "/data/media/a" in the underlying filesystem.

c. Advantages

● The emulated external storage capacity is shared with the /data partition, allowing for more flexible allocation of user data between internal and external storage.

● The emulated external storage itself cannot be unmounted, preventing issues with app access due to unmounting and reducing the chances of damage caused by external factors.

● All access requests go through the sdcard daemon process, allowing Android to customize access rules.

d. Disadvantages

● There is some performance loss.

e. Impact

● In Android 6.0 and later, due to the need for dynamic permission management, there will be multiple FUSE mount points. This causes inotify/FileObserver to lose events when monitoring file events on external storage.

Inotify is one of the Linux kernel subsystems and serves as an additional feature for the file system. It can monitor the file system and notify applications of changes. —— Wikipedia (https://zh.wikipedia.org/wiki/Inotify)

2. Multi-user support

a. Supported Versions

● Android 4.2 introduced multi-user support, but it was limited to tablets.

● Starting with Android 5.0, device manufacturers can enable multi-user modules during compilation.

b. Background Knowledge

● Mount --bind

MS_BIND (Linux 2.4 onward)

Perform a bind mount, making a file or a directory subtree visible at another point within a file system. Bind mounts may cross file system boundaries and span chroot(2) jails. The filesystemtype and dataarguments are ignored. Up until Linux 2.6.26, mountflagswas also ignored (the bind mount has the same mount options as the underlying mount point). —— mount(2) - Linux man page（https://linux.die.net/man/2/mount）

Legend (from https://xionchen.github.io/2016/08/25/linux-bind-mount) :

1) Bind the /home directory tree to /mnt/backup:

2) After the bind is completed, the access to /mnt/backup will be equivalent to the access to /home, and the original /mnt/backup will become invisible.

● Mount namespaces

Mount namespaces provide isolation of the list of mount points seen by the processes in each namespace instance. Thus, the processes in each of the mount namespace instances will see distinct single-directory hierarchies. —— mount_namespaces(7) - Linux manual page - man7 .org

mount_namespaces(7) - Linux manual page

In layman's terms, the mount namespace realizes the isolation of the mount point, and the process of different mount namespaces sees different directory levels.

● Mount propagation: shared mount, slave mount, and private mount

Mount namespaces provide complete isolation, but they are not suitable for some situations. For example, on a Linux system, if process A mounts a CD-ROM in namespace 1, namespace 2 will not be able to see the CD-ROM due to the isolation.

To solve this problem, mount propagation is introduced. Mount propagation defines the propagation types of mount points:

1) Shared mount: This type of mount point joins a peer group and propagates and receives mount events within the group.

2) Slave mount: This type of mount point joins a peer group and receives mount events within the group but does not propagate them.

3) Shared/Slave mount: A combination of the above two types. It can receive mount events from one peer group (with the type being slave mount) and propagate them to another peer group.

4) Private mount: This type of mount point has no peer group and neither propagates nor receives mount events.

5) Unbindable mount: Not explained in detail.

The formation conditions of a peer group are that a mount point is set as a shared mount and meets any one of the following situations:

1) The mount point is copied when creating a new namespace.

2) A bind mount is created from this mount point.

In addition, let's supplement the conversion of propagation types:

1) If a shared mount is the only remaining mount point in the peer group, applying a slave mount to it will cause it to become a private mount.

2) Applying a slave mount to a non-shared mount type is ineffective.

The background knowledge is explained up to this point. Among them, the propagation type of mount points may be difficult to understand but is very important. You can refer to the examples in the Linux Programmer's Manual for mount namespaces (search for "MS_XXX example") to learn more: mount_namespaces(7) - Linux manual page

c. Implementation Principle

To summarize the implementation of external storage isolation for multi-user: when an application process is created in a multi-user environment, a new mount namespace is created for the process. Then, by using bind mounting, the system exposes the current user's external storage space to the application.

Take the Android 4.2 code as an example [mountEmulatedStorage(dalvik_system_Zygote.cpp)]:

First, obtain the user ID. In a multi-user environment, the user ID is calculated by dividing the application's UID by 100,000.

● Create a new mount namespace through the unshare method.

● Obtain the environment variables related to external storage. The EXTERNAL_STORAGE environment variable is a legacy variable from older versions of Android, which records the traditional path of external storage. The EMULATED_STORAGE_SOURCE environment variable records the source path of bind mounting, and it is important to note that applications do not have permission to access this directory. The EMULATED_STORAGE_TARGET environment variable records the target path of bind mounting, and the external storage path obtained by the application is located in this directory.

● Prepare the mount path and perform bind mounting. Here, we look at the execution branch when mountMode is set to MOUNT_EXTERNAL_MULTIUSER. In this case, /mnt/shell/emulated/0 will be bind-mounted to /storage/emulated/0. If it's the second user, then /mnt/shell/emulated/1 is bind-mounted to /storage/emulated/1, where the number represents the user ID. Note that this is a new mount namespace, so only the application can see the bind mount under /storage/emulated/0. From adb shell, you would only see an empty directory.

● For backward compatibility with previous versions, bind the user's external storage path to the path specified by the EXTERNAL_STORAGE environment variable.

3. Dynamic Permission Management

a. Background

Android 6.0 introduced runtime permissions, allowing users to dynamically grant dangerous permissions, including external storage access permissions.

b. Implementation Principle

The dynamic authorization of external storage access permissions is achieved through the combination of FUSE and mount namespaces technologies. By looking at the following commit record https://android.googlesource.com/platform/system/core/+/f38f29c87d97cea45d04b783bddbd969234b1030^!/#F1, we can clearly understand the entire implementation.

In order to grant an application process read/write access to external storage without killing the process, Android uses FUSE to simulate three different access views for the /data/media directory, which are default, read, and write.

When an application is granted read/write permissions, the vold subprocess will switch to the application's mount namespace and rebind the corresponding view to the application's external storage path.

Switching the process's mount namespace requires a kernel version of 3.8 or above. The switching function is called setns. It seems that NDK does not expose this function to developers, but you can find the implementation for ARM in the source code. If needed, you can directly include it as it's just a sys call.

c. Code Analysis

● Source code version: Android 6.0.0_r1

● Start by analyzing the /xref/system/core/sdcard/sdcard.c file. Here, we only extract some parts of the code and add some comments:

● When an application process is created, the general process is as follows (/xref/frameworks/base/core/jni/com_android_internal_os_Zygote.cpp):

1) Create a new mount namespace.

2) Remove all previous mounts in the /storage directory of the mount namespace to avoid interference.

3) Choose a path based on the mount_mode.

4) Bind the selected path to the /storage directory.

● During the runtime of the process, when the access permissions of external storage change (due to user authorization), the basic process is as follows (/xref/system/vold/VolumeManager.cpp):

1) Obtain the mount namespace of init to compare with the mount namespace of the subsequent processes. If they are consistent, there is no need to rebind.

2) Traverse the process directories under /proc and filter them based on the UID.

3) After finding the corresponding PID, fork a subprocess to re-mount. Here, setns is used to switch the mount namespace.

The logic of the re-mounting part is basically the same as when the application process is created, and it is not difficult to understand.

Tencent WeTest provides thousands of real mobile devices for testing anytime, anywhere, ensuring the quality of applications and mobile games. It saves millions in hardware costs and accelerates the agile development process.

Sign up for WeTest Real Device Testing

At the same time, Tencent WeTest's compatibility testing team has accumulated 10 years of experience in mobile game testing. Their goal is to identify compatibility issues in game versions by devising targeted testing plans, accurately selecting target devices, and executing professional and comprehensive test cases. This allows for targeted corrections and optimizations to ensure the quality of mobile game products. Currently, the team supports all Tencent's mobile game projects in research and operation.

Sign up for WeTest Mobile Game Compatibility Testing