Finding and Removing Duplicate Files on Android Devices

1. Introduction to Duplicate Files on Android

Removing duplicates by manual operations, such as searching for all files and then deleting all unwanted duplicates, is extremely complex and time-consuming. As a result, how to find and remove duplicate files automatically from mobile devices in an efficient, simple, and correct manner is difficult to resolve. For this issue, we present two methods. The algorithms in the two methods process the files of mobile devices within the shared internal storage path and the mobile device's SD card path. If you have rooted the device, you can use all the method's functions in implementation. If not, you can only use the CompareDPF and RemoveDPF functions in the second method because most information is achieved based on an application self-scan. The two methods are constructed by two programs, and all the functions are invisible to users after the user verifies the permission. Our programs support desktop and command-line modes, so they can meet your demands in different situations. In addition, user interface programs can be generated based on our programs for your personal use. Finally, our methods are able to be used in combination with most non-device entities if the path details are offered.

In recent years, the use of mobile devices has greatly increased, and Android has become the most widely used mobile operating system. Usually, people use their mobile phones and tablets to manage their work. They may need to use their devices to check several versions of a document or multiple copies of almost the same photo in order to finish their tasks. Since the storage space of mobile devices is limited, this wastes storage space. What's worse, it makes finding and managing useful files difficult. These duplicate files exist in the form of videos, documents, photos, and music, among others. They may be created by the user because of disorganization or by the applications on the device when backing up data or creating cache files. Some applications also lead to the creation of duplicates. For example, the Communications application can create a user record and store multiple backups of multimedia messages every time the user operates it. Similarly, different camera applications can store the same content with different paths on the mobile device's storage system.

2. Manual Methods for Finding Duplicate Files

Some programming code allows the installation of small helper applications on the Android device. For example, Java enables a spreadsheet program. A spreadsheet is convenient for sorting and manually cleaning the list, and then a unique list can be generated by deleting the duplicate rows. Then, the unique list can be compared with a list on another Android device, and the files which appear as duplicates can be replaced by the empty files. Then, the user can generate the checksums of the unique files using the spreadsheet. At the end, only new files should be transferred. However, a device does not display files using an arbitrary extension, so it is impossible to find the needed file.

2.2 Spreadsheet

The approach of using a simple user interface is the easiest way of removing duplicate files from a device. Although this approach cannot be used to compare files that are under different names, it can be used to find files with the same name. The user can manually transmit the data to an external device and process the data, for example, on a personal computer. To compare the files on an external device, a list of files on the Android device should be made, transferred to this device, and then compared with the corresponding list from this device, and the files with the same names can be copied or deleted. This approach is suitable for programs that require unique images or music files saved on the Android device; for example, the Google Play Music application matches the music files on the Android device to the cloud version and generates a list of those files that need to be uploaded to the Google cloud in order to access this list on other devices. Using the Wi-Fi feature for file transfer by the programming language Java also requires the user to keep the Kindle application open while transferring the files because after another application becomes active and returns, the previously obtained list of the files is deleted. The list of files is made by sending an appropriate command, and then its content is read from the screen.

2.1 A Simple User Interface

3. Automated Tools for Identifying and Removing Duplicate Files

An overview of five tools is performed that offer solutions to the problem of search and removal of duplicate files on Android devices. The main goal of the overview is demonstrating capabilities of automated tools to thoroughly resolve such problem for the user. To this end, a specific test environment has been developed. The environment helps calculate space economy under different input conditions on a group of test datasets containing typical file types. It has been demonstrated that reduction from 17% to 6% in the number of duplicate files can be achieved by determining file category or comparing its content. The results are supported by statistical tests, demonstrating better qualities of specialized tools in comparison with the tools based on popular file search methods.

A special feature of a duplicate file is that it can be encountered multiple times while browsing the contents of memory storage and has the same data sequence. The organization of functionality of mobile system search is simplified because the file size, checksum, and fingerprint can be used for the preliminary comparison. The considered duplicates search methods can be divided into two groups based on whether the file content is taken into account. Tools for multimedia file search use metadata attributes (time, creation date, author) and extend them to mark image or sound file duplicates. To compare photo frames, sample dimensions and color distribution are used.

4. Best Practices for Managing Duplicate Files on Android

We remedied the problem by disassociating the ownership of files and directories and implementing the modules for detecting duplicates. Finally, we prepared both a menu to execute duplicate detection and deletion from the mountable Android file system on Linux and a program to execute duplicate detection and deletion from Windows. Based on the Android version 6 and 7, we compared the disk usage before and after purging the 14,463 duplicates and verified that 5.9GB of disk usage was reduced to 5.3GB, a 0.6GB save was recognized. The number of files per directory limits recommendations to remove duplicates and repair the Android device.

This chapter discusses the methodology and implementation for identifying and purging duplicate files on Android devices. With multiple landfills, batteries are the fastest growing waste stream. Battery companies spend up to $10 for each battery to be collected and taken for recycling. Duplicate detection and removal can make a huge contribution to solving these problems. By reducing the disk consumption of large files such as video files used for a long time, the lifespan of storage devices will increase and the electronic waste generation rate will decrease with the reduced number of spoilage. Deleting duplicates while preserving the most recent copy will also ensure that slow wearing files are retained. However, many files on Android devices are owned by an app, so deleting a file is not a trivial task.

5. Conclusion and Future Trends

Most of the current approaches for finding duplicate files require reading most of the data, which is highly time-consuming for the large-size datasets of today. This paper presents and evaluates the efficiency of novel sampling approximation algorithms for finding file duplicates developed in previous series. It describes several new key heuristics that further optimize the performance of these algorithms. Although the proposed heuristics are lossy in the sense of introducing some potential false negatives in the results, extensive experimental analysis provides evidence for two facts. Firstly, on most practical datasets, the effect of these false negatives is negligible. Secondly, the proposed techniques provide a significant decrease in running time compared to the asymptotically optimal Duplication Detection algorithm times.

In this paper, we propose a novel technique for detecting file duplicates on Android devices without worrying about the privacy policy, which limits the access of other applications to files created by a given user. Furthermore, our research provides an extensive experimental analysis concerning the number of duplicate multimedia-related files that are hosted on real-world datasets derived from more than 400 Android devices. Our extensive experimental analysis demonstrates that duplicates can amount to a large percentage of a device's data, demonstrating a pronounced need for novel deduplication techniques.