Giter Club home page Giter Club logo

libscout's Introduction

LibScout

LibScout is a light-weight and effective static analysis tool to detect third-party libraries in Android/Java apps. The detection is resilient against common bytecode obfuscation techniques such as identifier renaming or code-based obfuscations such as reflection-based API hiding or control-flow randomization. Further, LibScout is capable of pinpointing exact library versions including versions that contain severe bugs or security issues.

LibScout requires the original library SDKs (compiled .jar/.aar files) to extract library profiles that can be used for detection on Android apps. Pre-generated library profiles are hosted at the repository LibScout-Profiles.

Unique detection features:

  • Library detection resilient against many kinds of bytecode obfuscation (e.g. obfuscations by ProGuard)
  • Capability of pinpointing the exact library version (in some cases to a set of 2-3 candidate versions)
  • Capability of handling dead-code elimination, by computing a similarity score against baseline SDKs
  • Library API usage analysis upon detection

Over time LibScout has been extended to perform additional analyses both on library SDKs and detected libraries in apps:

  • API compatibility analysis across library versions and lib developers adherence to semantic versioning.
  • Library updatability analysis to infer if and to which extent detected libraries in apps can be updated without code changes based on their API usage.

In addition, there is an Android Studio extension up2dep that integrates the API compatibility information into the IDE to help developers keeping their dependencies up-to-date (and more).

Library History Scraper (./scripts)

The scripts directory contains a library-scraper python script to automatically download original library SDKs including complete version histories from Maven Central, JCenter and custom mvn repositories. The original library SDKs can be used to generate profiles and to conduct library API compatibility analyses (see modules below). Use the library-profile-generator script to conveniently generate profiles at scale.

The scrapers need to be configured with a json config that includes metadata of the libraries to be fetched (name, repo, groupid, artefactid). The scripts/library-specs directory contains config files to retrieve over 100 libraries from maven central and a config to download Amazon libraries and Android libraries from Google's maven repository (350 libraries, including support, gms, ktx, jetpack, ..).

NEW (07/30/19): Added list of 45 ad/tracking libraries with currently 1182 versions (trackers.json).

Detecting (vulnerable) library versions

Ready-to-use library profiles and library meta-data can be found in the repository LibScout-Profiles.

LibScout has builtin functionality to report library versions with the following security vulnerabilities.
The pre-generated profiles for vulnerable versions are tagged with [SECURITY], patches with [SECURITY-FIX].
This information is encoded in the library.xml files that have been used to generate the profiles. We try to update the list/profiles whenever we encounter new security issues. If you can share information, please let us know.

Library Version(s) Fix Version Vulnerability Link
Airpush < 8.1 > 8.1 Unsanitized default WebView settings Link
Apache CC 3.2.1 / 4.0 3.2.2 / 4.1 Deserialization vulnerability Link
Dropbox 1.5.4 - 1.6.1 1.6.2 DroppedIn vulnerability Link
Facebook 3.15 3.16 Account hijacking vulnerability Link
MoPub < 4.4.0 4.4.0 Unsanitized default WebView settings Link
OkHttp 2.1 - 2.7.4
3.0.0- 3.1.2
2.7.5
3.2.0
Certificate pinning bypass Link
Plexus Archiver < 3.6.0 3.6.0 Zip Slip vulnerability Link
SuperSonic < 6.3.5 6.3.5 Unsafe functionality exposure via JS Link
Vungle < 3.3.0 3.3.0 MitM attack vulnerability Link
ZeroTurnaround < 1.13 1.13 Zip Slip vulnerability Link

Identified Issues

On our last scan of free apps on Google Play (05/25/2017), LibScout detected >20k apps still containing one of these vulnerable lib versions. The findings have been reported to Google's ASI program. Unfortunately, the report seemed to be ignored. In consequence, we manually notified many app developers.

Among others, McAfee published a Security Advisory for one of their apps.

LibScout 101

  • LibScout requires Java 1.8 or higher and can be build with Gradle.
  • Generate a runnable jar with the gradle wrapper gradlew (Linux/MacOS) or gradlew.bat (Windows), by invoking it with the build task, e.g. ./gradlew build
    The LibScout.jar is output to the build/libs directory.
  • Some less frequently changing options can be configured via LibScout's config file LibScout.toml.
  • Most LibScout modules require an Android SDK (jar) to distinguish app code from framework code (via the -a switch). Refer to https://developer.android.com/studio/ for download instructions.
  • By default, LibScout logs to stdout. Use the -d switch to redirect output to files. The -m switch disables any text output. Depending on the operation mode (see below), LibScout's results can be written to disk in JSON format or JAVA serialization.
  • LibScout repo structure in a nutshell:

|_ gradlew / gradlew.bat (gradle wrappers to generate runnable LibScout.jar)
|_ assets
|    |_ library.xml (Library meta-data template)
|_ config
|    |_ LibScout.toml (LibScout's config file)
|    |_ logback.xml (log4j configuration file)
|_ data
|    |_ app-version-codes.csv (Google Play app packages with valid version codes)
|_ lib
|    Android axml
|_ scripts
|    |_ library-specs (pre-defined library specs)
|    |_ library-scraper.py   (scraper for mvn-central, jcenter, custom mvn)
|    |_ library-profile-generator.sh (convenience profile generator)
|_ src
    source directory of LibScout (de/infsec/tpl). Includes some open-source,
    third-party code to parse AXML resources / app manifests etc.
  • LibScout supports different use cases implemented as modules (modes of operation). Below a detailed description for each module.

Library Profiling (-o profile)

This module generates unique library fingerprints from original lib SDKs (.jar and .aar files supported). These profiles can subsequently be used for testing whether the respective library versions are included in apps. Each library file additionally requires a library.xml that contains meta data (e.g. name, version,..). A template can be found in the assets directory. For your convenience, you can use the library scraper (./scripts) to download full library histories from Maven repositories. By default, LibScout generates hashtree-based profiles with Package and Class information (omitting methods).

java -jar LibScout.jar -o profile [-a android_sdk_jar] -x path_to_library_xml path_to_library_file

Library Detection (-o match)

Detects libraries in apps using pre-generated profiles. Optionally, LibScout also conducts an API usage analysis for detected libraries, i.e. which library APIs are used by the app or by other libraries (-u switch).
Analysis results can be written in different formats.

  1. the JSON format (-j switch), creates subfolders in the specified directory following the app package, i.e. *com.foo* will create *com/foo* subfolders. This is useful when coping with a large number of apps. For detailed information about the information stored, please refer to the JSON output specification.
  2. the serialization option (-s switch) writes stat files per app to disk (deprecated)
java -jar LibScout.jar -o match -p path_to_profiles [-a android_sdk_jar] [-u] [-j json_dir] [-m] [-d log_dir] path_to_app(s)  

Library API compatibility analysis (-o lib_api_analysis)

Analyzes changes in the documented (public) API sets of library versions.
The analysis results currently include the following information:

Compliance to Semantic Versioning (SemVer), i.e. whether the change in the version string between consecutive versions (expected SemVer) matches the changes in the respective public API sets (actual SemVer). Results further include statistics about changes in API sets (additions/removals/modifcations). For removed APIs, LibScout additionally tries to infer alternative APIs (based on different features).

For the analysis, you have to provide a path to the original library SDKs. LibScout recursively searches for library jars|aars (leaf directories are expected to have at most one jar|aar file and one library.xml file). For your convenience use the library scraper. Analysis results are written to disk in JSON format (-j switch).

java -jar LibScout.jar -o lib_api_analysis [-a android_sdk_jar] [-j json_dir] path_to_lib_sdks

Library Updatability analysis (-o updatability)

This mode is an extension to the match mode. It first detects library versions in the provided apps and conducts a library usage analysis (-u is implied). In addition, it requires library API compat data (via the -l switch) as generated in the lib_api_analysis mode . Based on the lib API usage in the app and the compat info, LibScout determines the highest version that is still compatible to the set of used lib APIs.
Note: The new implementation still lacks some features, e.g. the results are currently logged but not yet written to json. See the code comments for more information.

java -jar LibScout.jar -o updatability [-a android_sdk_jar] [-j json_dir] -l lib_api_data_dir path_to_app(s)

Scientific Publications

For technical details and large-scale evaluation results, please refer to our publications:

If you use LibScout in a scientific publication, we would appreciate citations using these Bibtex entries: [bib-ccs16] [bib-ccs17]

libscout's People

Contributors

reddr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libscout's Issues

NPE when loading profiles, fails when sorting by version name

I get this NPE when working with profiles I generated myself from the attached file.

$ libscout --opmode match --android-sdk /home/fdroid/android-sdk/platforms/android-29/android.jar --libscout-conf /home/fdroid/code/reddr/LibScout/config/LibScout.toml --log-dir /data/ttt-apks/libraries/run-match.sh-1598623728-logs --profiles-dir ./profiles --json-dir /data/ttt-apks/random-google-play//json /data/ttt-apks/random-google-play/
Exception in thread "main" java.lang.NullPointerException
        at de.infsec.tpl.profile.LibProfile$LibProfileComparator.compare(LibProfile.java:68)
        at de.infsec.tpl.profile.LibProfile$LibProfileComparator.compare(LibProfile.java:65)
        at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
        at java.base/java.util.TimSort.sort(TimSort.java:234)
        at java.base/java.util.Arrays.sort(Arrays.java:1515)
        at java.base/java.util.ArrayList.sort(ArrayList.java:1750)
        at de.infsec.tpl.profile.Profile.loadLibraryProfiles(Profile.java:87)
        at de.infsec.tpl.TplCLI.main(TplCLI.java:122)

Here is my proposed fix for --opmode match:

--- a/src/de/infsec/tpl/profile/Profile.java
+++ b/src/de/infsec/tpl/profile/Profile.java
@@ -81,6 +81,10 @@ public abstract class Profile implements Serializable {
                        // de-serialize library profiles
                        for (File f : Utils.collectFiles(profilesDir, new String[]{LibraryProfiler.FILE_EXT_LIB_PROFILE})) {
                                LibProfile lp = (LibProfile) Utils.disk2Object(f);
+                               if (lp == null || lp.description == null) {
+                                   System.err.println("ERROR: " + f + " produces a null profile!");
+                                   continue;
+                               }
                                profiles.add(lp);
                        }
 

PyzeLibrary_2.5.0.libv.zip

Command line parsing failed:

Unable to execute the jar file, please help me, if there I did any wrong in below steps.

  1. Generated .jar using eclipse
  2. copied android API 28 jar file to Libscout\lib\android.jar
  3. downloaded profiles from https://github.com/reddr/LibScout-Profiles.git and extracted to profiles directory.
  4. Copied an apk to the Libscout\apk\123.apk directory
  5. Tried the below command
    java -jar build\LibScout.jar --opmode match -a lib\android.jar -p profiles\profiles\md5\Android -d log $apk\123.apk
  6. I'm getting an error as below
    Command line parsing failed:
    Argument is no valid file or directory!
    usage: LibScout --opmode match -a <path-to-android.jar> $path-to-app(-dir)

command
lib
android

[FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy

Hi, I got an error msg when using the Library Detection function (-o match). I downloaded the pre-generated profile files.

The command is:
java -jar build/libs/LibScout.jar -o match -p ../LibScout-Profiles -c config/LibScout.toml -a sdk/android.jar apks/103443com.rarlab.rar.apk

The error msg is:
20:40:04 INFO LibScoutConfig : This is LibScout 2.3.2
20:40:04 INFO Profile : Load library profiles:
20:41:31 INFO Profile : Loaded 8542 profiles in 1 min, 27 sec, 451 ms
20:41:31 INFO Profile :
20:41:31 INFO LibraryIdentifier : Process app: 103443com.rarlab.rar.apk
20:41:31 INFO LibraryIdentifier : = Manifest Parser =
20:41:31 INFO LibraryIdentifier : Package name: com.rarlab.rar
20:41:31 INFO LibraryIdentifier : Version code: 96
20:41:31 INFO LibraryIdentifier : minSdkVersion: 19
20:41:31 INFO LibraryIdentifier : targetSdkVersion: 29
20:41:31 INFO LibraryIdentifier : SharedUserId: - none -
20:41:31 INFO LibraryIdentifier : Library dependencies: - none -
20:41:31 INFO LibraryIdentifier :
20:41:31 INFO LibraryIdentifier : Found 402 unique libraries in 8542 library profiles
using /var/folders/zt/qpw4hbwn0_z6cl37q5v4p5d80000gn/T/name1738988254421587180_classes.dex
20:41:35 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
20:41:35 ERROR TplCLI : com.ibm.wala.ipa.cha.ClassHierarchyException: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
at com.ibm.wala.ipa.cha.ClassHierarchy.(ClassHierarchy.java:296)
at com.ibm.wala.ipa.cha.ClassHierarchy.(ClassHierarchy.java:209)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.make(ClassHierarchyFactory.java:78)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.makeWithRoot(ClassHierarchyFactory.java:89)
at com.ibm.wala.ipa.cha.ClassHierarchyFactory.makeWithRoot(ClassHierarchyFactory.java:53)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.createClassHierarchy(LibraryIdentifier.java:112)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:143)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84)
at de.infsec.tpl.TplCLI.main(TplCLI.java:141)

I googled this error msg, it seems to be a runtime error for wala. See (http://wala.sourceforge.net/wiki/index.php/UserGuide:Troubleshooting) The possible solution is to change the java_runtime_dir property in the wala.properties file. However, I guess you didn't use this properties file, right? Please correct me if it's wrong. By the way, I am using java 1.8.0_60.

Thank you very much!

Multi-dex APKs not completely processed

Hi,

I'm trying to use LibScout with an APK containing 100+ dex files. I found out (after increasing the log levels) that currently only the first 10 classes.dex files are processed, resulting in an incomplete package tree for APKs with 10+ dex files.
The cause of this is the regex in the WALA code: classes[1-9]{0,1}\\.dex.
I think it might be a good idea to get rid of this limitation.

I changed the regex in the DexFileModule.class file to classes[0-9]{0,9}\\.dex and every dex file is successfully processed now. But now I get an exception when exporting to JSON:

ERROR TplCLI                    : [FATAL EXCEPTION] analysis aborted: null
ERROR TplCLI                    : java.lang.NullPointerException
        at de.infsec.tpl.profile.ProfileMatch$Export.<init>(ProfileMatch.java:120)
        at de.infsec.tpl.profile.ProfileMatch.export(ProfileMatch.java:132)
        at de.infsec.tpl.stats.AppStats$Export.<init>(AppStats.java:130)
        at de.infsec.tpl.stats.AppStats.export(AppStats.java:48)
        at de.infsec.tpl.utils.Utils.obj2JsonFile(Utils.java:487)
        at de.infsec.tpl.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:207)
        at de.infsec.tpl.TplCLI.main(TplCLI.java:155)

Also, I know that you use Joana's WALA fork but any plans to publish your patches?

Update database

isn't possible to update the database when running
java -jar build/LibScout.jar -o db -p profiles/md5/ -s stats/
because i got this error message
12:19:40 INFO SQLStats : Generate DB from stats mode!
12:19:40 INFO SQLStats : Loaded 4965 profiles from disk
12:19:40 WARN SQLStats : DB file appStats.sqlite exists! -- No updates will be performed -Abort!
so i have to delete the existant file each time i wanna run the command

data parsing error

In XMLParser.java, the date format is "dd.MM.yyyy", but in the xml template, the suggested date format is 'DD/MM/YYYY'. Not a big deal, but may be good to have it fixed.

gibberish output

i tried to create a profile and try to find a match but i got this (ps the apk used and the second phase use the given retrofit library)

java -jar ../build/LibScout.jar -o profile -a ../lib/android-23.jar -x /root/Downloads/retrofit.xml /root/Downloads/retrofit-2.4.0.jar
10:18:48,756 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
10:18:48,757 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:18:48,757 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
10:18:48,763 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
10:18:48,918 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

java -jar ../build/LibScout.jar -o match -a ../lib/android-23.jar /root/Downloads/dgi_proguard.apk
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:24:45,349 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
10:24:45,355 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
10:24:45,514 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

java.lang.ArrayIndexOutOfBoundsException: 636
at android.content.res.StringBlock.getShort(Unknown Source)
at android.content.res.StringBlock.getString(Unknown Source)
at android.content.res.AXmlResourceParser.getName(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.loadClassesFromBinaryManifest(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest$1.handleManifest(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.handleAndroidManifestFile(Unknown Source)
at de.infsec.tpl.manifest.ProcessManifest.loadManifestFile(Unknown Source)
at de.infsec.tpl.LibraryIdentifier.parseManifest(Unknown Source)
at de.infsec.tpl.LibraryIdentifier.identifyLibraries(Unknown Source)
at de.infsec.tpl.TplCLI.main(Unknown Source)

confusing error message when LibScout finds no APKs in a dir

When running LibScout on a directory that has no APKs, it gives an error that made me think that I had wrong command line flags:

$ mkdir /tmp/newdir
$ java -jar build/LibScout.jar -o match -a lib/android-23.jar -p profiles  /tmp/newdir/
 
Command line parsing failed:
You have to provide a path to a single application file or a directory

no profiles found

the tool was working just fine but today when i run a new test here is what it said:

java -jar LibScout.jar -o match -a ../lib/android-23.jar /root/Downloads/dgi_proguard.apk
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
11:36:07,454 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
11:36:07,461 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Setting up default configuration.
11:36:07,623 |-ERROR in ch.qos.logback.classic.joran.JoranConfigurator@5d22bbb7 - Could not open [./logging/logback.xml]. java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at java.io.FileNotFoundException: ./logging/logback.xml (No such file or directory)
at at java.io.FileInputStream.open0(Native Method)
at at java.io.FileInputStream.open(FileInputStream.java:195)
at at java.io.FileInputStream.(FileInputStream.java:138)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:75)
at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:68)
at at de.infsec.tpl.TplCLI.initLogging(Unknown Source)
at at de.infsec.tpl.TplCLI.main(Unknown Source)

No profiles found in ./profiles. Check your settings!

i have all the files in place './logging/logback.xml' and i have the profiles you offered. i don't know what is the source of this error.

where can i find [logback.groovy] and [logback-test.xml]

android-X.jar

I was trying around different versions of android.jar and i can't figure out the difference when using LibScout.
so i wonder how does the android.jar version used either on profiling or matching affect the result?
what would happend if i create a profile with a given version and try to match it using an other android-X.jar version?

How to run the -profile

I do not know much about java and mobile app development. So I do not understand terms like android.jar file, library-description-files, etc. however, I would like to run this tool.

I have the apk file of the android app I want to profile. I used the java -jar LibScout.jar -o profile ... command but I do not know what or how to generate the SDK android.jar , library_xml and library_file.

Please help ASAP?

Unable to compile

I pulled the newest commit and tried to recompile the project, but i get failure messages from the java compiler.

clean:
[delete] Deleting directory /home/markus/bachelor/tools/LibScout/bin
[delete] Deleting directory /home/markus/bachelor/tools/LibScout/build
compile:
[mkdir] Created dir: /home/markus/bachelor/tools/LibScout/bin
[echo] Using Java version 1.8.
[javac] Compiling 65 source files to /home/markus/bachelor/tools/LibScout/bin
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:22: error: cannot find symbol
[javac] import android.util.TypedValue;
[javac] ^
[javac] symbol: class TypedValue
[javac] location: package android.util
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:28: error: cannot find symbol
[javac] import android.util.TypedValue;
[javac] ^
[javac] symbol: class TypedValue
[javac] location: package android.util
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:215: error: cannot find symbol
[javac] if (valueType!=TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:288: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_STRING) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:303: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_FLOAT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:313: error: cannot find symbol
[javac] if (valueType>=TypedValue.TYPE_FIRST_INT &&
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:314: error: cannot find symbol
[javac] valueType<=TypedValue.TYPE_LAST_INT)
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/android/content/res/AXmlResourceParser.java:328: error: cannot find symbol
[javac] if (valueType==TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXmlResourceParser
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:129: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_STRING) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:132: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_ATTRIBUTE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:135: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_REFERENCE) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:138: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_FLOAT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:141: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_INT_HEX) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:144: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_INT_BOOLEAN) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:147: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_DIMENSION) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:149: error: cannot find symbol
[javac] DIMENSION_UNITS[data & TypedValue.COMPLEX_UNIT_MASK];
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:151: error: cannot find symbol
[javac] if (type==TypedValue.TYPE_FRACTION) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:153: error: cannot find symbol
[javac] FRACTION_UNITS[data & TypedValue.COMPLEX_UNIT_MASK];
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:155: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_COLOR_INT && type<=TypedValue.TYPE_LAST_COLOR_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:155: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_COLOR_INT && type<=TypedValue.TYPE_LAST_COLOR_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:158: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_INT && type<=TypedValue.TYPE_LAST_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/AXMLPrinter.java:158: error: cannot find symbol
[javac] if (type>=TypedValue.TYPE_FIRST_INT && type<=TypedValue.TYPE_LAST_INT) {
[javac] ^
[javac] symbol: variable TypedValue
[javac] location: class AXMLPrinter
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: /home/markus/bachelor/tools/LibScout/src/pxb/android/axml/StringItems.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 22 errors

Comparative research on two applications that point out security errors with libscout assistance

Good afternoon, I'm a bachelor's student in computer science at the Universidade de Brasilia and I'm doing research using libscout as a tool for detecting external libraries. My goal is to verify if the security errors pointed out by the cognicrypt (https://www.eclipse.org/cognicrypt/) and cryptoguard (https://www.cryptoguard.com/) tools are mostly native or external code. I decided to use libscout as a helper tool thanks to the article 'Automated Third-Party Library Detection for Android Applications: Are We There Yet?'. I already got some results, however, talking to my tutor and analyzing some of the tool's outputs, we observed that sometimes the analysis of a certain application registers external libraries only in the "Library Identifier" step and does not show in the "ProfileMatch" step. This is not something that prevents the completion of my undergraduate work, however, it would be interesting to understand why this happens. I've already tried changing the proximity value of the cluster, nevertheless, the library continues to appear only in its identification and not in the match step.
The following attachment illustrates this. Okio3 and HTTP show in the Library Identify step but not in the ProfileMatch.

It is worth adding that I am using the flags to dull a full profile match and that I am using the profile dataset available here on the wiki.

libscout_result.txt

Run LibScout from any directory

#10 Before running the LibScout jar, and in order to make it work you should be in the LibScout/ directory or you will get this ERROR : Could not open [./logging/logback.xml]. this error means that when you were developping the solution you gave the code the partial parth (logical path using the ./ unix notion).

this situation prevent this program to be run from outside the LibScout folder, which means less flexibility. i hope you guys could solv this. you can add an option for the command java -jar LibScout.jar -o match -a lib/android-X.jar -p -s -d $someapp.apk in which we can indicate the logback.xml file location

json format

when i run LibSout using -j option:
java -jar build/LibScout.jar -o match -a lib/android-23.jar -p profiles/md5/ -j json/ -m /root/Downloads/dgi_proguard.apk

inside the json/ folder i found the json files stored in other folders arborescence, i wonder is it based on the application package?

Lacking information in generated json file

after i execute a matching test with an APK file and generate the result with the json option, the generated json file didn't contain the comments info about libraries i've put previousely inside library.xml files.

is there any option to get this comments after the tests

java.lang.NullPointerException

19:47:55 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: null 19:47:55 ERROR TplCLI : java.lang.NullPointerException at java.util.TreeMap.getEntry(TreeMap.java:347) at java.util.TreeMap.containsKey(TreeMap.java:232) at java.util.TreeSet.contains(TreeSet.java:234) at de.infsec.tpl.modules.libmatch.LibCodeUsage.checkUsage(LibCodeUsage.java:78) at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:204) at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84) at de.infsec.tpl.TplCLI.main(TplCLI.java:141)

Library version

when i run my test i just got the list of existing libraries but i don't got any information about the exact version of the library. can you help plz?

An error in the name of json output file generated by Library Detection (-o match)

I would like to express my appreciation for the creation of this tool in the first place.

I found an minor error regarding the json file name created by Library Detection. In the process of generating the JSON output file through Library Detection (-o match), any occurrence of '.aar' or '.jar' in the file name is automatically removed. For instance, if the APK name is "com.aarsomething", the final JSON file will be named as "comsomething.json".

Thank you.

LibScout cannot detected confused apk

I downloaded an open source code from gitee (https://gitee.com/DaLianZhiYiKeJi/xpay?_from=gitee_search), and I divided it into two steps to verify the libscout test results

Step 1: I compile the downloaded code into apK, and then use libscout to analyze the apK. The result is that the SDK contained in the APK can be detected. For example, the okhttp library was successfully detected 。

Step 2:, I compiled the downloaded code into apK using ProGuard confusion, and then analyzed the apK using LibScout. The okHTTP library I just detected was not detectable.

The question is: can LibScout support this scenario after ProGuard confusion ?

Thanks。

[FATAL EXCEPTION] analysis aborted: null

Hi, when I use Library Detection (-o match), It seems appear some errors.

I input the command:
java -jar ./build/libs/LibScout.jar -o match -p ./profiles/Cloud -a ./sdk/android.jar -j ./json app.apk
The profiles are generated by myself, from some SDK jar files.
I also change the asserts/library.xml.

And then errors appear:
the command output is
`
11:07:08 DEBUG HashTree : Generate hash tree..

11:07:14 DEBUG HashTree : - generated 698 package hashes.

11:07:14 DEBUG HashTree : - generated 8939 clazz hashes.

11:07:14 DEBUG HashTree : - generated 60936 method hashes.

11:07:14 DEBUG HashTree : => Library Hash: b66cf0802b150b4e24eac7a566f61709

11:07:14 INFO AppProfile : - generated app hash trees (in 5 sec, 636 ms)

11:07:14 INFO AppProfile :

11:07:14 INFO LibraryIdentifier : = Scan for library root packages (heuristic) =

11:07:14 INFO LibraryIdentifier :

11:07:14 INFO LibraryIdentifier : = Match profiles =

11:07:14 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: null

11:07:14 ERROR TplCLI : java.lang.NullPointerException
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.identifyLibraries(LibraryIdentifier.java:183)
at de.infsec.tpl.modules.libmatch.LibraryIdentifier.run(LibraryIdentifier.java:84)
at de.infsec.tpl.TplCLI.main(TplCLI.java:141)
`
Please help me why does this issue occur and how to solve this issue?

[FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy

PS C:\Users\z\Desktop\Lib-tools\LibScout-2.3.2> java -jar .\LibScout.jar -o profile -x C:\Users\z\Desktop\ttst\library.xml C:\Users\z\Desktop\ttst\aboutlibraries-6.0.0.jar
22:08:01 INFO LibScoutConfig : This is LibScout 2.3.2
22:08:01 INFO LibraryProfiler : Process library: aboutlibraries-6.0.0.jar
22:08:01 INFO LibraryProfiler : Library description:
22:08:01 INFO LibraryProfiler : name: aboutlibraries
22:08:01 INFO LibraryProfiler : category: Utilities
22:08:01 INFO LibraryProfiler : version: 6.0.0
22:08:01 INFO LibraryProfiler : release-date: 21.05.2017
22:08:01 INFO LibraryProfiler : comment: no
22:08:02 ERROR TplCLI : [FATAL EXCEPTION] analysis aborted: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
22:08:02 ERROR TplCLI : com.ibm.wala.ipa.cha.ClassHierarchyException: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy
how can i solve this error

More detailed reporting of detected mappings?

Is there a way to retrieve the mappings from obfuscated to real class/package name? This would be very useful if I want to do some analysis on the library files as found in the app - which for various reasons may not 100% match the files in the original library JAR (e.g. due to dead-code elimination).

Example:

{
	"libName": "Facebook",
	"package-mappings": {
		"a.a.a": "com.facebook.applinks".
		"a.a.b": "com.facebook.login",
		...
	},
	...
}

Or even

{
	"libName": "Facebook",
	"class-mappings": {
		"a.a.a.a": "com.facebook.applinks.AppLinkData".
		"a.a.a.b": "com.facebook.applinks.FacebookAppLinkResolver",
		...
	},
	...
}

Outputting the profiles of library-profile-generator.sh

Hi!
Iv'e ran the crawler of the android repo json file, and downloaded the libs.
Since there are new versions of the libraries, i want to create new profiles for each, and then run the match command on an apk.
I ran the library-profile-generator.sh, and it works fine, but there is not output of the new profiles to anywhere.
Would love some help,
Thank you.

$python3 ./scripts/library-scraper.py ./scripts/library-specs/google-libraries.json


== maven/jcenter scraper ==
- Load library info from ./scripts/library-specs/google-libraries.json
- Store libs to my-lib-repo/
- Update libraries (skip alpha/beta versions):
  # check library com.google.android.play::core [Android]   (g:"com.google.android.play" AND a:"core")
    - retrieved meta data for 31 versions:
       - update version: 1.6.5   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.6.5
       - update version: 1.7.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.0
       - update version: 1.7.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.1
       - update version: 1.7.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.2
       - update version: 1.7.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.7.3
       - update version: 1.8.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.0
       - update version: 1.8.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.2
       - update version: 1.8.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.8.3
       - update version: 1.9.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.9.0
       - update version: 1.9.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.9.1
       - update version: 1.10.0   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.0
       - update version: 1.10.1   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.1
       - update version: 1.10.2   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.2
       - update version: 1.10.3   type: aar  date: n/a  target-dir: my-lib-repo/Android/com.google.android.play::core/1.10.3
  # check library com.android.support.constraint::constraint-layout-solver [Android]   (g:"com.android.support.constraint" AND a:"constraint-layout-solver")
    - retrieved meta data for 10 versions:
       - update version: 1.0.2   type: jar  date: n/a  target-dir: my-lib-repo/Android/com.android.support.constraint::constraint-layout-solver/1.0.2

am I correct? If not pleae help me

Hi Derr,

I'm trying to use Libscout for detecting the third party dependencies from apk file. However, I'm not succeeded yet by using the Libscout. Below are the steps I did, after downloading Libscout code.

Step 1: Generated the jar file using the ant script
Step 2: Downloaded the Android SDK from the Android Studio
Step 3: Downloaded Profiles from "LibScout Profiles" https://github.com/reddr/LibScout-Profiles
Step 4: Executed the below command to run the Libscout.jar on apk file, find the results as an attachment.
123.log
Calendar.log
diva-beta.log
LeafPic.log

java -jar c:\LibScout\build\LibScout.jar -opmode match -a c:\LibScout\lib\android.jar (downloaded in step2) -p c:\LibScout\profiles\profiles\md5\Android (Downloaded in Step3) -d c:\LibScout\log c:\LibScout\apk

Please correct me if there is any process wrong,

Also, I would like to request you to share "Original Library SDK" with me.

LibScout doesn't detect Vulnerability

Dear LibScout Team

Currently, I am integrating LibScout into MobSF for a university project. In my testcase, I have an Android APK depending on OkHttp version 2.3.0. Referring to this repository's README.md, this version should be detected as vulnerable. However, this does not happen.

Specifics:
Java OpenJDK 8, tested on Ubuntu 20 LTS x86-64 and macOS Mojave

How to replicate:
Since the vulnerability is still present in the app's latest version, I am not going to disclose the exact APK used. Therefore, the placeholder VULN.apk

(executed in LibScout root directory, PROFILE_PATH is pointing to this):

java -jar build/libs/LibScout.jar -a <SDK_PATH> -p <PROFILE_PATH> -o match <VULN.apk>

Enabling the comments in the config file does display some information, but not the important [SECURITY] section. Has this been found after the latest update to above repo, or are we using the tool wrongly?

Yours sincerely

mixed up paths when outputting JSON files

In the WU run on 1.2TB of APKs, I'm using the JSON output. It outputs mostly using a directory hierarchy based on the segments of the Java Package Name. But sometimes it outputs straight into the json/ dir:

$ ls -1 json/org.mariotaku.twidere*.json json/org/mariotaku/*.json
json/org/mariotaku/org.mariotaku.imageviewergl_1_1.json
json/org/mariotaku/org.mariotaku.twidere_135_135.json
json/org/mariotaku/org.mariotaku.twidere_166_166.json
json/org/mariotaku/org.mariotaku.twidere_181_181.json
json/org/mariotaku/org.mariotaku.twidere_196_196.json
json/org/mariotaku/org.mariotaku.twidere_213_213.json
json/org/mariotaku/org.mariotaku.twidere_260_260.json
json/org/mariotaku/org.mariotaku.twidere_370_370.json
json/org/mariotaku/org.mariotaku.twidere_371_371.json
json/org/mariotaku/org.mariotaku.twidere_379_379.json
json/org.mariotaku.twidere_389_0.json
json/org.mariotaku.twidere_393_0.json
json/org.mariotaku.twidere_394_0.json

Use WALA master instead of custom version

LibScout uses JOANA's WALA fork that adopted and improved ScanDroid's dex frontend to allow Android app analyses. At the time of integration, the state was still immature and a lot of integral parts were missing, i.e. multi-dex support. During LibScout's development, some issues have been fixed and workaround patches have been applied to a custom version (lib/wala.jar) that is statically linked during compilation time.

Relying on a custom version is definitively sub-optimal, both in terms of integration and maintainability.
To be able to move from this old and customized WALA to upstream WALA it needs to be checked, whether all features currently used in LibScout (through custom patches etc.) are supported in the WALA master.

I already checked some of LibScout's patches against WALA master. I commented on the following (important) ones that are either unresolved or should already work with the latest WALA version:

  • [wala_dex_dexfilemodule_multidex_support.patch]
    Adds multi-dex support to WALA dex frontend that was based on dexlib1
    [Status: resolved] WALA master uses dexlib2 that has builtin support for multi-dex. This would also solve #19 .

  • [wala_cha_workaround_unresolvable_superclasses.patch]
    WALA's CHA builder discards classes when their superclasses cannot be resolved. This affects library classes that extend other library classes not being in the classpath. However, the same library class might be resolved within an app where all dependencies are statically linked. This is a workaround patch that forces the CHA builder to keep such classes by changing it's superclass to java.lang.Object to allow code reuse detection.
    [Status: resolved] Addressed in issue wala/WALA#322. Instead of rewriting to Object, WALA's patch creates a synthetic PhantomSuperClass.

  • [wala_add_check_synthethic_methods.patch]
    Addition of isMethodSynthetic call as a workaround (see also #16). This became necessary since a proper implementation of isSynthetic for IMethod subclasses (DexIMethod, ShrikeBTMethod,..) caused some unintended side-effects. Did not dig deeper, as this affects all IMethod subclasses.
    [Status: not implemented] WALA master still returns false for isSynthetic calls.

  • [wala_dex_ssa_fix_for_methods_without_parameters.patch]
    WALA creates an empty IR for static initializer methods without arguments. This patches fixes the SSA Builder.
    [Status: probably not, needs to be checked]

I'm going to test WALA master and create issues for the open problems.
CC @eighthave

output json format

i cloned LibScout after last commit and when running the command:
java -jar build/libs/LibScout.jar -o match -a lib/android-23.jar -p profiles/ -j ./json/ app-release.apk
the json files are no more created. i already checked privileges issues and gives writing privileges to all for json folder and subfolders and still not working.
i tried it with differents androif-sdk versions.

Library Version - Incorrect

Hi,
I am scanning a mobile app using LibScout. We are using the okhttp3 library and the version used is 3.10.0. This is done using gradle in Android Studio : 'com.squareup.okhttp3:mockwebserver:3.10.0'.

When I scan this binary using LibScout it reports 3.6.0 version. I was wondering which file is used within okhttp3 to fingerprint the version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.