New upstream version 2.0.0

parent 6dd50047
......@@ -13,16 +13,38 @@ For technical details and large-scale evaluation results, please refer to our pu
> Reliable Third-Party Library Detection in Android and its Security Applications<br>
> https://www.infsec.cs.uni-saarland.de/~derr/publications/pdfs/derr_ccs16.pdf
For comments, feedback, etc. contact: Erik Derr [lastname@cs.uni-saarland.de]
If you use LibScout in a scientific publication, we would appreciate citations using this Bibtex entry: [[bib]](https://www.infsec.cs.uni-saarland.de/~derr/publications/bib/derr_ccs16.bib)<br>
## Library Profiles and Scripts
To facilitate usage of LibScout we are happy to release our datasets to the community. <br>
You can find the following resources in the data/scripts directory:<br>
### Library Profiles (last updated: 06/27/2017)
You can find all <b>library profiles</b> (ready-to-use) for lib detection in apps in the data directory as compressed .zip file.<br>
It currently includes <b>205</b> unique libraries and <b>3,071</b> library versions.<br> For convenience, data/library-data.csv contains a
complete list of library/-versions including meta data such as release dates.
### Scripts (scripts/mnv-central)
The scripts directory further contains a python script to automatically download original library SDKs including complete version histories from maven-central.<br>
The set of libraries we currently retrieve is stored in a json file.<br>
Due to copyright reasons we cannot publicy provide the original library SDKs. If you are interested in this data, send us an email.<br>
We also welcome contributions to LibScout or our library database (either original SDKs or scripts for automatic retrieval from sources other than mvn central).<br><br>
Contact us for comments, feedback, how to contribute: Erik Derr [lastname@cs.uni-saarland.de]
## LibScout Repo Structure
<pre><code>
|_ build.xml (ant build file to generate runnable .jar)
|_ data
| |_ library-data.sqlite (library meta data)
| |_ library-profiles.zip (all library profiles)
|_ data (library profiles and supplemental data sets)
| |_ library-data.csv (library meta data)
| |_ library-profiles-21.06.zip (all library profiles)
| |_ app-version-codes.csv (app packages with valid version codes)
|_ lib
| pre-compiled WALA libs, Apache commons*, log4j, Android SDK
......@@ -40,48 +62,60 @@ For comments, feedback, etc. contact: Erik Derr [lastname@cs.uni-saarland.de]
## Getting Started
<ol>
<li>LibScout requires Java 1.7 or higher. If you're using OpenJDK you need to use either 1.7 <b>or</b> 1.9 (1.8 seems to have some strange bytecode verification bug)<br>
A runnable jar can be generated with the build.xml</li>
<li>LibScout has three modes of operation:<br>
<li>LibScout requires Java 1.8 or higher. A runnable jar can be generated with the ant script <i>build.xml</i></li>
<li><b>Modes of operation (provided via -o switch):</b><br>
Profile and Match mode require an Android SDK, provided via the -a switch, to distinguish app code from framework code.<br>
For your convenience, you can use the one provided in the lib directory.
<ol type="a">
<li>
Generate library profiles from original library SDKs:<br>
<pre>java -jar LibScout.jar -o profile -a lib/android-X.jar -x ${lib-dir/library.xml} ${lib-dir/lib.jar} </pre>
Library Profiling (-o profile)<br>
Generate library profiles from original library SDKs (.jar and .aar files supported). Besides the library file, this mode requires a library.xml that
contains some meta-data about the library (e.g. name, version, etc.). A library.xml template can be found in the assets directory. Use the -v switch to generate trace profiles,
i.e. profiles with class and method signatures, where methods are limited to public methods (Trace profiles are required as input for the library api analysis):<br>
<pre>java -jar LibScout.jar -o profile -a lib/android-X.jar -x ${lib-dir/library.xml} ${lib-dir/lib.[jar|aar]} </pre>
</li>
<li>
Detect libraries in apps using pre-generated profiles (log to directory + serialize results):<br>
Library Matching (-o match)<br>
Detect libraries in apps using pre-generated profiles (this example logs to directory + serializes results):<br>
<pre>java -jar LibScout.jar -o match -a lib/android-X.jar -p &lt;path-to-lib-profiles&gt; -s -d &lt;log-dir&gt; $someapp.apk </pre>
</li>
<li>
Database creation (-o db)<br>
Generate a SQLite database from library profiles and serialized app stats:<br>
<pre>java -jar LibScout.jar -o db -p &lt;path-to-lib-profiles&gt; -s &lt;path-to-app-stats&gt; </pre>
</li>
<li>
Library API robustness analysis (-o lib_api_analysis)<br>
Analyzes changes in the set of library APIs across versions (additions/removals/modifcations). Checks for <a href="http://semver.org">SemVer</a> compliance, i.e. whether the change in the version string matches
the changes in the public API set. SemVer compliance statistics are logged, while API robustness data is written out in JSON format (use -j switch to configure).
If you use this mode you have to provide trace profiles (generated via -o profile -v).<br>
<pre>java -jar LibScout.jar -o lib_api_analysis -p &lt;path-to-lib-profiles&gt; -j &lt;json-output-path&gt; </pre>
</li>
</ol>
</li>
<li>
Some classes to start with:
<ul>
<li><b>de.infsec.tpl.TplCLI</b>: &nbsp;&nbsp; Starting class including CLI parsing and logging init</li>
<li><b>de.infsec.tpl.LibraryHandler</b>:&nbsp;&nbsp; Starting class to extract library profiles</li>
<li><b>de.infsec.tpl.LibraryIdentifier</b>:&nbsp;&nbsp; Code to match lib profiles and application bytecode</li>
<li><b>de.infsec.tpl.hash.HashTree</b>:&nbsp;&nbsp; main data structures used for profiles</li>
<li><b>Output formats:</b> There are three different output formats available (individually configurable).
<ol type="a">
<li>
<b>Textual logging</b>. Per default, LibScout logs to stdout. Use the -d switch to redirect output to files. The -m switch disables any text output.
</li>
<li>
<b>JSON</b> output can be enabled via -j switch.
</li>
<li>
The analysis results per app can also be <b>serialized</b> to disk using the -s switch. This is particularly useful for large-scale evaluations.
After all apps have been processed, you can use operation mode c) to generate one convenient SQLite file from the serialized results
(the DB structure can be found in class de.infsec.tpl.stats.SQLStats).
</li>
</ol>
</li>
<li><i>How to aggregate per-app results during large-scale evaluation?</i><br>
While the tool consumes one app at a time, it can serialize the app results to disk. Using
operation mode c), LibScout loads all app results to generate one convenient SQLite file<br>
(the DB structure can be found in class de.infsec.tpl.stats.SQLStats)
<li>
If you are interested in digging into the source, here are some classes to start with:
<ul>
<li><b>de.infsec.tpl.TplCLI</b>: &nbsp;&nbsp; Starting class including CLI parsing and logging init</li>
<li><b>de.infsec.tpl.LibraryProfiler</b>:&nbsp;&nbsp; Starting class to extract library profiles</li>
<li><b>de.infsec.tpl.LibraryIdentifier</b>:&nbsp;&nbsp; Code to match lib profiles and application bytecode</li>
<li><b>de.infsec.tpl.hash.HashTree</b>:&nbsp;&nbsp; main data structures used for profiles</li>
</ul>
</li>
</ol>
## Library Profiles
While we can not make the original library SDks publicly available for legal reasons, we provide the following:<br>
<ul>
<li>all library profiles (ready-to-use for detection in apps)&nbsp;&nbsp; [data/library-profiles.zip]</li>
<li>an accompanying SQLite DB with parsed library data (name, version, release date, ..)&nbsp;&nbsp; [data/library-data.sqlite]</li>
<li>a python script to automatically download complete version histories from maven-central
incl. config script&nbsp;&nbsp; [scripts/mvn-central/mvn-central-crawler.py]</li>
</ul>
</ol>
<!--
Library description template
To use in profile mode (-o profile).
Put a copy of this file in each directory with an original library SDK (.jar|.aar).
-->
<?xml version="1.0"?>
<library>
<!-- library name, e.g. "Google Admob" -->
<name></name>
<!-- category, any of "Advertising", "Analytics", "Android", "Cloud", "SocialMedia", "Utilities" -->
<category></category>
<!-- optional: version string, e.g. "4.0.4" -->
<version></version>
<!-- optional: date (format: DD/MM/YYYY) -->
<releasedate></releasedate>
<!-- optional: comment -->
<comment></comment>
</library>
......@@ -37,37 +37,16 @@
<include name="android/util/AttributeSet.class" />
</patternset>
</unjar>
<unjar src="${lib.dir}/joana.api.jar" dest="${bin.dir}">
<patternset>
<exclude name="com/ibm/wala/cast/" />
<exclude name="edu/kit/joana/api/" />
<exclude name="edu/kit/joana/ui/" />
<exclude name="junit/" />
<exclude name="ch/" /> <!-- logback -->
<exclude name="net/"/> <!-- cscott -->
<exclude name="gnu/"/> <!-- trove -->
<exclude name="stubs/"/> <!-- stubs -->
<exclude name="tests/"/> <!-- tests -->
<exclude name="org/junit/" />
<exclude name="org/slf4j/" />
<exclude name="org/apache/commons/io/" />
</patternset>
</unjar>
<unjar src="${lib.dir}/wala-dalvik.jar" dest="${bin.dir}">
<patternset>
<include name="com/ibm/" />
<include name="com/google/common/base/*.class" />
<include name="org/" />
</patternset>
</unjar>
<unjar dest="${bin.dir}">
<fileset dir="${lib.dir}">
<include name="**/commons-cli-1.2.jar" />
<include name="**/commons-io-2.4.jar" />
<!-- <include name="**/guava-20.0.jar" /> -->
<include name="**/gson-2.8.0.jar" />
<include name="**/sqlite-jdbc-3.7.15-SNAPSHOT.jar" />
<include name="**/java-semver-0.9.0.jar"/>
<include name="**/wala.jar" /> <!-- pre-compiled wala library including dex frontend -->
</fileset>
<fileset dir="${lib.dir}/logging">
<include name="**/*.jar" />
......
This diff is collapsed.
......@@ -8,7 +8,10 @@
<logger name="de.infsec.tpl.profile.ProfileMatch" level="info"/>
<logger name="de.infsec.tpl.hash.HashTree" level="debug"/>
<logger name="de.infsec.tpl.utils.WalaUtils" level="info"/>
<logger name="de.infsec.tpl.TplCLI" level="info"/>
<logger name="de.infsec.tpl.LibCodeUsage" level="debug"/>
<logger name="de.infsec.tpl.eval.LibraryApiAnalysis" level="info"/>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
......
/*
* Copyright (c) 2015-2016 Erik Derr [derr@cs.uni-saarland.de]
* Copyright (c) 2015-2017 Erik Derr [derr@cs.uni-saarland.de]
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at
......@@ -16,7 +16,7 @@ package de.infsec.tpl;
import java.util.Collection;
// NECESSARY?
public interface IFilter<T> {
public Collection<T> filter(Collection<T> input);
}
/*
* Copyright (c) 2015-2017 Erik Derr [derr@cs.uni-saarland.de]
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under
* the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package de.infsec.tpl;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.ibm.wala.classLoader.CallSiteReference;
import com.ibm.wala.classLoader.CodeScanner;
import com.ibm.wala.classLoader.IClass;
import com.ibm.wala.classLoader.IMethod;
import com.ibm.wala.ipa.cha.IClassHierarchy;
import com.ibm.wala.shrikeCT.InvalidClassFileException;
import de.infsec.tpl.hash.AccessFlags;
import de.infsec.tpl.profile.ProfileMatch;
import de.infsec.tpl.utils.Utils;
import de.infsec.tpl.utils.WalaUtils;
/**
* Library Code Usage Analysis
* Checks which library code is used by the application in terms of API calls
* To this end, for each full match (partial matches are excluded) it is checked which calls are used within
* the code base that does not belong the identified library root package name.
*/
public class LibCodeUsage {
private static final Logger logger = LoggerFactory.getLogger(de.infsec.tpl.LibCodeUsage.class);
// TODO
// how to deal with libs with ambiguous root pckg (currently excluded)
// -> need to check all matched package name for such libs
public static void checkUsage(final IClassHierarchy cha, final List<ProfileMatch> results) {
logger.info("== Check lib usage ==");
long starttime = System.currentTimeMillis();
// get unique libraries (multiple lib versions with an exact match have the same API, thus checking only one of them suffices)
final HashMap<String,ProfileMatch> lib2Profile = new HashMap<String,ProfileMatch>(); // libname -> profile
final HashMap<String,String> rootPckg2Lib = new HashMap<String,String>(); // root package -> libname
for (ProfileMatch pm: results) {
if (!pm.doAllConfigsMatch()) continue;
String libName = pm.lib.description.name;
if (!lib2Profile.containsKey(libName)) {
lib2Profile.put(libName, pm);
}
String rootPckg = pm.getMatchedPackageTree().getRootPackage();
// TODO: currently exclude libs with ambiguous root packages
if (!LibraryIdentifier.ambiguousRootPackages.contains(rootPckg) && !rootPckg2Lib.containsKey(rootPckg)) {
rootPckg2Lib.put(rootPckg, libName);
}
}
// shortcut - if there are no lib matches there is no need to walk over the bytecode
if (lib2Profile.isEmpty()) {
logger.info(Utils.INDENT + ">> lib code usage analysis done - No libraries matched to scan for (" + Utils.millisecondsToFormattedTime(System.currentTimeMillis() - starttime) + ")");
return;
}
// scan code once
// library root package -> signature, access specifier
Map<String, TreeMap<String, Integer>> usedMethods = new TreeMap<String, TreeMap<String, Integer>>();
Map<String, Set<String>> unresolvableMethods = new TreeMap<String, Set<String>>();
// iterate all classes
for (Iterator<IClass> it = cha.iterator(); it.hasNext(); ) {
IClass clazz = it.next();
if (!WalaUtils.isAppClass(clazz)) continue;
// iterate all methods
for (IMethod im: clazz.getDeclaredMethods()) {
if (im.isAbstract() || im.isNative()) continue;
try {
// iterate all call sites
for (CallSiteReference csr: CodeScanner.getCallSites(im)) {
for (String rootPckg: rootPckg2Lib.keySet()) {
if (csr.getDeclaredTarget().getSignature().startsWith(rootPckg) && !WalaUtils.simpleName(im.getDeclaringClass()).startsWith(rootPckg)) {
//TODO how to store? extend DB? flag whether usage was found in app dev code (based on (fractions of) package name) or other lib code? we could also classify by using matched lib packages
// TODO: add additional heuristic to check for app package name levels
boolean usedInApp = !rootPckg2Lib.keySet().contains(WalaUtils.simpleName(im.getDeclaringClass()));
// check access specifier for target method
IClass targetClazz = cha.lookupClass(csr.getDeclaredTarget().getDeclaringClass());
if (targetClazz == null) { // no lookup possible - dead / legacy code?
logger.debug(Utils.INDENT + "Unresolvable class for lib method in use: " + csr.getDeclaredTarget().getSignature());
if (!unresolvableMethods.containsKey(rootPckg2Lib.get(rootPckg)))
unresolvableMethods.put(rootPckg2Lib.get(rootPckg), new TreeSet<String>());
unresolvableMethods.get(rootPckg2Lib.get(rootPckg)).add(csr.getDeclaredTarget().getSignature());
break;
}
IMethod targetMethod = targetClazz.getMethod(csr.getDeclaredTarget().getSelector());
if (targetMethod == null) { // e.g. if clazz is interface without declared methods
targetMethod = WalaUtils.resolveMethod(clazz, csr);
}
int accessSpecifier = AccessFlags.getMethodAccessCode(targetMethod);
String accessSpec = AccessFlags.flags2Str(accessSpecifier);
logger.trace(Utils.INDENT + "- method in use (in " + (usedInApp? "app" : "lib") +"): " + csr.getDeclaredTarget().getSignature() + " in bm: "+ im.getSignature() + " access: " + accessSpec);
String normalizedSig = csr.getDeclaredTarget().getSignature();
// normalize signature if lib root package does not match app lib root package (e.g. due to id renaming)
if (!rootPckg.equals(lib2Profile.get(rootPckg2Lib.get(rootPckg)).lib.packageTree.getRootPackage())) {
// replace package name in dot notation
String r = rootPckg.replaceAll("\\.", "\\\\.");
String rx = lib2Profile.get(rootPckg2Lib.get(rootPckg)).lib.packageTree.getRootPackage().replaceAll("\\.", "\\\\.");
// replace arguments in / notation
String r2 = rootPckg.replaceAll("\\.", "/");
String rx2 = lib2Profile.get(rootPckg2Lib.get(rootPckg)).lib.packageTree.getRootPackage().replaceAll("\\.", "/");
normalizedSig = csr.getDeclaredTarget().getSignature().replaceAll(r,rx).replaceAll(r2,rx2);
}
// update usedMethods
if (!usedMethods.containsKey(rootPckg2Lib.get(rootPckg)))
usedMethods.put(rootPckg2Lib.get(rootPckg), new TreeMap<String, Integer>());
usedMethods.get(rootPckg2Lib.get(rootPckg)).put(normalizedSig, accessSpecifier);
}
}
}
} catch (InvalidClassFileException e) {
logger.error(Utils.stacktrace2Str(e));
}
}
}
// debug output
for (String lib: usedMethods.keySet()) {
ProfileMatch pm = lib2Profile.get(lib);
String libRootPckg = pm.lib.packageTree.getRootPackage();
// TODO: currently unresolvable methods and access spec are not stored!
// store used methods
pm.usedLibMethods = new TreeSet<String>(usedMethods.get(lib).keySet());
logger.info("- check lib: " + pm.lib.getLibIdentifier() + " root package: " + libRootPckg + (!libRootPckg.equals(pm.getMatchedPackageTree().getRootPackage())? " vs matched root package: " + pm.getMatchedPackageTree().getRootPackage() : ""));
// retrieve number of unique lib classes used
Set<String> uniqueClazzes = new HashSet<String>();
for (String sig: usedMethods.get(lib).keySet()) {
uniqueClazzes.add(Utils.getFullClassName(sig));
}
logger.info(" (found " + usedMethods.get(lib).size() + " unique library methods and " + uniqueClazzes.size() + " unique lib clazzes");
for (String sig: usedMethods.get(lib).keySet()) {
logger.debug(Utils.INDENT + "- method in use: " + sig + " " + AccessFlags.flags2Str(usedMethods.get(lib).get(sig)));
}
// This can happen if a library contains multiple sub-libraries like OrmLite-[android,core,jdbc] and only a subset is profiled.
if (unresolvableMethods.containsKey(lib)) {
logger.info("");
for (String sig: unresolvableMethods.get(lib))
logger.debug(Utils.INDENT + "!! unresolvable library method in use by app: " + sig);
}
logger.info("");
}
logger.info(Utils.INDENT + ">> lib code usage analysis done (" + Utils.millisecondsToFormattedTime(System.currentTimeMillis() - starttime) + ")");
}
}
This diff is collapsed.
/*
* Copyright (c) 2015-2016 Erik Derr [derr@cs.uni-saarland.de]
* Copyright (c) 2015-2017 Erik Derr [derr@cs.uni-saarland.de]
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at
......@@ -20,6 +20,8 @@ import java.text.ParseException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
import java.util.jar.JarFile;
import javax.xml.parsers.ParserConfigurationException;
......@@ -37,79 +39,46 @@ import com.ibm.wala.ipa.cha.ClassHierarchyException;
import com.ibm.wala.ipa.cha.IClassHierarchy;
import com.ibm.wala.types.ClassLoaderReference;
import de.infsec.tpl.utils.AndroidClassType;
import de.infsec.tpl.TplCLI.CliOptions;
import de.infsec.tpl.TplCLI.OpMode;
import de.infsec.tpl.hash.HashTree;
import de.infsec.tpl.pkg.PackageTree;
import de.infsec.tpl.profile.LibProfile;
import de.infsec.tpl.profile.LibraryDescription;
import de.infsec.tpl.profile.Profile;
import de.infsec.tpl.utils.AarFile;
import de.infsec.tpl.utils.AndroidClassType;
import de.infsec.tpl.utils.Utils;
import de.infsec.tpl.utils.WalaUtils;
import de.infsec.tpl.xml.XMLParser;
/**
* @author Erik Derr
*/
public class LibraryHandler implements Runnable {
private static final Logger logger = LoggerFactory.getLogger(de.infsec.tpl.LibraryHandler.class);
private File targetFile; // target file either library.jar or app.apk, depending on the mode
private File libDescriptionFile; // xml file with basic facts about the library (only in non-matching mode)
private LibraryDescription libDesc;
private IClassHierarchy cha;
private List<LibProfile> libProfiles;
public class LibraryProfiler {
private static final Logger logger = LoggerFactory.getLogger(de.infsec.tpl.LibraryProfiler.class);
public LibraryHandler(File targetFile, File libDescriptionFile, List<LibProfile> profiles) {
this.targetFile = targetFile;
this.libDescriptionFile = libDescriptionFile;
this.libProfiles = profiles;
}
public static String FILE_EXT_LIB_PROFILE = "lib";
@Override
public void run() {
try {
if (CliOptions.opmode.equals(OpMode.MATCH)) {
init(false);
new LibraryIdentifier(targetFile).identifyLibraries(libProfiles);
} else if (CliOptions.opmode.equals(OpMode.PROFILE)) {
init(true);
extractFingerPrints();
}
} catch (Throwable t) {
logger.error("[FATAL " + (t instanceof Exception? "EXCEPTION" : "ERROR") + "] analysis aborted: " + t.getMessage());
logger.error(Utils.stacktrace2Str(t));
}
}
private File libraryFile; // library.jar || library.aar
private LibraryDescription libDesc; // library description parsed from an XML file
public void init(boolean readLibXML) throws ParserConfigurationException, SAXException, IOException, ParseException {
// read library description
if (readLibXML)
libDesc = XMLParser.readLibraryXML(libDescriptionFile);
public LibraryProfiler(File libraryFile, File libDescriptionFile) throws ParserConfigurationException, SAXException, IOException, ParseException {
this.libraryFile = libraryFile;
// read library description
libDesc = XMLParser.readLibraryXML(libDescriptionFile);
// set identifier for logging
String logIdentifier = CliOptions.logDir.getAbsolutePath() + File.separator;
if (readLibXML) {
logIdentifier += libDesc.name.replaceAll(" ", "-") + "_" + libDesc.version;
} else {
logIdentifier += targetFile.getName().replaceAll("\\.jar", "").replaceAll("\\.apk", "").replaceAll("\\.aar", "");
}
logIdentifier += libDesc.name.replaceAll(" ", "-") + "_" + libDesc.version;
// set identifier for log
MDC.put("appPath", logIdentifier);
}
public void extractFingerPrints() throws IOException, ClassHierarchyException {
public void extractFingerPrints() throws IOException, ClassHierarchyException, ClassNotFoundException {
long starttime = System.currentTimeMillis();
logger.info("Process library: " + targetFile.getName());
logger.info("Process library: " + libraryFile.getName());
logger.info("Library description:");
for (String desc: libDesc.getDescription())
logger.info(desc);
......@@ -117,12 +86,20 @@ public class LibraryHandler implements Runnable {
// create analysis scope and generate class hierarchy
final AnalysisScope scope = AnalysisScope.createJavaAnalysisScope();
scope.addToScope(ClassLoaderReference.Application, new JarFile(targetFile));
JarFile jf = libraryFile.getName().endsWith(".aar")? new AarFile(libraryFile).getJarFile() : new JarFile(libraryFile);
scope.addToScope(ClassLoaderReference.Application, jf);
scope.addToScope(ClassLoaderReference.Primordial, new JarFile(CliOptions.pathToAndroidJar));
cha = ClassHierarchy.make(scope);
IClassHierarchy cha = ClassHierarchy.make(scope);
getChaStats(cha);
// cleanup tmp files if library input was an .aar file
if (libraryFile.getName().endsWith(".aar")) {
File tmpJar = new File(jf.getName());
tmpJar.delete();
logger.debug(Utils.indent() + "tmp jar-file deleted at " + tmpJar.getName());
}
PackageTree pTree = Profile.generatePackageTree(cha);
if (pTree.getRootPackage() == null) {
logger.warn(Utils.INDENT + "Library contains multiple root packages");
......@@ -140,20 +117,23 @@ public class LibraryHandler implements Runnable {
logger.info("");
File targetDir = new File(CliOptions.profilesDir + File.separator + libDesc.category.toString());
logger.info("Serialize library fingerprint to disk (dir: " + targetDir + ")");
String proFileName = libDesc.name.replaceAll(" ", "-") + "_" + libDesc.version + ".lib";
Utils.object2Disk(new File(targetDir + File.separator + proFileName), new LibProfile(libDesc, pTree, hTrees));
String proFileName = libDesc.name.replaceAll(" ", "-") + "_" + libDesc.version + "." + FILE_EXT_LIB_PROFILE;
LibProfile lp = new LibProfile(libDesc, pTree, hTrees);
Utils.object2Disk(new File(targetDir + File.separator + proFileName), lp);
logger.info("");
logger.info("Processing time: " + Utils.millisecondsToFormattedTime(System.currentTimeMillis() - starttime));
}
public static void getChaStats(IClassHierarchy cha) {
public static Set<String> getChaStats(IClassHierarchy cha) {
TreeSet<String> publicMethods = new TreeSet<String>();
int clCount = 0;
int innerClCount = 0;
int publicMethodCount = 0;
int publicClCount = 0;
int miscMethodCount = 0;
HashMap<de.infsec.tpl.utils.AndroidClassType, Integer> clazzTypes = new HashMap<AndroidClassType, Integer>();
for (AndroidClassType t: AndroidClassType.values())
clazzTypes.put(t, 0);
......@@ -169,12 +149,13 @@ public class LibraryHandler implements Runnable {
clCount++;
if (WalaUtils.isInnerClass(clazz)) innerClCount++;
if (clazz.isPublic()) publicClCount++;
for (IMethod im: clazz.getDeclaredMethods()) {
if (im.isBridge() || im.isMethodSynthetic()) continue;
if (im.isPublic()) {
publicMethodCount++;
publicMethods.add(im.getSignature());
} else {
miscMethodCount++;
}
......@@ -186,14 +167,16 @@ public class LibraryHandler implements Runnable {
logger.info("= ClassHierarchy Stats =");
logger.info(Utils.INDENT + "# of classes: " + clCount);
logger.info(Utils.INDENT + "# thereof inner classes: " + innerClCount);
logger.info(Utils.INDENT + "# thereof public classes: " + publicClCount);
for (AndroidClassType t: AndroidClassType.values())
logger.info(Utils.INDENT2 + t + " : " + clazzTypes.get(t));
logger.info(Utils.INDENT + "# methods: " + (publicMethodCount + miscMethodCount));
logger.info(Utils.INDENT2 + "# of public methods: " + publicMethodCount);
logger.info(Utils.INDENT2 + "# of non-public methods: " + miscMethodCount);
logger.info(Utils.INDENT + "# methods: " + (publicMethods.size() + miscMethodCount));
logger.info(Utils.INDENT2 + "# of publicly accessible methods: " + publicMethods.size());
logger.info(Utils.INDENT2 + "# of non-accessible methods: " + miscMethodCount);
logger.info("");
return publicMethods;
}
}
This diff is collapsed.
/*
* Copyright (c) 2015-2017 Erik Derr [derr@cs.uni-saarland.de]
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
* file except in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under
* the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package de.infsec.tpl.eval;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
/**
* Container class to store library API changes across versions
* Includes information such as
* - # public APIs per version
* - APIs -> set of versions in which they are included
* - set of alternative APIs (if any), if API is no longer in some version
*/
public class LibApiRobustnessStats implements Serializable {
private static final long serialVersionUID = -3951094682839938234L;
public String libName;
public Map<String, Integer> versions2pubApiCount; // sorted by version
// maps public API signatures to list of library versions including them
public Map<String, List<String>> api2Versions;
// if API is not stable for the entire version set, include a list of candidate APIs of the first non-matching version
public Map<String, Set<String>> api2CandidateApis;
public LibApiRobustnessStats(String libName) {
this.libName = libName;
this.versions2pubApiCount = new LinkedHashMap<String, Integer>();
this.api2Versions = new HashMap<String, List<String>>();
this.api2CandidateApis = new HashMap<String, Set<String>>();
}
/*
* Update functions
*/
public void updateApi(String signature, String version) {
if (!api2Versions.containsKey(signature))