Open Source Ecosystems

jar2xml is a tool to extract API information from a java archive and associated documentation. The information is stored in XML format.

Prerequisites
- mono (to run build tools)
- xmllint (for doc scraping)
- xmlstarlet (ditto)
Also it uses ASM (included in the sources)
Tools
- jar2xml : the main converter to generate the primary target xmls.
- scraper.exe : annotation doc scraper to generate annotations/.xml.
  scraper-.sh : shells used for scraper input. They are piped.
Easy comparison

There are two levels of comparisons.

DLL: gui-compare in mono-tools is your friend. Compare new one against old assembly of the corresponding API profile.

XML: It is now implemented as "make test-8" as well as "make test-13" See Makefile for details. Things are not really simple to explain shortly.

$ java -jar jar2xml.jar --jar=/home/atsushi/android-sdk-linux_x86/platforms/android-8/android.jar --out=8.xml --docpath=/home/atsushi/android-sdk-linux_x86/docs/reference $ mono-xmltool --prettyprint 8.xml > 8.xml $ xmlstarlet c14n 8_.xml > 8.xml $ mono-xmltool --prettyprint _8.xml > 8.xml $ xmlstarlet c14n 8.xml > _8.xml $ diff -u 8.xml _8.xml
Difference from AOSP API XML

Type parameters on Class and Method are now explicitly written as:

see e.g. android.accounts.AccountManagerCallback

There are some cases that we dare output type parameters embedded into type name (basically those in XML attributes):
- genericConstraint/@type (see above)
- exception/@type (it would hardly happen though)
These are temporarily disabled and will be back when we really move to the new generic design.
- class/@extends
- implements/@name
- field/@type

TODO

** Postponed

java.security.Provider.put() : missing in jar2xml output. Disappeared in the shade of Hashtable.put(). -> workarounded by explicitly excluding the case (hardcoded in jar2xml!) This issue should rather get fixed by improving overrload resolution in XML level i.e. change XML format to include any overloaded methods that AOSP XML does not do.
java.net.ssl.KeyStoreBuilderParameters.Parameters and .ctor() : types differ
- generic list
Now there are some abstract methods that somehow lacked in AOSP XML and it uncovered issues regarding overriden methods. While those overriden methods are sometimes "final", which should be "sealed", they are not marked as sealed. And we cannot mark them as sealed right now since those methods are overriden in *Invoker classes. e.g. java.nio.Buffer.arrayOffset() and those overriden methods in *Buffer.
android.hardware.SensorManager.GRAVITY_DEATH_STAR_I always returns 0
- while reference and AOSP indicates value, it returns 0 on reflection,
  asm or even on ASUS Transformer.
android.test.MoreAsserts.assertEquals(Set). They are just XML difference between <?> (wrong) and <? extends Object> (correct), which affected doc scraping for parameter name retrieval.
java.util.Calendar AllStyles, Long and Short are now properties, but they should be actually enums. The thing is, they are missing in AOSP XML so they had weren't converted to enums by our generator stuff.
android.util.Pair : ThresholdClass and ThreasholdType are extraneous. They are not in jar2xml output either, maybe added by api fixup stage. (First and Second are looking as extra, but they were rather missing)

** things changes are maybe preferred

*** int->enum

Android.App.DownloadManager
Android.Appwidget.AppWidgetManager
Android.Bluetooth.BluetoothA2dp
Android.Bluetooth.BluetoothHeadset
Android.Content.PM.PackageManager
Android.Content.Res.Configuration
Android.Database.Sqlite.SQLiteDatabase
Android.Graphics.PowerDuff.Mode
Android.Media.AudioManager.ScoAudioState*
Android.OS.BatteryManager
Android.Views.InputDevice
Android.Views.View.Measured*
Android.Views.Window.Progress*
Android.Views.Window.WindowManagerLayoutParams
Android.Views.View.SystemUiVisibilityChangeEventArgs has int Visibility
which should be StatusBarVisibility.
Android.Widget.OnScrollListenerConsts

*** methods->properties

Android.App.Admin.DevicePolicyManager: there are lots of GetPassword*() and
SetPassword*() methods which we might be able to change to indexers.

Notes (that you don't have to read)

** API difference between android.jar and J2SE

These API differences could be problematic when we reflect java system types (reflection limitation).

javax.net.ssl.ServerSocketFactory.getDefault() is synchronized in android
and not in J2SE.
-> in jar2xml, it is manually fixed.

** AOSP XML issues

Lots of missing pieces: http://code.google.com/p/android/issues/detail?id=19569

** Notes on Annotations

Annotation such as java.lang.Deprecated do not seem to exist in android.jar. That includes, types themselves. java.lang.annotation types suffer from this.

I find it almost desparate and the only solution I couold think of was to build AOSP android.jar or some equivalence that does not strip those annotations off, for each platform version. That sounds so time-eater.

The AOSP API document is generated as out/target/common/obj/PACKAGING/android_jar_intermediaries/public_api.xml (at that state there still must be annotations, or this xml should not contain annotation information at all). Also, out/target/common/obj/JAVA_LIBRARIES/framework_intermediaries and out/target/common/obj/JAVA_LIBRARIES/core_intermediaries contains debug jars that still seem to hold annotations.

development/tools/apkcheck/README.txt might be of interest too.

This is from the latest AOSP build (as of Aug.17, 2011). I'm not sure if this applies to older versions.

On the other hand, I noticed that Annotation types are very easy to fill; they have almost fixed members (from java.lang.Object). So I rather think they can be manually filled (or if we want to be puritan, scrape "Known Indirect Subclasses" from java/lang/annotation/Annotation.html).

Annotations are: android.test.FlakyTest android.test.UiThreadTest android.test.suitebuilder.annotation.LargeTest android.test.suitebuilder.annotation.MediumTest android.test.suitebuilder.annotation.SmallTest android.test.suitebuilder.annotation.Smoke android.test.suitebuilder.annotation.Suppress android.view.ViewDebug$CapturedViewProperty android.view.ViewDebug$ExportedProperty android.view.ViewDebug$FlagToString android.view.ViewDebug$IntToString android.widget.RemoteViews$RemoteView dalvik.annotation.TestTarget dalvik.annotation.TestTargetClass java.lang.Deprecated java.lang.Override java.lang.SuppressWarnings java.lang.annotation.Documented java.lang.annotation.Inherited java.lang.annotation.Retention java.lang.annotation.Target java.lang.annotation.Documented

** Old Notes on Annotations

Current implementation uses java.lang.reflect API to get member annotations, but it won't work as expected.

I found that .class files in android.jar are stripped off field annotations. For example, java.net.HttpURLConnection#HTTP_SERVER_ERROR should be annotated with @Deprecated, but it isn't. On the other hand, class annotations are alive. (e.g. java/security/Certificate.class) I guess is that AOSP uses proguard which strips member annotations.

The reason why we can somehow retrieve @Deprecated on HttpURLConnection#HTTP_SERVER_ERROR by jar2xml is likely that URLClassLoader.getClass() actually does not load java.* classes from the jar contents but just returns the runtime one (such as openjdk).

Javassist.Loader javadoc explains its limitation on loading java.* / javax.* classes. [*1]

(So, it is not even about "runtime visible/invisible annotations". I tried to read android.jar classes using ASM ClassReader [*2] and it couldn't find any annotations on FieldNode (and from what I read from asm sources, I believe it is by no means bug in ASM).

I have added separate doc scraper, which uses xmllint (to xmlize html docs) and xmlstarlet (for convenient XPath query). Run "make annotations/13.xml" for instance (should be available to all versions, I'm just lazy to hack makefiles before confirming that it's all good).

[*1] http://www.csg.is.titech.ac.jp/~chiba/javassist/html/javassist/Loader.html [*2] http://asm.ow2.org/asm33/javadoc/user/org/objectweb/asm/ClassReader.html

** Notes on per-version doc scraping

We used to scrape only reference for the latest API which was problematic. Now we have version-aware scraper and resolved the issue described below.

Actually this is not going to be very good; there are some methods that are removed in the latest API (the example method above is exactly the case). We'll have to sort out some alternative approach. A possible approach is to checkout API docs for every version (from AOSP).

** Byte code engineering

Now we use ASM to solve the issues below:

java.io.File.pathSeparatorChar has the same problem, but it is worse; it returns non-zero value so we cannot depend on the value.

Solution ideas: scrape constant values, or use ASM.

Constant values for "protected" fields need to be retrieved, but Java reflection API throws IllegalAccessException for such attempt.

This likely has to be resolved by bytecode engineering (such as ASM).

** Issues regarding stripped non-public class (resolved)

Those methods (or those AbstractStringBuilder methods) are excluded by checking special modifier 0x1000 (4096). Also this involves check for derived methods (many overriden methods are excluded, but this brings another check to not exclude methods when the corresponding base methods are excluded).

** Method override resolution

	if (base_method != null) {
		int base_mods = base_method.getModifiers ();
		if (!Modifier.isAbstract (base_mods) && (Modifier.isPublic (mmods) == Modifier.isPublic (base_mods)))
			continue;
	}

This causes GridView.setAdapter() ignored and thus fails to generate valid property. It hides overriden methods.

I simply commented out, then a bunch of hidden overriden methods appeared. And since we don't handle int->enum conversions on those methods, they simply fail to compile. Maybe fixing above to match existing format is a good-enough compromization.

Another notable thing is that AbsListView.setAdapter() does not exist in api-4.xml. It is likely because abstract methods are ignored (it is abstract, though that fact does not appear on android javadoc).

android.view.MotionEvent.getSource() is also missing.