xamarin / jar2xml Goto Github PK

jar2xml is a tool to extract API information from a java archive
and associated documentation.  The information is stored in XML format.


* Prerequisites

	- mono (to run build tools)
	- xmllint (for doc scraping)
	- xmlstarlet (ditto)

	Also it uses ASM (included in the sources)

* Tools

	- jar2xml : the main converter to generate the primary target xmls.
	- scraper.exe : annotation doc scraper to generate annotations/*.xml.
	  scraper-*.sh : shells used for scraper input. They are piped.


* <del>Easy</del> comparison

  There are two levels of comparisons.

  DLL: gui-compare in mono-tools is your friend. Compare new one against
  old assembly of the corresponding API profile.

  XML: It is now implemented as "make test-8" as well as "make test-13" 
  See Makefile for details. Things are not really simple to explain shortly.

	$ java -jar jar2xml.jar --jar=/home/atsushi/android-sdk-linux_x86/platforms/android-8/android.jar --out=_8.xml --docpath=/home/atsushi/android-sdk-linux_x86/docs/reference
	$ mono-xmltool --prettyprint 8.xml > 8_.xml
	$ xmlstarlet c14n 8_.xml > 8.xml
	$ mono-xmltool --prettyprint _8.xml > _8_.xml
	$ xmlstarlet c14n _8_.xml > _8.xml
	$ diff -u 8.xml _8.xml


* Difference from AOSP API XML

- Type parameters on Class and Method are now explicitly written as:
  <typeParameters>    <!-- if any -->
    <typeParameter name="...">
      <genericConstraints>    <!-- if any -->
        <genericConstraint type="..."></genericConstraint>
      </genericConstraints>
    </typeParameter>
  </typeParameters>

  see e.g. android.accounts.AccountManagerCallback<V>

  There are some cases that we dare output type parameters embedded
  into type name (basically those in XML attributes):
  
  - genericConstraint/@type (see above)
  - exception/@type (it would hardly happen though)

  These are temporarily disabled and will be back when we really move
  to the new generic design.

  - class/@extends
  - implements/@name
  - field/@type


* TODO

** Postponed

- java.security.Provider.put() : missing in jar2xml output.
  Disappeared in the shade of Hashtable.put().
  -> workarounded by explicitly excluding the case (hardcoded in jar2xml!)
     This issue should rather get fixed by improving overrload resolution
     in XML level i.e. change XML format to include *any* overloaded methods
     that AOSP XML does not do.

- java.net.ssl.KeyStoreBuilderParameters.Parameters and .ctor() : types differ
  - generic list

- Now there are some abstract methods that somehow lacked in AOSP XML
  and it uncovered issues regarding overriden methods. While those overriden
  methods are sometimes "final", which should be "sealed", they are not
  marked as sealed. And we cannot mark them as sealed right now since
  those methods are *overriden* in *Invoker classes.
  e.g. java.nio.Buffer.arrayOffset() and those overriden methods in *Buffer.

- android.hardware.SensorManager.GRAVITY_DEATH_STAR_I always returns 0
  - while reference and AOSP indicates value, it returns 0 on reflection,
    asm or even on ASUS Transformer.

- android.test.MoreAsserts.assertEquals(Set<?>, Set<?>).
  They are just XML difference between <?> (wrong) and <? extends Object>
  (correct), which affected doc scraping for parameter name retrieval.

- java.util.Calendar AllStyles, Long and Short are now properties, but
  they should be actually enums. The thing is, they are *missing* in AOSP
  XML so they had weren't converted to enums by our generator stuff.

- android.util.Pair : ThresholdClass and ThreasholdType are extraneous.
  They are not in jar2xml output either, maybe added by api fixup stage.
  (First and Second are looking as extra, but they were rather missing)

** things changes are maybe preferred
 
*** int->enum

- Android.App.DownloadManager
- Android.Appwidget.AppWidgetManager
- Android.Bluetooth.BluetoothA2dp
- Android.Bluetooth.BluetoothHeadset
- Android.Content.PM.PackageManager
- Android.Content.Res.Configuration
- Android.Database.Sqlite.SQLiteDatabase
- Android.Graphics.PowerDuff.Mode
- Android.Media.AudioManager.ScoAudioState*
- Android.OS.BatteryManager
- Android.Views.InputDevice
- Android.Views.View.Measured*
- Android.Views.Window.Progress*
- Android.Views.Window.WindowManagerLayoutParams
- Android.Views.View.SystemUiVisibilityChangeEventArgs has int Visibility
  which should be StatusBarVisibility.
- Android.Widget.OnScrollListenerConsts

*** methods->properties

- Android.App.Admin.DevicePolicyManager: there are lots of GetPassword*() and
  SetPassword*() methods which we might be able to change to indexers.

* Notes (that you don't have to read)

** API difference between android.jar and J2SE

These API differences could be problematic when we reflect java system types (reflection limitation).

- javax.net.ssl.ServerSocketFactory.getDefault() is synchronized in android
  and not in J2SE.
  -> in jar2xml, it is manually fixed.

** AOSP XML issues

Lots of missing pieces:
http://code.google.com/p/android/issues/detail?id=19569

** Notes on Annotations

  Annotation such as java.lang.Deprecated do not seem to exist
  in android.jar. That includes, types themselves.
  java.lang.annotation types suffer from this.

  I find it almost desparate and the only solution I couold think of was
  to build AOSP android.jar or some equivalence that does *not* strip
  those annotations off, *for each platform version*.
  That sounds so time-eater.

  The AOSP API document is generated as out/target/common/obj/PACKAGING/android_jar_intermediaries/public_api.xml (at that state there still must be annotations, or this xml should not contain annotation information at all).
  Also,
  out/target/common/obj/JAVA_LIBRARIES/framework_intermediaries and
  out/target/common/obj/JAVA_LIBRARIES/core_intermediaries
  contains debug jars that still seem to hold annotations.

  development/tools/apkcheck/README.txt might be of interest too.

  This is from the latest AOSP build (as of Aug.17, 2011). I'm not sure
  if this applies to older versions.

  On the other hand, I noticed that Annotation types are very easy to fill;
  they have almost fixed members (from java.lang.Object). So I rather
  think they can be manually filled (or if we want to be puritan, scrape
  "Known Indirect Subclasses" from java/lang/annotation/Annotation.html).

  Annotations are:
	android.test.FlakyTest
	android.test.UiThreadTest
	android.test.suitebuilder.annotation.LargeTest
	android.test.suitebuilder.annotation.MediumTest
	android.test.suitebuilder.annotation.SmallTest
	android.test.suitebuilder.annotation.Smoke
	android.test.suitebuilder.annotation.Suppress
	android.view.ViewDebug$CapturedViewProperty
	android.view.ViewDebug$ExportedProperty
	android.view.ViewDebug$FlagToString
	android.view.ViewDebug$IntToString
	android.widget.RemoteViews$RemoteView
	dalvik.annotation.TestTarget
	dalvik.annotation.TestTargetClass
	java.lang.Deprecated
	java.lang.Override
	java.lang.SuppressWarnings
	java.lang.annotation.Documented
	java.lang.annotation.Inherited
	java.lang.annotation.Retention
	java.lang.annotation.Target
	java.lang.annotation.Documented


** Old Notes on Annotations

Current implementation uses java.lang.reflect API to get member
annotations, but it won't work as expected.

I found that .class files in android.jar are stripped off field
annotations. For example, java.net.HttpURLConnection#HTTP_SERVER_ERROR 
should be annotated with @Deprecated, but it isn't.
On the other hand, class annotations are alive.
(e.g. java/security/Certificate.class)
I guess is that AOSP uses proguard which strips member annotations.

The reason why we can somehow retrieve @Deprecated on
HttpURLConnection#HTTP_SERVER_ERROR by jar2xml is likely that
URLClassLoader.getClass() actually does not load java.* classes from
the jar contents but just returns the runtime one (such as openjdk).

Javassist.Loader javadoc explains its limitation on loading java.*
 / javax.* classes. [*1]

(So, it is not even about "runtime visible/invisible annotations".
I tried to read android.jar classes using ASM ClassReader [*2] and
it couldn't find any annotations on FieldNode (and from what I read
from asm sources, I believe it is by no means bug in ASM).

I have added separate doc scraper, which uses xmllint (to xmlize html docs)
and xmlstarlet (for convenient XPath query). Run "make annotations/13.xml" 
for instance (should be available to all versions, I'm just lazy to hack
makefiles before confirming that it's all good).


[*1] http://www.csg.is.titech.ac.jp/~chiba/javassist/html/javassist/Loader.html
[*2] http://asm.ow2.org/asm33/javadoc/user/org/objectweb/asm/ClassReader.html


** Notes on per-version doc scraping

We used to scrape only reference for the latest API which was problematic.
Now we have version-aware scraper and resolved the issue described below.

<historical>
  Doc scraper still misses some parameter names
  e.g. android.text.method.ScrollingMovementMethod.onKeyUp()
  This matters when api-fixup is involved (it matches nodes by parameter
  names and then it dismisses the target nodes!).

  Actually this is not going to be very good; there are some methods
  that are removed in the latest API (the example method above is exactly
  the case). We'll have to sort out some alternative approach.
  A possible approach is to checkout API docs for *every* version (from
  AOSP).
</historical>


** Byte code engineering

Now we use ASM to solve the issues below:

<historical>
- Some fields were incorrectly marked as constant. For example,
  android.os.Build.TIME is not a constant, but since the field.getLong()
  returns 0, it is set as 0 (and it does not return null for get()).

  java.io.File.pathSeparatorChar has the same problem, but it is worse;
  it returns non-zero value so we cannot depend on the value.

  Solution ideas: scrape constant values, or use ASM.

- Constant values for "protected" fields need to be retrieved, but Java reflection API throws IllegalAccessException for such attempt.

  This likely has to be resolved by bytecode engineering (such as ASM).
</historical>


** Issues regarding stripped non-public class (resolved)

<maybe-historical>
- java.lang.AbstractStringBuilder exposes an issue that
  interface members on non-public types are returned by
  class#getDeclaredMethods() while they are *not* declared.
  
  Those methods (or those AbstractStringBuilder methods) are excluded by
  checking special modifier 0x1000 (4096).
  Also this involves check for derived methods (many overriden methods are
  excluded, but this brings another check to *not* exclude methods when
  the corresponding base methods are excluded).
</maybe-historical>

** Method override resolution

<maybe-historical>
- Do not skip certain overriden methods. This in JavaClass.java gives
  equivalent output to AOSP xml, but it's not good:

		if (base_method != null) {
			int base_mods = base_method.getModifiers ();
			if (!Modifier.isAbstract (base_mods) && (Modifier.isPublic (mmods) == Modifier.isPublic (base_mods)))
				continue;
		}

  This causes GridView.setAdapter() ignored and thus fails to generate
  valid property. It hides overriden methods.

  I simply commented out, then a bunch of hidden overriden methods
  appeared. And since we don't handle int->enum conversions on those
  methods, they simply fail to compile. Maybe fixing above to match
  existing format is a good-enough compromization.

  Another notable thing is that AbsListView.setAdapter() does not
  exist in api-4.xml. It is likely because abstract methods are
  ignored (it is abstract, though that fact does not appear on
  android javadoc).

  android.view.MotionEvent.getSource() is also missing.
</maybe-historical>