Kotlin Upload

changed the description

Hi,

I tried building kotlin last night after making a couple of changes to kotlinx-metadata. The package tasks are building fine, but the :kotlin-compiler:proguard is failing!

Here is the failure log: https://paste.debian.net/1150532/

As suggested by ebourg there might be a possibility that we have a regression with the Guava library. "Guava uses java 11, while this version of kotlin supports java 8."

On researching I saw we can ignore the proguard warnings by using flags -dontwarn and -ignorewarnings.

Here is the patch for the same: https://paste.debian.net/1150533/

But it wasn't successful at all. Is there anything we can look for right now? Can't we use an old-specific version for Guava in Build-Depends?

The original description was:

Hi,

Kotlin Update: here

Blockers:

The kotlin source package is building fine but with the kotlin bootstrap. Checked for the kotlinc it was working fine ran a little HelloWorld program, it compiled and showed a positive result.

The next problem is to know about the *upload* plan (We still need to figure one).

Mailed on the java team about the issue with bootstrapping. The way ebourg suggested also failed, as after installing the kotlin binaries the kotlin wasn't able to build by its binaries. It failed with an error kotlin-gradle-plugin issue in offline mode.

I had a chat with m36 about this issue, he explained me the structure and why we can't package kotlinx as it is totally kotlin specific, later on, I realized it myself.

kotlin-gradle-plugin:1.3.30-eap-28 fails to generate jar files from the package itself. Mailing Threads: https://lists.debian.org/debian-java/2020/05/msg00046.html

I myself tried converting some of the kts files to groovy for the kotlin Gradle plugin but afaik there is no progress. There are some issue which was rectified on the build but kolin-gradle-plugin is not generating the jar for now.

Therefore it is really very important to discuss the upload plan before anything.

Today after struggling for a week and coming to square one I am opening this issue!

That's something very concerning as kotlin being the blocker of all!

All suggestions and help are appreciated :)

Thanks!

Hi,

The kotlin package is building fine once again. The proguard warning is resolved. Thanks to @ebourg :D He fixed libguava-java by making it compatible with Java 8 in (29.0-5).

Here's a doubt:

I just added jars for kotlin-gradle-plugin,kotlin-gradle-plugin-api and kotlin-gradle-plugin-model (from the kotlin-bootstrap) https://paste.debian.net/1150872/

It tried to find 1.3.30-eap-28 but once we install kotlin, the jar files are stored with 1.3.30

The debugged package exists here: https://salsa.debian.org/samyak-jn/debugging

mentioned in issue android-tools-team/admin#12

added GSoC in Android Tools / admin kotlin in Android Tools / admin labels

assigned to @samyak-jn

The pre-final stage is to build kotlin-gradle-plugin and kotlin-sam-with-receiver from the source package itself.

After digging deeper into it; I found two workarounds for it ()

Converting kotlin buildSrc files for embeddable.kt to make it compatible groovy closures. (kotlin-conversion branch)
Why not compile these jars from the kotlin-compiler instead of kotlin-compiler-embeddable? (Compiler-use branch)

On Testing both of the workarounds:

The first one compiles successfully for the plugin, but it ain't functional. It is not able to generate the runtime jars. Hence, this way isn't feasible.
For the second workaround for switching compiler-embeddable to the compiler, I changed some of the build.gradle function of required submodules. It took a while to make it free from compilation errors. But the saddest part is:

org.gradle.api.CircularReferenceException: Circular dependency between the following tasks:
:kotlin-gradle-plugin:classes
\--- :kotlin-gradle-plugin:compileJava
     +--- :kotlin-gradle-plugin:compileKotlin
     |    \--- :kotlin-gradle-plugin:jar
     |         +--- :kotlin-gradle-plugin:classes (*)
     |         +--- :kotlin-gradle-plugin:compileKotlin (*)
     |         \--- :kotlin-gradle-plugin:inspectClassesForKotlinIC
     |              \--- :kotlin-gradle-plugin:classes (*)
     \--- :kotlin-gradle-plugin:jar (*)

(*) - details omitted (listed previously)

I'm getting back to resolve this issue, and bring back some good news :/

Clarification from IRC andrewsh: kotlin-bootstrap will not be uploaded to the debian archives.

So assuming all the good self-build work pays off the final steps would be:

build locally in a clean chroot plus kotlin-bootstrap, do not upload the built .debs
build locally in a clean chroot plus the built .debs from step 1
do a binary upload to the NEW queue, wait until it is approved
do a source-only upload to the archive

I’d also add that the step 2 would be best done multiple times to verify things don’t regress randomly. That would also allow us to detect where the generated files change to make the builds completely reproducible.

moved from android-tools-team/admin#18 (moved)

unassigned @samyak-jn

As I understand, all outstanding blockers are now fixed. We need to look forward to uploading Kotlin (perhaps into experimental).

Can we already build Kotlin with itself?

Yes, I have tested as follows:

upstream binary 1.3.31 --(builds)--> 1.3.31.deb --(builds)--> 1.3.31.deb --(builds)--> 1.3.31.deb

Awesome, and it no longer depends on kotlin-bootstrap?

kotlin-bootstrap was a package prepared from upstream binaries, if I understand correctly. We are still using upstream binaries in stage1.

We are simplifying their use in this merge request. Summary of that MR is that we have a script to download a minimal set of up upstream binaries and generate kotlin.deb. This can then be used to build our Debian package during stage 1. After stage 1 when we have a fully built kotlin.deb we can use that in stage 2 instead of upstream binaries.

However, if we are using kotlin.deb built by others here, we don't need kotlin-bootstrap.deb any more.

kotlin-bootstrap was a package prepared from upstream binaries, if I understand correctly. We are still using upstream binaries in stage1.

Yes, the kotlin bootstrap package is built from upstream binaries itself, it contains all the jars files needed to build it.

yep, I intend to update the corresponding build instructions (#14 (closed)) to omit the kotlin-deps repo step entirely once !16 (merged) is out of draft. This also helps us jump versions, or at least reconfirm our bootstrapping process still works with any future version.

changed title from Kotlin Upload? to Kotlin Upload

changed the description

Added myself to list of uploaders.

changed the description

marked the checklist item build locally in a clean chroot plus kotlin.deb from debian/bootstrap, do not upload the built .debs as completed

marked the checklist item build locally in a clean chroot plus the previously built kotlin.deb (repeat N times to check for regression) as completed

marked the checklist item build locally in a clean chroot plus the previously built kotlin.deb (repeat N times to check for regression) as incomplete

marked the checklist item build locally in a clean chroot plus the previously built kotlin.deb (repeat N times to check for regression) as completed

I am currently reviewing the copyrights.

I've moved the other steps to below Upload, since only the copyright needs to be as correct as possible first, the others are optional.

https://www.emorrp1.name/kotlin/

For interest of where to look for reproducibility, I have uploaded a full diffoscope of consecutive builds, and these are the jars that change in stage 2 (increase in size, usually about 200 bytes):

jvm-minimal-for-test-1.3.31.jar
kotlin-compiler-1.3.31.jar
kotlin-compiler-embeddable-1.3.31.jar
kotlin-main-kts-1.3.31.jar
kotlin-reflect-1.3.31.jar
kotlin-reflect-api-1.3.31.jar
kotlin-stdlib-1.3.31.jar
kotlin-stdlib-minimal-for-test.jar

Perhaps we can aim for reproducibility from state 2 onwards (i.e. matching stage 2 and stage 3).

https://www.emorrp1.name/kotlin/

I hadn't built that one at the time, you are correct that diff_03 is actually massively more reproducible. There is no difference in the uncompressed file listing (same size jars). There's a handful of classes with bitwise differences and two ordering changes in coroutines: AwaitAll and ConflatedBroadcastChannel

backend-1.3.31.jar
frontend-1.3.31.jar
kotlin-compiler-1.3.31.jar
kotlin-compiler-embeddable-1.3.31.jar
kotlin-main-kts-1.3.31.jar

and just for fun, diff_04 is exactly two differences: ExpressionCodegen and FunctionCodegen

backend-1.3.31.jar
kotlin-compiler-1.3.31.jar

now I should probably get back to actual useful packaging rather than just rebuilding

Functions with suspend keywords are generated differently from regular functions as they can be stop during execution and later resumed. The return logic, continuation logic, lookup table and some local variables are inserted into the function for all this. Looks like the order of insertion of the variables is not stable perhaps due to the use of a hashmap or so. The culprit is likely to be in or around CoroutineTransformerMethodVisitor.kt. The nature of the problem also means that sometimes, it will lead to diffs while at other times, it will generate identical files. This explains why diff_03 is different from diff_04.
The diffoscope output says that the specialized diff tool for .class files could not produce any output. This hints us that the issue is likely superficial (related to .class fiole format). Also, again, in some cases the output value seems to be flip-flopping. So, differences between stage2 and stage4 are likely to be interesting.

Thank you so much for providing the diffs. It saved me a lot of effort even with working build setups and some builds.

changed the description

I have done a thorough review of the copyright file and did my best to update it. Pushed the change:

d/copyright: Several minor updates after review

Changes:

- Sort all the sections alphabetically.

- Add /* at the end of directory paths. The machine-readable copyright format
does not mention anything about specifying a directory path instead of a glob
pattern.

- Add all authors to copyright list for debian/*.

- Mention that js/js.ast has several files that are copyright JetBrains.

- When code is taken from another project and modified significantly, the
copyright belongs to the modifier too. Update copyright for such files.
  - pcollections
  - libraries/stdlib/**/collections
  - libraries/stdlib/unsigned/src/kotlin/UnsignedUtils.kt

- When code with a compatible license is taken and modified under the project's
license, both the license terms must be followed on the resulting code.
  - compiler/backend/src/org/jetbrains/kotlin/codegen/...
  - core/reflection.jvm/src/kotlin/reflect/jvm/internal/pcollections
  - libraries/stdlib/jvm/src/kotlin/util/MathJVM.kt
  - libraries/stdlib/js/src/js/polyfills.js

- Add more files to existing sections:
  - libraries/stdlib/js/irRuntime/longjs.kt
  - js/js.translator/qunit/qunit.css

- Fix outdated paths:
  - libraries/stdlib/js-v1 -> libraries/stdlib/js/
  - core/.../src/kotlin.reflect -> core/.../src/kotlin/reflect

- Fix a duplicate glob patterns and fix lintian warnings:
  - third-party/annotations/org/eclipse

- Add new copyright sections or add more entities:
  - .../webidl/*
  - .../kotlin-gradle-plugin-integration-tests/...
  - third-party/annotations/edu/umd/cs/findbugs/*
  - core/metatdata/src/google/protobuf/descriptor.proto
  - js/js.parser/src/com/google/gwt/dev/js/parserExceptions/parserExceptions.kt
  - js/js.parser/src/com/google/gwt/dev/js/JsAstMapper.java

Signed-off-by: Sunil Mohan Adapa <sunil@medhas.org>

Edit: I have done a thorough review.

Added README.source file in the latest commit. It talks of the build process, challenges and solutions.

marked the checklist item write debian/README.source as completed

changed the description

Marking debian/copyright as reviewed. If someone wants to go another round on it, unmark it.

marked the checklist item review debian/copyright as completed

changed the description

I have pushed several commits:

Add kotlin as build-dependency. Tested that works with our current bootstrap approach without any modifications.
Eliminated all lintian warnings and errors except for writing man pages.
Prefix all jars in /usr/share/java with kotlin- to make them unique across Debian.
Fixed handling for 3 jar files.

I also started clean up of patches and I am about 75% done.

changed the description

marked the checklist item review lintian output and debian/kotlin.lintian-overrides as completed

Marked the lintian output as reviewed after separating out the task to write manual pages.

For reference to anyone wishing to write the manpages, but no pressure as I'm not convinced how much of this detail is worth duplicating/importing:

Online reference for CLI https://kotlinlang.org/docs/compiler-reference.html but presumably that's generated from e.g. kotlinc -help. I did find two Usage texts for kotlin and kotlinp, plus "KotlinCompilerSettings" under idea/. Aha, found CommonCompilerArguments.kt with the description texts.

I pushed a complete refactor of the patches. After squashing, splitting and dropping the patch count is down from 89 to 59. For easy review of the final code changes, here is the code diff: debian_patches_cleanup.patch.

Pushed another change to remove debug prints in patches.

On the overall, I believe the patches are in reasonable shape to be understood/used for building with newer Gradle when it becomes available. Further clean ups to the patches are more fruitful when doing that than now. So, I am marking the cleanu patches are done. Please unmark if you have more ideas on how to cleanup further now.

Wow, that was unexpected, but extremely useful. By cleanup patches I did actually mean the latter i.e. dropping all the build.gradle{.kts,} debian-specific rewrites Saif and Samyak did during their GSoCs (once gradle v6 is available). As you say, anything that doesn't happen to get done for the initial upload I'll make sure gets split out into separate issues.

marked the checklist item cleanup debian/patches as completed

I shall now wait for openjdk-8 and bootstrap upload before further work on the issue. Also perhaps we should create separate issues for the remaining non-upload-related tasks.

Also, who is doing the honours of uploading the package once openjdk-8 is available? :) Any volunteers?

Please wait with uploading kotlin until 8u292 is both uploaded and built by the buildds and in status Installed, so it’s ready as B-D for further builds.

I pushed two sets of changes:

Build speed-ups: Remove disabling of parallel builds and disable use of kotlin compiler daemon. This speeds up builds significantly (now about 14 minutes).
Build with maven: Kotlin is meant to be compiled with maven for some sub-projects after the build with Gradle. Do this to have two additional jars built: kotlin-maven-plugin and kotlin-osgi-bundle.

@emorrp1 spotted a bug with build in stage2 with the maven changes. I pushed a fixed for it.

changed the description

marked the checklist item wait until openjdk-8 reentry as completed

changed the description

Kotlin package is ready for upload as all the immediate and important review comments have been addressed. Here is compilation of review comments given on the Kotlin package and status of work on them (compiled at https://pad.riseup.net/p/5PS4jzZ48jpmCLJshpDw).

Review Comments for Kotlin upload

Kotlin package must be uploaded after openjdk-8 >= 8u292-b10-1 is available in unstable.
- DONE: required openjdk-8 is now available in Debian
Run lintian with --pedantic
- DOING: https://www.emorrp1.name/kotlin/kotlin_1.3.31%2B~~1.0.1%2B~~0.11.12-1_amd64-2021-05-07T16:09:51Z.build (~/.sbuildrc: $lintian_opts = ['--display-level', '>=pedantic'];)
- package-does-not-install-examples
- source-contains-prebuilt-java-object
  - POSTPONED: When we import the source tarball again, we can take care of this.
- spelling-error-in-copyright
Write manual pages for command line commands
- POSTPONED
Simplify versioning of the package. Atleast remove the + sign.
- POSTPONED: It is best to maintain the components within the Kotlin package. The two components are tightly tied with Kotlin and are not very relavant outside. Kotlin also needs strict version of the components. Since the versions of the components required are not the latest, downloading the required version is easier with uscan if the version contains the component versions. When we move to the latest versions of these components, we can drop the long version number and simply use the Kotlin version or use the checksum scheme supported by uscan.
debian/bootstrap creates a deb, which has the deb itself in the root directory
- DONE: Fixed. .deb is generated in ../ directory now
debian/rules has unnecessary exports like "export JDK_16="/usr/lib/jvm/java-8-openjdk-amd64"
- DONE: Unnecessary exports have been removed.
After installing the bootstrap deb trying the build fails
- DONE: Fixed and tested
Apparently the package doesn't clean properly. Try building with dpkg-buildpackage -b twice.
- DONE: Fixed and tested by running 'debuild -us -uc' twice after 'gbp'
The deb has no dependencies. Symlinks in /usr/share/kotlin/kotlinc/lib/ are broken and kotlinc fails with a FileNotFoundException.
- DONE: Extract the list of dependencies from POMs and added Debian packages that provide those. Tested by performing a sample compilation.
Is there any reason to use debhelper 13? This makes backporting harder.
- mirabilos: That’s Ubuntu you’re talking about. I’m still totally annoyed for my PPA builds that bionic and focal don’t have debhelper 13 available. Debian backports, at least for the last 3+ years, have had reasonably up-to-date debhelper, as downgrading the version just for backporting is really undesired as backports change and hindering things in sid. 13~bpo10+1 has a changelog date of 27 Apr 2020.
- Phil Morrell: Are you planning to upload to stretch or ubuntu 18.04? 13 is in buster-backports and I believe these changes are relevant for java: v13 This is the recommended mode of operation.
  - The third-party gradle build system (from gradle-debian-helper package) now runs the upstream-provided test suite automatically.
  - The dh_missing command will now default to --fail-missing.
  - The dh command sequencer will now skip all hook and override targets for dh_auto_test when DEB_BUILD_OPTIONS lists the relevant nocheck / nostrip options.
lintian reports unused paragraph in debian/copyright
- POSTPONED: Many of these are relavant for tests data. Currently test data is stripped from input tarball and tests are not run. Since the effort to check licenses on them is non-trivial, keeping the unused paragraphs might be better.
The are also two (unused?) android-support-v4.jar files that could be removed from the source package.
- POSTPONED: When we import the source tarball again, we can take care of this.
Did you start working on packaging kotlinx coroutines and atomicfu separately yet? That will be the next step once kotlin is uploaded.
- POSTPONED: We think it might best to continue to keep the MUT format since the two packages are not very relavant to other packages.
- Phil Morrell: The reasoning here is that we can keep the bootstrap problem isolated to the gradle and kotlin source packages. By splitting out the kotlin build-deps they would have to be included in the dependency loop.
Also, are all the jar files downloaded by debian/bootstrap rebuilt? None is copied directly into the stage 2 .deb, right?
- DONE: This has been verified
A backport fails to build. So it looks like some of the build dependencies need an update, or at least a required version. Can you see which ones?
- POSTPONED
aim for reproducibility
- POSTPONED
enable salsa CI
- POSTPONED: requires at least the first binary upload

marked the checklist item add yourself to Uploaders if you want as completed

Kotlin Upload

Designs

Child items ...

Activity

Review Comments for Kotlin upload