Skip to content

Instantly share code, notes, and snippets.

@srl295

srl295/unicodetools-jsp-notes.md

Last active July 12, 2021 19:19
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save srl295/a788b4de315cd919bb5f0f4e3b2f69c7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save srl295/a788b4de315cd919bb5f0f4e3b2f69c7 to your computer and use it in GitHub Desktop.
Download ZIP
Steven's Notes on updating UnicodeJsp with a new Unicode version
Raw
unicodetools-jsp-notes.md

git # Building UnicodeJsp

Compiling

Prerequisites

  • Java
  • Maven (download instructions here
  • GitHub account
  • UnicodeJsp source downloaded

Maven Environment setup

Setup your maven environment’s .m2/settings.xml as per CLDR instructions. You don’t need to actually need to download or build CLDR itself.

Building from the command line

Run mvn package in the UnicodeJsps directory to build the JSPs.

Building from Eclipse

UnicodeJsps can be imported into eclipse using the Import->existing maven project menu option. If you already have UnicodeJsps in eclipse, it might be better to remove it from eclipse before the import.

Running

Command Line

mvn jetty:run

You can now connect to http://127.0.0.1:8080 as suggested from the command line. Use Control-C to stop the server.

Running from within Eclipse

This screenshot shows creating a jetty:run maven run configuration.

Running jetty:run from eclipse

Debugging from within eclipse

Jetty provides instructions here for debugging within eclipse. It is unknown whether this has been attempted with the UnicodeJsps.

Upgrading the UnicodeJsps

⚠️SRL: Steven Notes Here

Text Files

Into {$workspace}/UnicodeJsps/src/main/resources/org/unicode/jsp

Copy in newest versions of:

{$workspace}/unicodetools/data/security/\<VERSION\>

  • confusables.txt
  • IdentifierStatus.txt
  • IdentifierType.txt

{$workspace}/unicodetools/data/ucd/\<VERSION\>-Update/

  • NameAliases.txt
  • NamesList.txt
  • ScriptExtensions.txt
  • StandardizedVariants.txt

{$workspace}/unicodetools/data/idna/\<VERSION\>

  • IdnaMappingTable.txt

{$workspace}/unicodetools/data/emoji/\<VERSION\>

  • emoji-sequences.txt
  • emoji-zwj-sequences.txt
  • <emoji-variants> 🚧 TODO

Run org.unicode.text.tools.GenerateSubtagNames to generate results on the console; paste the results into subtagNames.txt

⚠️SRL: This should be done so that subtagNames.txt is updated automatically…

Other files:

  • alpha2_3.txt — mapping from 3 letter to 2 letter. No change needed.
  • fixCodes.txt — dump of language/territory/region alias from CLDR. Fix to use ICU or CLDR directly instead.

Files to investigate

  • annotations.txt - hieroglyphs
  • Categories.txt
  • Transforms?
    • Deva-IPA.txt
    • en-IPA.txt
  • Globe.txt
  • nameprep.txt
  • nfm.txt — not sure what this is; investigate and use ICU instead.
  • tables.txt
  • temp.html
  • test.htm

Adding/Updating New Properties

Go to XPropertyFactory.java to add new properties other than the ones in /props/

⚠️SRL: Not sure what the work is here. What other properties do I expect?

Adding "/Prop/" Properties

UnicodeJsps/src/main/resources/org/unicode/jsp/data

Update the following files by copying from org/unicode/props

  • ExtraPropertyAliases.txt
  • ExtraPropertyValueAliases.txt

Update the following files by copying from unicodetools/data/ucd/XX.0.0-Update

  • PropertyAliases.txt
  • PropertyValueAliases.txt

UnicodeJsps/src/main/resources/org/unicode/jsp/props/

Run ListProps. It will copy .bin files into {generated}bin/XX.0.0.0. Examples:

  • Age.bin
  • Alphabetic.bin
  • ASCII_Hex_Digit.bin
  • Bidi_Class.bin
  • ...

⚠️SRL: Confusing as to what the prerequisites for this are. Copy from where? ListProps failed with a lot of complaints where Generated/bin-props files weren't found. I don't see bin-props mentioned anywhere else in the project. So, I symlinked Generated/BIN to Generated/bin-props which allowed ListProps to proceed.

Copy those into /UnicodeJsps/src/main/resources/org/unicode/jsp/props using CopyPropsToUnicodeJsp

Using Beta Properties

Set CachedProps.IS_BETA to true.

⚠️SRL: In the source?

Build & Test

  1. Run mvn test

🚧 TODO: These need a lot of work; they mostly print out a lot of gorp that you need to scan over.

Run the server (see above)

Look at http://localhost:8080/UnicodeJsps/properties.jsp, and make sure that there aren't any Z-Other props at the bottom (you'll need to update via Adding New Properties if there are).

⚠️SRL: failed to startup,

Caused by: com.ibm.icu.impl.IllegalIcuArgumentException: Invalid name: Ougr
    at com.ibm.icu.lang.UCharacter.getPropertyValueEnum (UCharacter.java:5176)
    at org.unicode.jsp.ScriptTester$ScriptExtensions$MyHandler.handleLine (ScriptTester.java:282)
    at org.unicode.jsp.FileUtilities$SemiFileReader.process (FileUtilities.java:97)
    at org.unicode.jsp.FileUtilities$SemiFileReader.process (FileUtilities.java:45)
    at org.unicode.jsp.ScriptTester$ScriptExtensions.make (ScriptTester.java:300)
    at org.unicode.jsp.ScriptTester.<clinit> (ScriptTester.java:325)
    at org.unicode.jsp.XPropertyFactory.<init> (XPropertyFactory.java:147)
    at org.unicode.jsp.XPropertyFactory.make (XPropertyFactory.java:39)
    at org.unicode.jsp.UnicodeUtilities.<clinit> (UnicodeUtilities.java:247)
    at org.unicode.jsp.UnicodeJsp.showPropsTable (UnicodeJsp.java:168)
    at org.apache.jsp.properties_jsp._jspService (properties_jsp.java:208)

(:construction: TODO: explain how to do a Docker-based build here.)

Commit/PR

  1. Commit the code to your own branch, create a PR on GitHub
  2. Verify that all of the checks (build steps) succeeded, for example https://github.com/unicode-org/unicodetools/pull/22/checks in particular "build JSP.". Request reviews
  3. Merge PR into master when reviewed.

Make a release / Deploy

When it is time to push a new version of the JSPs and tools, it is time to make a "release". A release is a snapshot of the entire repository, including all Unicode Tools.

  1. Go to the UnicodeTools repository release page at https://github.com/unicode-org/unicodetools/releases

  2. Draft a New ReleaseClick the button,

    1. Set the "Tag version" field to a release tag with the current date, such as "release-2020-09-15"
    2. The Target is what the tag refers to. Leave this at master to tag the current master.
    3. Enter a descriptive title (such as "2020-09-15 Release"
    4. Optionally fill in some content (see image at right). PRs and issues will autocomplete if you want to mention a specific PR or issue.
      GitHub release page
    5. click Publish Release (Green button at the bottom)

  3. Now, go to the (A) Actions tab in GitHub, and click on the (B) "Push to GCR Github Action" workflow. (see illustration)

    GitHub Actions Tab

    1. Every tagged release creates a new run of this workflow. Look for your new tag to show up and wait for it to finish building (should only take a few minutes) and show a green checkmark. ( Example at (C) : "test3" and "test4" )

  4. Now, login to https://console.cloud.google.com/ with your unicode.org account

    1. At the top of the page, switch the project to Unicode Dev Infra
      choose a deployment

    2. From the left nav / hamburger menu, find "Cloud Run". Click on the link for the service "unicode-jsps-staging". This way you can update the staging release before going to production.

    3. Now, choose "Edit and Deploy a New Revision"

    4. You will get a page that prompts you to choose a docker image to be deployed. See the illustration below.

      Google Cloud Run Image Selection Page

      1. Click Select
      2. expand the "us.gcr.io/dev-infra…/unicode-jsps" images
      3. You should see the release tags (test3, test4) show up as "images".
      4. Choose the image that corresponds with your deployment (such as 2020-09-15-release) - the full title may not show up.
      5. Click Select below the image.

    5. Make sure "√ Serve this revision immediately" at the bottom is checked.

    6. Finally, click the Deploy button (way at the bottom) to schedule the deployment

    7. Allow 15 seconds or so to update (you will be given visual progress as the new revision takes over traffic.

  5. Check

    1. Test the new deployment at https://unicode-jsps-staging-5ocgitonaa-uw.a.run.app
    2. To check new characters (for example): https://unicode-jsps-staging-5ocgitonaa-uw.a.run.app/UnicodeJsps/list-unicodeset.jsp?a=\p{age=8.0}-\p{age=7.0}
    3. Check at the bottom for the right Unicode and ICU versions.
    4. Click once on each top link to do simple sanity check.
    5. Once that's successful then:

  6. Deploy to Production

    1. Repeat step §4.2 above, but choose the service unicode-jsps instead of unicode-jsps-staging.

  7. Check

    1. Go to http://unicode.org/cldr/utility/ to check that that works, using the same steps as #3, that is:
    2. To check new characters (for example): http://unicode.org/cldr/utility/list-unicodeset.jsp?a=\p{age=8.0}-\p{age=7.0}
    3. Check at the bottom for the right Unicode and ICU versions.
    4. Click once on each top link to do simple sanity check.

  8. Revert?

    1. You can choose a back-level image tag (Step 4.4 above) in order to revert to previous versions.
@srl295
Copy link
Author

srl295 commented Jul 12, 2021

notes on https://github.com/unicode-org/unicodetools/blob/main/docs/unicodejsps/index.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment