September 20, 2010

Calling all JPEG 2000 profiles

We plan to provide delegates of our JPEG 2000 Seminar (16 Nov) with a list of JPEG 2000 profiles from a number of organisations who are currently using the format. Some of these will be briefly presented during a "Profiles" session on the day.

Do you have a profile you are currently using, and would like to distribute to your peers? If so, please send the following details to Christy at c.henshaw@wellcome.ac.uk.

The specific information we're looking for includes:

Used for: (e.g Newspapers)
Conversion software used: (e.g. Kakadu)
File format: (e.g. Part 1 (.jp2)
Lossy or lossless: (choose)
Typical compression: (expressed as a ratio)
Tiling: (e.g. 1024 x 1024)
Progression order: (e.g. RPCL)
No. of decomposition levels: (e.g. 6)
Number of quality layers: (e.g. 12)
Code block size (xcb = yxb): (e.g. 6)
Transformation: (e.g. 9-7 irreversible filter)
Precinct size: (e.g. 128 x 128)
Regions of interest: (yes or no)
Code block size : (e.g. 64 x 64)
TLM markers: (yes or no)

September 14, 2010

New Wellcome Digital Library blog

The Wellcome Library has launched a new blog (wellcomedigitallibrary.blogspot.com/), centered on the development of the Wellcome Digital Library. The blog will be a "a real-time progress report, discussion outlet, and notification area."

This JPEG 2000 blog will still remain focused specifically on the work being done around JPEG 2000 at the Library, but the Wellcome Digital Library blog will provide much broader information on the programme, including:

  • What will be digitised, and how the content will be of use to researchers.
  • How we will facilitate research activity, learning, and discovery.
  • Logistics of digitisation and workflows.
  • In-house vs. outsource options.
  • Metadata.
  • Long-term data management.
  • Delivery formats, speeds, and functions.

September 10, 2010

Guest post: JPEG2000 recipes for the Aware encoder

As our first guest poster, Bill Comstock, Head of Imaging Services at Harvard College Library, writes about the specific "recipes" used at Harvard for producing JPEG2000 images.

I needed help remembering when it was that we began making JPEG2000 images. I ran a search against the Harvard Library’s preservation digital repository, DRS, and it looks like we first deposited a JP2 image in 2004.

Over the intervening six years, we’ve refined and settled on a single recipe that we use to produce lossy-compressed images, and another that we use to produce losslessly-compressed JP2s. I’ll share these recipes with you below. My reasons for sharing are two:

1) Depending upon the software used, there are many encoding combinations to consider - many more than one would have to consider when cooking up the more familiar and less complex TIFF. My group uses the Aware JPEG2000 SDK encoder. Flipping through Aware’s 330 page manual (“AccuRad J2KSuite Developer’s Guide”), I count...152 different command line options. In sharing our recipes (the combination of options and parameters that we invoke ), I’d like to speed others along in developing their own encoding formulations and JP2 production workflows.

2) Not only are there many encoding options to consider, but some are complex and a bit intimidating. Honestly, I don’t know what the “--set-input-raw-channel-subsampling” or “--set-output-j2k-rd-slope” options do. I do think that I understand the options that we use, but I may hear from one of you that my understanding is flawed and that our recipe could be improved upon, or at least better understood and explained. Here you go.

Lossless encoding (Windows command line)

j2kdriver.exe --set-input-image --set-output-j2k-color-xform YES --set-output-j2k-error-resilience ALL --wavelet-transform R53 --set-output-j2k-bitrate 0 --set-output-j2k-progression-order RLCP --tile-size 1024 1024 --output-file-type JP2 --output-file-name

Notes on individual options:

  • “--set-input-image ” reads file into the encoder's memory-buffer and auto-detects the input file format
  • “-- set output-j2k-color-xform YES” I believe that one does not need to call this option explicitly; YES seems to be the default value. The transform referred to here is a colorspace transformation from RGB to YUV. This conversion is made prior to compressing the data. Applying compression to YUV data is more efficient, yielding smaller files than the same compression applied to the unconverted RGB data.
  • --set-output-j2k-error-resilience ALL This function will take the following parameters, each explained below by Aware’s Alexis Tzannes.
  • SOP to enable Start of Packet markers
  • EPH to enable End of Packet Header markers
  • SEG to enable segmentation symbols
  • ALL to enable all of the above
  • NONE to disable all of the options
  • Resynchronization markers: Start of Packet (SOP), End of Packet Headers (EPH). These are used to signal the beginning of each packet and the end of each packet header, and can be used to resynchronize in the case of missing or corrupted data. This allows the decoder to detect and discard entire corrupted packets. So the SOP and EPH are basically tags that signal the beginning and end of a packet (a piece of coded data) in the file. If a packet gets corrupted the error resilient decoder can resync using the next SOP packet marker. The idea here is that if one packet in a file is bad, we don't lose everything that comes after it (as was the case with original JPEG). With JPEG 2000, you could lose a packet and have a small area of the image go bad, but the decoder can recover and keep on decoding.
  • Segmentation symbols: this adds a special four symbol code to specific locations in the compressed data stream, enabling error resilient decoders to detect errors, if this symbol is corrupted. This allows the decoder to detect and discard corrupted bitplanes. So this is similar, but less granular, as it operates at the bitplane level, each bitplane may include multiple packets. Overall, these features would be useful in noise prone environments or over unreliable networks.
  • --wavelet-transform R53” specifies use of the reversible "integer 5-3 filter" (compression) to produce a losslessly encoded JPEG2000 file.
  • --set-output-j2k-bitrate 0 Quoting from the “AccuRad J2KSuite Developer’s Guide”: “Sets the output image bitrate, in bits per pixel. A bitrate of 0 indicates that all the quantized data should be included in the image. This creates lossless images if the R53 wavelet is chosen Sets the output image bitrate, in bits per pixel. A bitrate of 0 indicates that all the quantized data should be included in the image. This creates lossless images if the R53 wavelet is chosen [...].”
  • “--progression-order RLCP” “For a given tile, the packets contain data from a specific layer, a specific component, a specific resolution, and a specific precinct. The order in which these packets are interleaved is called the progression order. The interleaving of the packets can progress along four axes: layer, component, resolution and precinct.” [1] A progression order that begins with “R” (resolution) indicates that the data is organized so that low resolution information will be decoded first, followed and augmented by the remaining higher resolution data in the codestream.
  • --tile-size 1024 1024 This tile size was prescribed, as it was said to be optimally matched to the software our library uses to dynamically generate and deliver JPEG files from stored JP2 masters.

Lossy encoding

j2kdriver.exe --set-input-image-file --set-output-j2k-color-xform YES --set-output-j2k-error-resilience ALL --wavelet-transform I97 --set-output-j2k-progression-order RLCP --set-output-j2k-psnr 46 --tile-size 1024 1024 --output-file-type JP2 --output-file-name

  • --wavelet-transform I97” specifies use of the “irreversible 9-7 filter” to produce a lossy encoded JPEG2000 file.
  • “--set-output-j2k-psnr 46” The pSNR function was selected because it effectively, although imperfectly, modulates the level of compression applied to each image based on the image's particular characteristics: the arrangement and variability of raster values. When we set the pSNR value to 46 db for the page-images that we create, we've come to expect a very high-quality encoded image. There are cases (certain kinds of photographs, illustrations with fine thin lines) where a 46 db setting would produce a too heavily compressed file. Too guard against over-compression, we have developed an effective (although a bit crude) method for dynamically resetting the db value if the file appears to have been too heavily compressed. This is a topic for another day.
  • I believe that most users set a fixed compression ratio, e.g., “--set-output-j2k-ratio ”.

Creating a Windows batch file

If you would like to run your encoding script over a directory of TIFF images (for example), you can create a simple batch file.

Example:

for %%f in (*.tif) do j2kdriver.exe --set-input-image-file "%%f" --set-output-j2k-error-resilience ALL --set-output-j2k-progression-order RLCP --set-output-j2k-ratio 8 --tile-size 1024 1024 --output-file-type JP2 --output-file-name "%%~nf.jp2"

Good luck putting together your own recipes and workflows. Again, please let me know if you have any suggestions for improving our practice.

[1] SO/IEC JTC1/SC29 WG1, JPEG 2000 Editor Martin Boliek, Co-editors Charilaos, C. and E.Majani. "JPEG 2000 Part I Final Committee Draft Version 1.0.", 2000, http://www.jpeg.org/public/fcd15444-4.pdf (accessed September 3, 2010).

Bill Comstock
Head, Imaging Services
Harvard College Library
Widener, D70C
Harvard Yard
Cambridge, MA 02138
http://imaging.harvard.edu/