JX3fExtract 2.1 library and command line interface

JX3fExtract is a set of Java modules which provides the following features:

  • extracting of processed for preview images and metadata embedded in Foveon X3F files ("FOVb")
  • extraction of X3F data from X3I files ("FOVi")
  • a command line interface to the above

The minimum Java source code level to compile this library is 9.

Module dependencies

The following diagram shows the dependencies between the modules (module names are abbreviated by omitting the common domain name prefix):

[Module dependencies for JX3fExtract]

Library usage

The easiest way to use the library is to call one of the overloaded FovDataFactory.parseOrRecover(…) methods.

For a more advanced usage see their implementation in the source code. Two of the implementations are shown here as an example – the following is for file access:

public static <R> Optional<R> parseOrRecover(Path file, boolean readFirstFOVbOnly,
  BiFunction<RandomAccessData, FovDataType, Optional<R>> resultHandler)
  throws RandomAccessDataException
{
  FovDataType fovData;
  try (RandomAccessData randomAccess = RandomAccessDataFactory.createInstance(file))
  {
    fovData = parse(randomAccess, readFirstFOVbOnly);
    if (!fovData.isParsedSuccessfully() && fovData.isRecoverable())
    {
      try (RandomAccessData stream = RandomAccessDataFactory
        .createInstance(file.toFile(), AccessType.STREAM))
      {
        fovData = recover(stream);
      }
    }
    return (resultHandler != null) ?
      resultHandler.apply(randomAccess, fovData) : Optional.empty();
  }
}
                        

and the following is for in memory data given as a byte array:

public static <R> Optional<R> parseOrRecover(byte[] bytes, boolean readFirstFOVbOnly,
  BiFunction<RandomAccessData, FovDataType, Optional<R>> resultHandler)
  throws RandomAccessDataException
{
  FovDataType fovData;
  try (RandomAccessData randomAccess = RandomAccessDataFactory.createInstance(bytes))
  {
    fovData = parse(randomAccess, readFirstFOVbOnly);
    if (!fovData.isParsedSuccessfully() && fovData.isRecoverable())
    {
      randomAccess.setCurrentOffset(0);
      fovData = recover(randomAccess);
    }
    return (resultHandler != null) ?
      resultHandler.apply(randomAccess, fovData) : Optional.empty();
  }
}
                        

For a more comprehensive, working example see the source code of the command line module de.bernd_michaely.x3f.extractor.cli .

Upgrading from version 1.0

Module de.bernd_michaely.x3f.types essentially provides – together with its dependencies – the functionality of version 1.0 of this library.

It is still possible to use this module on its own (package names have changed because of the Java9 modularisation, they would have been to quirky otherwise, so you might have to adapt some import statements in existing code). This is the easiest upgrade path, if you e.g. just want to modularize an existing working application.

Nonetheless it is strongly recommended to use the new de.bernd_michaely.x3i.types module which encapsulates and extends the functionality of the former module. (In particular the performance of the recovery process has been dramatically improved.)

Command line usage

The *.jar distribution of the command line interface (without a linked Java Runtime Environment) can be run as shown with the provided JX3fExtract bash script:

#!/bin/bash

VM_OPTIONS=
#VM_OPTIONS=--show-version

# location of *.jar modules:
DIR=`dirname $0`
# (in this case the *.jar modules reside in the same directory as this script)

# alternative layout:
# script in /usr/local/bin and *.jar modules in:
#DIR=/usr/local/share/java/jx3fextract

$JAVA_HOME/bin/java $VM_OPTIONS -p "$DIR" -m de.bernd_michaely.x3f.extractor.cli "$@"
                        

For help on command line usage simply run the application without arguments or one of the --help or --help-extended arguments.

The output of the latter is shown here:

Utility for extraction of processed for preview images and
metadata from X3F and X3I files, including automatic recovery.

Library version            : 2.1 (build r481)
Supported X3F data version : 4.x
X3I support                : yes

OPTIONS:
-x  --extract        same as -j and -f
-j  --extract-jpg    extract embedded jpg from X3F or X3I <file> to <file.jpg>
-f  --extract-x3f    extract embedded X3F data from X3I files
-F  --first-only     for X3I files, with options -j, -m, -f,
                     use the first embedded X3F entry only
-h  --help           print this help to stdout
-H  --help-extended  print extended help with examples
-m  --metadata       print metadata from X3F and X3I file to stdout
-o  --overwrite      overwrite existing files
-s  --subdir         write embedded data from X3I to a subdirectory
                     named after the X3I files basename
-S  --subdir-single  write embedded data from all X3I to one single separate directory
                     with the name given as parameter (relative or absolute)
-d  --target-dir     target base directory for extracted files,
                     default is the current working directory
-v  --verbose        be verbose
-V  --version        print version information

For input files preceded with a relative path, this path is replicated in the target directory.
For input files preceded with an absolute path, this path is ignored.

EXAMPLES:

Print metadata for SomeFile.X3F to stdout:
> JX3fExtract -m SomeFile.X3F
or
> JX3fExtract --metadata SomeFile.X3F

Extract all data embedded in X3F and X3I files in current directory:
> JX3fExtract -vx *

Extract all X3F data embedded in X3I files in current directory,
for each X3I file into a separate subdirectory:
> JX3fExtract -vfs *.X3I

Use the find command to extract JPGs from all X3F files
found recursively in all subdirectories to 'outputdir':
> find . -type f -name '*.X3F' -exec JX3fExtract -vjd outputdir '{}' +

(Note that using the form of the -exec action closed with '+' may be
 significantly faster than the form closed with ';' .)
                        

Camera model support

The library supports the following camera models with the denoted features:

Supported camera models
Camera model Processed for preview image Thumbnail Header information X3F properties
Polaroid x530 JPG Uncompressed RGB (supported, converted to JPG) Supported Supported
Sigma SD9, SD10 Proprietary data format, currently unsupported Uncompressed RGB (supported, converted to JPG) Supported Supported
Sigma SD14, SD15, DP1, DP2, DP1s, DP2s, DP1x, DP2x JPG JPG Supported Supported
Sigma Merrill generation JPG JPG Partially supported (format 2.x incomplete, format 3.x unsupported) Supported
Sigma Quattro generation JPG (Not applicable, access possible to the thumbnail embedded in the JPG's Exif metadata) Partially supported (Not applicable)

Furthermore it is possible to extract the X3F data embedded in the X3I file format.

Note that for the processed for preview images in JPG format there is the RadInputStream adapter class which provides an input stream interface to a random access data source. This allows for efficient access to the Exif metadata embedded in the JPG files, e.g. by third party libraries.

Metadata output example

Reading file »SDIM5300.X3F« …
file type                 : FOVb
header version            : 2.2
is reader compatible ?    : yes
has extended 2.1 header ? : yes
Unique Identifier         : 30 31 30 30 39 36 37 38 0D 6B 17 52 32 31 34 66
Mark bits                 : 00000000 00000000 00000000 00000000
                          : 76543210 76543210 76543210 76543210
Image columns             : 2640
Image rows                : 1760
Rotation                  : 0 degrees
White balance label       : »Auto«
Extended header data [ 1] : Type  1 : Exposure adjust         :  -0.300
Extended header data [ 2] : Type  2 : Contrast adjust         :   0.500
Extended header data [ 3] : Type  3 : Shadow adjust           :  -0.100
Extended header data [ 4] : Type  4 : Highlight adjust        :   0.800
Extended header data [ 5] : Type  5 : Saturation adjust       :   0.000
Extended header data [ 6] : Type  6 : Sharpness adjust        :   0.000
Extended header data [ 7] : Type 10 : X3 Fill Light adjust    :   0.300
Extended header data [ 8] : Type  7 : Color adjust red        :   1.000
Extended header data [ 9] : Type  8 : Color adjust green      :   1.000
Extended header data [10] : Type  9 : Color adjust blue       :   1.000
parser in recovery mode ? : no
–––––––––––––––––
DIRECTORY SECTION
–––––––––––––––––
Size of data source       : 0x00EF4F00
Directory Pointer @       : 0x00EF4EFC
Directory section @       : 0x00EF4EB4
Directory version         : 2.0
Number of dir entries     : 5
Directory entry #  1      : X3fDirectoryEntry [@ 0x000000E8 + 0x0014BACC = 0x0014BBB4 (IMA2)]
Directory entry #  2      : X3fDirectoryEntry [@ 0x0014BBB4 + 0x00068B84 = 0x001B4738 (CAMF)]
Directory entry #  3      : X3fDirectoryEntry [@ 0x001B4738 + 0x00000454 = 0x001B4B8C (PROP)]
Directory entry #  4      : X3fDirectoryEntry [@ 0x001B4B8C + 0x00D3C13C = 0x00EF0CC8 (IMA2)]
Directory entry #  5      : X3fDirectoryEntry [@ 0x00EF0CC8 + 0x000041EC = 0x00EF4EB4 (IMA2)]
–––––––––––––
PROPERTY LIST
–––––––––––––
                               Auto exposure mode [»AEMODE«] = 8-segment [»8«]
                                  Auto focus mode [»AFMODE«] = single [»AF-S«]
                         Aperture value (exact) [»APERTURE«] = 8.00000 [»8.00000«]
                                Aperture display [»AP_DESC«] = 8.0 [»8.0«]
                           Auto bracketing index [»BRACKET«] =   [» «]
                                  Burst mode index [»BURST«] = 0 [»0«]
                            Camera manufacturer [»CAMMANUF«] = SIGMA [»SIGMA«]
                                   Camera model [»CAMMODEL«] = SIGMA DP1 [»SIGMA DP1«]
                                     Camera name [»CAMNAME«] =   [» «]
                          Camera serial number [»CAMSERIAL«] = ####### [»#######«]
                               Drive dial position [»DRIVE«] = Single [»SINGLE«]
                Exposure compensation (user-set) [»EXPCOMP«] = +0.00 [»+0.00«]
     Exposure compensation (user plus bracketing) [»EXPNET«] = +0.00 [»+0.00«]
                                 Exposure length [»EXPTIME«] = 12500 µs [»12500«]
                               Firmware version [»FIRMVERS«] = 2.00 [»2.00«]
                                     Flash setting [»FLASH«] = Off [»OFF«]
                                    Focal length [»FLENGTH«] = 16.6 mm [»16.6«]
                            Focal length (35mm) [»FLEQ35MM«] = 28 mm [»28«]
                                      Focus status [»FOCUS«] = Auto focus locked [»AF«]
                            Imager board ID [»IMAGEBOARDID«] =   [» «]
         Temperature of sensor / header board [»IMAGERTEMP«] = 48 °C [»48«]
                                           ISO speed [»ISO«] = 100 [»100«]
                          Lens aperture range [»LENSARANGE«] = 4.0 [»4.0«]
                      Lens focal length range [»LENSFRANGE«] = 16.6 mm [»16.6«]
                               Lens identifier [»LENSMODEL«] =   [» «]
                                     Shooting mode [»PMODE«] = Aperture priority [»A«]
                          Resolution of image [»RESOLUTION«] = Hi [»HI«]
                       Sensor type and revision [»SENSORID«] =   [» «]
                           Shutter speed (exact) [»SHUTTER«] = 0.01250 s [»0.01250«]
                         Shutter speed (display) [»SH_DESC«] = 1/80 [»1/80«]
                                 Date/Time original [»TIME«] = Aug 23, 2013, 2:00:45 PM [»1377266445«]
                           White balance setting [»WB_DESC«] = Auto [»Auto«]
–––––––––––––––
IMAGE DATA LIST
–––––––––––––––
  1 : version  : 2.0
    : type     :   2 - is processed for preview ? yes
    : format   :  18 - JPEG-compressed 8/8/8 RGB
      size     : 2640 x 1760 pixel
      row size : 0 bytes
  2 : version  : 2.0
    : type     :   3 - is processed for preview ? no
    : format   :  30 - RESERVED
      size     : 2688 x 1792 pixel
      row size : 13877536 bytes
  3 : version  : 2.0
    : type     :   2 - is processed for preview ? yes
    : format   :  18 - JPEG-compressed 8/8/8 RGB
      size     : 221 x 147 pixel
      row size : 16846 bytes
                        
Modules 
Module Description
de.bernd_michaely.common.cli.parser
This module contains utilities to provide a GNU getopt style command line argument parsing.
de.bernd_michaely.common.io.random
This module contains classes for efficient random access of miscellaneous data sources like files or byte arrays.
de.bernd_michaely.x3f.extractor.cli
This module provides a command line interface to the JX3fExtract library.
de.bernd_michaely.x3f.types
This module contains classes for creating instances of parser objects for X3F (FOVb) files and types describing their content.
de.bernd_michaely.x3i.types
This module contains classes for creating instances of parser objects for X3I (FOVi) files and types describing their content.