JPEG codec and scaling
Source:alpine/aports/fplinux-jpeg
On UMS9117, fplinux-jpeg uses the image accelerator to decode JPEG files, encode NV16 frames as JPEG, and halve NV16 frames. fplinux-jpeg-cpu is a separate software tool for the same kinds of work. Neither command falls back to the other. Both are included in UMS9117 images; run them on the phone as described in Applications. The raw files they read and write are described in Raw image formats.
This guide covers UMS9117. These tools are not supplied for SC9820E.
Command
Section titled “Command”fplinux-jpeg --input FILE --output FILE [--operation decode|encode|scale] [--width N --height N] [--scale 1|2|4] [--quality 1..100] [--repeat N] [--timing] [--device PATH]--operationselects the work;decodeis the default.--repeat Nruns the same job1to4096times, checks that every result is identical, and writes the last one.--timingadds lines with the time spent in each stage, this process’s CPU time and its peak memory.--deviceselects the accelerator device; by default it is found automatically.
Each job may take at most ten seconds. A command-line, device, timeout, file or repeat mismatch error makes the command exit with a nonzero status.
Decode a JPEG file
Section titled “Decode a JPEG file”fplinux-jpeg --input /run/photo.jpg --output /run/photo.rawThe input must be a non-empty file of at most 1 MiB. The output has the JPEG’s full size: NV12 for a 4:2:0 JPEG, NV16 for a horizontal 4:2:2 JPEG. On success the command prints the output format and size; the stride and byte figures in that line describe the device buffers, not the packed file.
For half-resolution decoding, add --scale 2. It accepts both supported JPEG sampling formats, including images with padded edges. Each output dimension is rounded up after division by two.
For a quarter-size NV16 image, add --scale 4. This works only for a horizontal 4:2:2 JPEG whose width is a multiple of 16 and height a multiple of 8; the output is exactly a quarter of each dimension.
Output files have packed Y rows followed by interleaved Cb,Cr rows. A Y row contains one byte per visible pixel; a chroma row has 2 × ceil(width / 2) bytes. NV12 has ceil(height / 2) chroma rows, and NV16 has one for every Y row. The output is full-range BT.601 YCbCr without dimensions or other metadata.
The command does not apply EXIF orientation or color conversion. Its pixels can differ slightly from those of software JPEG decoders.
A large image makes a large output file: a 2048×2048 JPEG decodes to 8 MiB (4:2:2) or 6 MiB (4:2:0). /run and /tmp share phone memory with everything else, so the largest images may not fit. If the output cannot be written, the command fails with the reason and may leave a partial output file.
Supported JPEG files
Section titled “Supported JPEG files”Use a baseline YCbCr JPEG with 4:2:0 or horizontal 4:2:2 sampling, no larger than 2048×2048 pixels and 1 MiB. Progressive JPEG, grayscale, RGB and CMYK files are not supported. Exact restrictions for tools that produce these files follow below.
JPEG format reference
The decoder accepts baseline JPEG with:
- eight-bit samples, Huffman coding and one interleaved scan;
- three YCbCr components, sampled 4:2:0 or horizontal 4:2:2;
- a width and height from 1 to 2048 pixels, in a file of at most 1 MiB;
- consecutive component IDs in frame and scan order:
0,1,2,1,2,3or2,3,4, and APP14 either absent or declaring YCbCr transform1; - all quantization tables present in the file. Custom quantization tables, optimized Huffman tables and restart markers are fine, but Cb and Cr must use the same quantization values and the same Huffman tables;
- valid Huffman tables, or an abbreviated MJPEG frame that omits all Huffman definitions and uses the standard luma selector
0and chroma selector1. Partial or malformed definitions do not receive default tables.
Quantization table IDs can be 0 through 3; DC and AC Huffman table IDs can be 0 or 1. Every reference must have a valid definition, apart from the complete omitted-Huffman case above. Restart markers must follow their declared interval and RST0 through RST7 order, with only the final EOI after the entropy data.
DC tables contain 1–12 unique categories from 0–11 and at most four symbols of length 16. AC tables contain 1–162 unique symbols, coefficient sizes up to 10, and use size zero only for EOB and ZRL. The canonical tree leaves the all-ones padding code unused. The scan uses Ss=0, Se=63, Ah=Al=0. After SOS, an entropy 0xff byte must be stuffed as ff 00; only the declared restart markers and final EOI may follow the entropy data.
It rejects progressive and extended JPEG, arithmetic coding, grayscale, 4:4:4, 4:1:1 and vertical 4:2:2 sampling, RGB, CMYK and YCCK files, multiple scans, 16-bit quantization tables, files without quantization tables, and data after the end of the image.
A damaged file can pass the first checks and fail during decoding. Do not rely on a particular error for a particular defect.
Encode NV16 as JPEG
Section titled “Encode NV16 as JPEG”fplinux-jpeg --operation encode --width 320 --height 240 \ --input /run/frame.nv16 --output /run/frame.jpgThe input is a packed NV16 frame in one of these sizes:
| Width × height | Input bytes |
|---|---|
1200×32 |
76,800 |
320×240 |
153,600 |
240×320 |
153,600 |
640×480 |
614,400 |
1600×1200 |
3,840,000 |
1200×1600 |
3,840,000 |
For a camera frame, choose one of the encoder sizes above. The encoder accepts 240×320 and 1600×1200 directly; 800×600 is not an encoder size. For 1200×1600, rotate a 1600×1200 frame with fplinux-rotate first.
The input is full-range BT.601 NV16. NV12 input is not supported by the hardware encoder. The output is a baseline 4:2:2 JPEG with restart markers. Quality defaults to 85; use --quality 1..100 to change it:
fplinux-jpeg --operation encode --quality 60 --width 320 --height 240 \ --input /run/frame.nv16 --output /run/frame.jpgThe command prints the quality read back from the encoder. If the result would exceed 1 MiB, it fails instead of writing it. Custom tables, sampling and restart interval cannot be changed.
Halve an NV16 frame
Section titled “Halve an NV16 frame”fplinux-jpeg --operation scale --width 640 --height 480 \ --input /run/frame.nv16 --output /run/half.nv16The source size selects one of two fixed conversions:
| Source | Result | Input bytes | Output bytes |
|---|---|---|---|
640×480 |
320×240 |
614,400 | 153,600 |
320×240 |
160×120 |
153,600 | 38,400 |
Both files are packed NV16. Do not pass --scale. Other sizes, ratios, cropping and format conversion are not available.
If the accelerator fails
Section titled “If the accelerator fails”If the accelerator cannot be confirmed idle after an error, it refuses all further jobs until the phone is powered off and started again. fplinux-jpeg-cpu keeps working in the meantime.
Before combining camera capture with JPEG jobs, check phone support status and the camera resource limits.
Software tool
Section titled “Software tool”fplinux-jpeg-cpu never uses the accelerator. Its command forms are:
fplinux-jpeg-cpu --operation encode --input NV16 --output JPEG --width W --height H [--quality 1..100] [--restart 0..65535] [--dqt FILE] [--repeat 1..4096]fplinux-jpeg-cpu --operation decode --input JPEG --output NV12_OR_NV16 [--scale 1|2|4] [--decode-mode reduced|box] [--repeat 1..4096]fplinux-jpeg-cpu --operation scale --input NV16 --output NV16 --width 640 --height 480 [--scale 2] [--repeat 1..4096]fplinux-jpeg-cpu --operation scale --input NV16 --output NV16 --width 320 --height 240 [--scale 2] [--repeat 1..4096]- Encode reads packed full-range BT.601 NV16 with a positive even width and a positive height, and produces a 4:2:2 JPEG. The default quality is 75 and the default restart interval is 0.
--dqt FILEreads 128 nonzero bytes: the luma and then the chroma quantization table, each in natural DCT order. - Decode requires non-progressive eight-bit YCbCr JPEG with 4:2:0 or horizontal 4:2:2 sampling. It writes packed NV12 or NV16 respectively. Its accepted files can differ from the accelerator’s; success does not extend what
fplinux-jpegcan decode. The default divisor is1.--decode-mode reduced, the default, decodes directly at the smaller size;boxdecodes at full size and then averages. - Scale supports the same two conversions as
fplinux-jpeg. Its pixels can differ from the accelerator’s.
The input must be a non-empty regular file of at most 32 MiB. Raw input must contain exactly width × height × 2 bytes; decoded raw buffers are also limited to 32 MiB. Memory needed for the codec and additional buffers comes on top of these sizes, so a job can run out of phone memory below those limits.
--repeat defaults to 1. Repeated jobs must produce identical results or the command fails. The software tool always prints the time spent in each stage.
Timing reference
Section titled “Timing reference”Use --timing to inspect where the hardware command spends time. fplinux-jpeg-cpu always prints timing. These figures measure the command’s work, not the time until an image becomes visible on the screen.
Timing fields and measurement boundaries
Add --timing to fplinux-jpeg to print nanosecond wall-clock components, total process user and system CPU time in microseconds, peak resident memory, and page-fault counts. The wall fields have these boundaries:
wall_input_nsreads the input file;wall_prepare_nsopens and negotiates the device, prepares buffers and queues, and allocates the result buffer;wall_loops_nscovers all job-loop iterations, whilewall_warm_loops_nsexcludes the first iteration;wall_queue_nsmeasures queue calls andwall_wait_nsmeasures completed-job waits; theirwall_warm_*forms exclude the first iteration;wall_output_nswrites the result file;wall_total_nsspans input reading through the completed output write, including the stream-off, unmapping, and device close before that write.
Queue and wait measurements are components inside the loop measurements, so they must not be added to wall_loops_ns. The warm fields are meaningful only when --repeat is greater than one. user_total_us and sys_total_us are this process’s CPU time, not total system CPU load.
The CPU tool always prints wall, user, system, and combined process CPU milliseconds for its input load, codec or filter work, batch, output write, and measured_stages. Operation-specific codec/conversion values are components of the batch and must not be added to it.