JPEG CORRUPTION

Photographs stored as JPEG files on a Windows-based PC are in danger of being severely corrupted over some period of time. Here is the story behind this statement.

Two years ago I copied a folder of jpeg files from my PC's desktop to an internal back-up hard disk in my computer. After the copy was complete, I opened the newly made folder on the back-up disk, and browsed the jpeg file copies in the Windows Picture and Fax Viewer utitlity. I was surprised to find that about 5 to 10 per cent of the copied files had a strange corruption. A thin light grey horizontal line was visible at random heights and starting points in each of the corrupted images. A sample is shown below:

The arrow points to the tell-tale grey horizontal line.

Photoshop complains that the file is damaged.

And if you ask Photoshop to continue, this is what you get.

I checked the original images in the source folder on the desktop and found that the files were all okay - no corruptions. So I did a test.. I copied the folder again to a different partition on the back-up hard disk. The same problem occurred - some of the images were corrupted with the thin grey horizontal line. But they were not the same images that became corrupted during the first copy I made of the folder.

I was getting worried. It seemed that I couldn't trust my computer to reliably make copies of my JPEG files. After a lot of trials and investigations, I concluded that there was something wrong with my PC's motherboard. So I built a new PC based on a motherboard that used a different chipset, and then the problem seemed to go away. I was relieved to think that I had solved the problem.

A few months ago, I browsed some of the old folders from the old back up hard disk that was now living in my new PC. I was shocked and dissapointed to find that not only were there corrupted files, but the degree of corruption was now far worse than what I had seen two years ago. It seems that the JPEG files on my hard disk are deteriorating. There are multiple lines of random colour now, not just simple grey lines. And some files are so badly corrupted that they are now a mish-mash of horizontal slices, or they can't be opened at all.

Here are some examples:

Note that there are multiple horizontal stripes, indicating multiple corruptions.

Multiple corruptions, again......

The corrupted JPEG files are now not only more numerous than they were two years ago, but the degree of corruption is becoming much worse. Many of the corrupted JPEG files cannot be opened at all by Windows Picture & Fax Viewer, nor can that program make thumbnails of the very badly corrupted images.

Where Windows Picture & Fax Viewer can open the file at all, I'm now seeing some very serious damage to the images - far beyond simple grey lines. Here are some samples:

This is a very serious problem, and it will just get worse with time, because two things are inevitable:

Firstly, my hard disks are ageing, and with that age comes an increased "bit error rate" (BER). The JPEGS are corrupted because the BER for the hard disk is too high for the usual error correction mechansim built into the disk drive. When one bit becomes corrupted in a JPEG file, it can cause the loss of a whole section of the image (the horizontal grey stripe, as a minimum).

Secondly, I am accumulating more and more JPEG files every month, as I pursue my photography business and hobby. There are now over 34,000 jpeg files on my workstation, and the really scary thing is that I don't know how many of them are corrupted. It's impractical for me to check them all manually by trying to view all 34,000 of them with my own eyes.

I wondered if the problem was unique to JPEG files, or whether any type of file is prone to corruption. Investigation found that any file is prone to the same type of corruption. But JPEG files show the corruption in a more dramatic way, due to the fact that small bit errors can destroy the 'key' to decoding larger sections of a photograph.

Here's an example of what the bit errors look like in a TIFF file:

The bit errors in a TIFF file only damage a small number of pixels. There's no gross multiplier effect like there is with bit errors in a JPEG file. The small corruptions in the TIFF file, shown as short, narrow "drop-outs" (a video term) could be repaired easily enough in Photoshop.

So it seems that TIFF files are more robust as an archival format for photographs.

Photoshop .PSD files are also victim to the bit error corruptions, and Photoshop can detect the fact of corrupted pixels in a .PSD file when opening a corrupted file. The following message will appear:

I figure that there must be other people in the world who are having the same problem, so perhaps the solution that I come up with might be useful to others too, not just me. For that reason, I am investing in the development of a software solution, and I hope I might be able to recover some of that investment by sharing the solution with other photographers who are suffering from the same difficulty.

Here's the basic overview of the technical problem, and the approach I am taking to solve it forever:

1. Consumer-oriented operating systems like Windows XP and the typical hardware that they run on do not have the "high integrity" features of the more professional operating systems like one would find on say, an IBM server running z/OS or a Sun Microsystems workstation running the Solaris operating system. A simple example is how the Windows XP CD-burning utility offers no option to verify the data on the CD after it was burned. ( I don't use Apple's Mac OS-X so I can't compare it to WIndows or know if it provides better data integrity or not.)

2. Due to the ubiquitous accumulation of rich-media data such as photographs, digital video, digital audio and the like, consumers are relentlessly using up hundreds of gigabytes of data storage on their home PC systems. The standards for consumer PC data integrity have not kept up with the growth of consumer data accumulation.

3. Many professionals, such as photographers and musicians, are trying to run their business on unprofessional operating systems like WIndows XP. (I am one of them.) The business risk associated with lost data is very high.

4. Data storage media like hard disk drives and recordable CD's and DVD's and even digital videotape, are not perfect. Any storage medium has a native bit error rate (BER) and is equipped with Error Checking and Correction (ECC) to improve the native bit error rate from something like one bad bit in a million (1 x 10e-6) to a net error rate after correction of something like 1 bad bit in 100 quadrillion (1 x 10e-14).

5. After just a few years, typically soon after the warranty has expired, hard disk performance and reliability will noticeably degrade. Replacing the hard disk with a new one is recommended.

And here is the overview of my proposed solution:

Step 1: find out if your PC has the problem or not, and to what degree it may have the problem.

Step 2: Locate every corrupted JPEG file. See if you can find a good copy of it elsewere.

Step 3: Determine the 'fitness' of your hard disk drive and make a judgment as to whether you should continue using it or replace it.

Step 4: To prevent the problem from occurring again, do the following:

4.1 employ a 'verify after copy' method to confirm that newly copied files and folders are not corrupted

4.2 employ an ECC wrapper around JPEG files so that errors can be detected and corrected when they are read from degrading media some time later

4.3 upgrade your computer hardware to "server class" equipment, which has ECC on the memory and robust ECC on the hard disc storage subsystem

I am currently working on Steps 1 & 2. Please see the next page on this web site, about my JPEG FILE VERIFIER utility (JFV). Let me know if it's of interest to you.

This link will take you to the JFV page: JPEG FILE VERIFIER

You can email me via craig [at] alkiratech [dot] com.