Downing was born in Oak Grove, Missouri. At the time of his high school graduation, America was in the Great Depression. Jobs were scarce, so Downing enlisted in the Navy in September 1932. He began serving at Long Beach Naval Shipyard, California, the port for most of the Pacific naval fleet. He became a gunner's mate and postmaster aboard the USS West Virginia. On the morning of December 7, 1941, Downing and his new wife Morena were staying with his shipmates in Honolulu. The radio announced an attack on the Naval Station Pearl Harbor, and the sailors rushed to the port. By the time Downing arrived at the West Virginia, Japanese bombers had targeted her with nine torpedoes. Moored at a shallow water depth of only 40 feet (12 m), the ship had not completely sunk, and her deck was on fire. Downing assisted the injured crew and used a water hose from the USS Tennessee to keep the ammunition onboard from exploding.

Downing tried to commit to memory the names on the dog tags of those killed, and later visited the wounded at the naval hospital. He recorded the names of the wounded and fatally injured in a notebook, along with messages they wished to convey. As the postmaster, Downing had access to the home addresses of these men, and he later wrote notes to each family, explaining what had happened to their relatives and their current condition.

Prior to Pearl Harbor, Downing befriended Dawson Trotman, a Christian minister who eventually founded The Navigators, an interdenominational ministry. Trotman died the same year that Downing retired from the Navy, and the Downings moved to Colorado Springs, Colorado, taking over Glen Eyrie, home of the branch there. Downing served as deputy president, chairman of the board of directors and director of the ministries in Europe, the Middle East, and Africa. He retired as a full-time minister in 1983

James meet Morena Mae Holmes and were married July 1941. They had seven children and remained in Colorado Springs CO. after his retirement.