038. Use unformatted stream of Fortran 2003 for large-scale I/O

topic: Input and Output

Consider using unformatted stream I/O introduced in Fortran 2003 for large data sets. Formatted I/O converts numerical data to and from ASCII and is much slower than unformatted.

This program times the writing and reading of \(10^6\) double precision floats1 using formatted and unformatted stream I/O.

stream-io.f90 | | ₀ | |

program stream_io
   use iso_fortran_env, only: wp => real64, int64
   implicit none

   integer, parameter  :: n = 10**6
   real(kind=wp)       :: x(n), xchk(n)
   integer(kind=int64) :: iunit, irate, times(6)

   call system_clock(count_rate=irate)  ! # of clock ticks per second
   print *, "n =", n
   print *, "irate =", irate
   call random_number(x)
   call system_clock(count=times(1))

   ! Use formatted write and read
   open (newunit=iunit, file="temp.txt", action="write", status="replace")
   write (iunit, "(f0.16)") x
   close (iunit)
   call system_clock(count=times(2))
   open (newunit=iunit, file="temp.txt", action="read", status="old")
   read (iunit, *) xchk  ! List-directed read with `*` gets data from as many lines as are needed
   call system_clock(count=times(3))
   print *, maxval(x - xchk), minval(x), maxval(x), minval(xchk), maxval(xchk) ! check
   call system_clock(count=times(4))

   ! Use unformatted stream write and read
   open (newunit=iunit, file="temp.bin", action="write", status="replace", &
         form="unformatted", access="stream")
   write (iunit) x  ! Note no format supplied for unformatted I/O
   close (iunit)
   call system_clock(count=times(5))
   open (newunit=iunit, file="temp.bin", action="read", status="old", &
         form="unformatted", access="stream")
   read (iunit) xchk
   call system_clock(count=times(6))
   print *, maxval(x - xchk), minval(x), maxval(x), minval(xchk), maxval(xchk)  ! check

   ! Count differences divided by `count_rate` gives seconds elapsed
   print "(/, *(a12))", "formatted", "formatted", "unformatted", "unformatted"
   print "(*(a12))", "write", "read", "write", "read"
   print "(*(g12.3))", [times(2:3) - times(1:2), times(5:6) - times(4:5)] / real(irate, wp)

   ! Enhancement factors
   print "(/, *(a12))", "enhancement", "enhancement"
   print "(*(a12))", "write", "read"
   print "(*(g12.3))", real(times(2) - times(1), wp) / (times(5) - times(4)), &
                       real(times(3) - times(2), wp) / (times(6) - times(5))

end program stream_io

Output2

 n =     1000000
 irate =           1000000000
   5.5511151231257827E-017   4.0471064854941119E-007  0.99999926540348483        4.0471064850000000E-007  0.99999926540348483     
   0.0000000000000000        4.0471064854941119E-007  0.99999926540348483        4.0471064854941119E-007  0.99999926540348483     

   formatted   formatted unformatted unformatted
       write        read       write        read
   0.585       0.445       0.936E-02   0.137E-02

 enhancement enhancement
       write        read
    62.5        325.    

The degree of enhancement depends on \(n\) and your hardware, but it should be considerable.

Note

The Tweet, using \(n=10^7\), reported that unformatted stream write and read were, respectively, 25 and 260 times faster than formatted.

Note

Using 64-bit integer arguments when calling system_clock allows for greater timing precision3. For 32-bit integers (default) or smaller, we only get millisecond precision, which was consistently not sufficient to measure the unformatted write time on my laptop.

---

Use unformatted stream of Fortran 2003 for large-scale I/O. pic.twitter.com/I2PHa3YiaA

— FortranTip (@fortrantip) December 21, 2021

---

1

float64; \(10^6 \times 64\,\text{bits} = 8\,\text{MB}\)

2

Compiled using GNU Fortran (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 with no flags

3

https://gcc.gnu.org/onlinedocs/gfortran/SYSTEM_005fCLOCK.html