Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Suggestion: Preallocated file open
#9
As explained above, fragmentation is inherent of OS/filesystem.

On a side-note, when reading huge textfiles, this approach is a lot faster:

Code: (Select All)
Function BIG.read& (fileName$, eol$) ' 4M lines/sec
  Const BLOCKSIZE = 4194304 '=64*65536 = 4 MB
  If Not _FileExists(fileName$) Then BIG.read& = 0: Exit Function
  'Print "Reading lines from "; fileName$; " ";: cpos% = Pos(0)
  eoll% = Len(eol$)
  Dim block As String * BLOCKSIZE
  ff% = FreeFile
  Open fileName$ For Binary Access Read As #ff%
  blocks& = .5 + LOF(ff%) / Len(block)
  sep& = 0
  lines& = -1
  $Checking:Off
  For curblock& = 1 To blocks&
    Get #ff%, , block
    If curblock& > 1 Then
      buf$ = Mid$(buf$, sep&) + block
      r0& = InStr(buf$, eol$) + eoll%
    Else
      buf$ = block
      r0& = 1
    End If
    r1& = InStr(r0&, buf$, eol$)
    Do While r1& >= r0& And r0& > 0
      lin$ = Mid$(buf$, r0&, r1& - r0& + eoll%)
      ret% = BIG.line(lin$) ' Process lin$
      lines& = lines& + 1
      sep& = r1&: r0& = r1& + eoll%: r1& = InStr(r0&, buf$, eol$)
    Loop
    'Locate , cpos%, 0: Print lines&;
  Next curblock&
  $Checking:On
  Close #ff%
  buf$ = ""
  'Locate , cpos%, 0
  BIG.read& = lines&
End Function

Function BIG.line% (lin$)
  'process lin$ here
End Function
45y and 2M lines of MBASIC>BASICA>QBASIC>QBX>QB64 experience
Reply


Messages In This Thread
Suggestion: Preallocated file open - by doppler - 01-09-2023, 01:01 AM
RE: Suggestion: Preallocated file open - by mdijkens - 01-09-2023, 04:50 PM



Users browsing this thread: 8 Guest(s)