QB64 Phoenix Edition v3.12.0 Released!

[mdijkens]

I know regular string are 2GB max and arrays can by up to RAM limit.
But I thought I once found a way to have bigger strings, but maybe that was in another life :-)

Yes I normally read in blocks of 4MB (on average fastest) with files over 100GB :-)
Somewhere on the forum I already explained that a lot of my utilities are about dealing with huge (log)files: conversion, splitting, concatenating, extraction, counting etc. and QB64 is great for that

This is one of my most used functions that works extremely fast:

Code: (Select All)

Function BIG.read& (fileName$, eol$) ' 4M lines/sec
  Const BLOCKSIZE = 4194304 '=64*65536 = 4 MB
  If Not _FileExists(fileName$) Then CSV.read& = 0: Exit Function
  eoll% = Len(eol$)
  Dim block As String * BLOCKSIZE
  ff% = FreeFile
  Open fileName$ For Binary Access Read As #ff%
  blocks& = LOF(ff%) \ BLOCKSIZE: blocks& = blocks& - ((LOF(ff%) Mod blocks&) > 0)
  sep& = 0
  lines& = -1
  $Checking:Off
  For curblock& = 1 To blocks&
    Get #ff%, , block
    If curblock& > 1 Then
      buf$ = Mid$(buf$, sep&) + block
      r0& = InStr(buf$, eol$) + eoll%
    Else
      buf$ = block
      r0& = 1
    End If
    r1& = InStr(r0&, buf$, eol$)
    Do While r1& >= r0& And r0& > 0
      lin$ = Mid$(buf$, r0&, r1& - r0& + eoll%)
      ret% = BIG.line(lin$) ' Process lin$
      lines& = lines& + 1
      sep& = r1&: r0& = r1& + eoll%: r1& = InStr(r0&, buf$, eol$)
    Loop
  Next curblock&
  $Checking:On
  Close #ff%
  buf$ = ""
  BIG.read = lines&
End Function