Has anyone used the RtlCompressBuffer NT API?
José, it is undocumented in your WinAPI header.
...
It is not part of the SDK, but of the DDK (Driver Development Kit), and I think it's only available statically linking to Ntoskrnl.lib.
It is part of the Runtime Library Routines
http://msdn.microsoft.com/en-us/library/bb981783.aspx (http://msdn.microsoft.com/en-us/library/bb981783.aspx)
Available in Microsoft Windows XP and later versions of all Windows operating systems.
...
RTL means Runtime Library, but RtlCompressBuffer belongs to Windows Driver Kit -> Device and Driver Technologies -> Installable File System Drivers.
Headers: Declared in Ntifs.h. Include Fltkernel.h or Ntifs.h.
Library: Contained in Ntoskrnl.lib.
You need to install the DDK if you want the headers, and the headers aren't worth a penny if the function is only available in a static library.
José,
Here is the correct declaration to use it.
%COMPRESSION_FORMAT_LZNT1 = &H0002
%COMPRESSION_ENGINE_STANDARD = &H0000 '// Standart compression
%COMPRESSION_ENGINE_MAXIMUM = &H0100 '// Maximum compression
%STATUS_BAD_COMPRESSION_BUFFER = &HC0000242&
%STATUS_UNSUPPORTED_COMPRESSION = &HC000025F&
%STATUS_INVALID_PARAMETER = &HC000000D&
DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer" ( _
BYVAL CompressFormat AS WORD, _
BYVAL ptrSrceBuffer AS DWORD, _
BYVAL SrceBufferSize AS LONG, _
BYVAL ptrDestBuffer AS DWORD, _
BYVAL DestBufferSize AS LONG, _
BYVAL ChunkSize AS LONG, _
BYREF CompressedSize AS LONG, _
BYVAL WorkspaceBuffer AS DWORD) AS LONG
DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize" ( _
BYVAL CompressFormat AS WORD, _
BYREF pNeededBufferSize AS LONG, _
BYREF pFragmentSize AS LONG) AS LONG
DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer" ( _
BYVAL CompressionFormat AS WORD, _
BYVAL ptrDestBuffer AS DWORD, _
BYVAL DestBufferSize AS LONG, _
BYVAL ptrSrceBuffer AS DWORD, _
BYVAL SceBufferSize AS LONG, _
BYREF pDestinationSize AS LONG ) AS LONG
It works very well, except in one case, where i got %STATUS_BAD_COMPRESSION_BUFFER and i have no idea why.
Hence the reason why i was asking if someone else was using it.
My goal was to use it to produce lossless compression, and a good alternative to zLib, that would not require an extra DLL as it is used by the OS itself to perform file compression/decompression.
...
Anyway, the headers are not included in the Platform SDK, but in the Windows Driver Kit, that I don't have: http://www.microsoft.com/downloads/details.aspx?FamilyID=2105564e-1a9a-4bf4-8d74-ec5b52da3d00&displaylang=en
José,
It has been available for a long time, since Windows XP, however it was an undocumented feature. Microsoft started to document it, as well as many other API, with the new SDK for Seven.
I thought it was worth mentioning it for the community.
...
When using RtlDecompressBuffer with very small files, it is VERY IMPORTANT to use the same size for the resulting buffer than the original before compression, or you may encounter a %STATUS_BAD_COMPRESSION_BUFFER error!
The best solution is to use a header to store the information about the original buffer size, and add it to the resulting compressed buffer. If you do this, then you will have a very reliable alternative to ZIP, producing small and fast lossless compressed file.
...
Patrice, I was able to use your fragments from here and the PB forum and got this working (below).
Now I'm trying to figure out how to open an existing "file.zip" file.
Thx! : )
#COMPILE EXE
#DIM ALL
DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer" ( _
BYVAL CompressionFormat AS LONG, _
BYVAL SourceBuffer AS DWORD PTR, _
BYVAL SourceBufferLength AS LONG, _
BYVAL DestinationBuffer AS DWORD, _
BYVAL DestinationBufferLength AS DWORD PTR, _
BYVAL Unknow AS LONG, _
BYREF pDestinationSize AS DWORD, _
BYVAL WorkspaceBuffer AS DWORD PTR ) AS LONG
DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize" ( _
BYVAL CompressionFormat AS LONG, _
BYREF pNeededBufferSize AS LONG, _
BYREF pUnknown AS LONG ) AS LONG
DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer" ( _
BYVAL CompressionFormat AS LONG, _
BYVAL DestinationBuffer AS DWORD PTR, _
BYVAL DestinationBufferLength AS LONG, _
BYVAL SourceBuffer AS DWORD PTR, _
BYVAL SourceBufferLength AS LONG, _
BYREF pDestinationSize AS LONG ) AS LONG
FUNCTION RTL_DECOMPRESS ( BYVAL input_Buffer AS STRING, BYREF output_buffer AS STRING ) AS LONG
LOCAL pOutBuffSize AS LONG
' make maximal destination buffer
IF LEN( output_buffer ) = 0 THEN
output_buffer = STRING$( LEN( input_buffer ) * 12.5, 0 )
END IF
' decompress buffer
RtlDecompressBuffer ( BYVAL &H02, _
BYVAL STRPTR( output_buffer ), _
BYVAL LEN(output_buffer), _
BYVAL STRPTR( input_buffer ), _
BYVAL LEN( input_buffer), _
pOutBuffSize )
output_buffer = LEFT$(output_buffer, pOutBuffSize )
' function returns output buffer size, zero if error
FUNCTION = pOutBuffSize
END FUNCTION
FUNCTION RTL_COMPRESS ( BYVAL input_Buffer AS STRING, BYREF output_buffer AS STRING ) AS LONG
LOCAL ComprFormat, L, pOutBuffSize, NeedBuffSize AS LONG
LOCAL Need_Buff AS STRING
' set compression format
' use LZNT1 with MAXIMUM compression
ComprFormat = MAK ( LONG, &H02, &H0100 ) '
' Calculate the workspace for compression engine
RtlGetCompressionWorkSpaceSize ( ComprFormat, NeedBuffSize, &h4000 )
' define compress parameters
Need_Buff = STRING$( NeedBuffSize, 0 )
L = LEN( input_buffer )
IF LEN( output_buffer ) = 0 THEN output_buffer = STRING$ ( L, 0 )
' compress input buffer ...
RtlCompressBuffer ( BYVAL ComprFormat, _
BYVAL STRPTR( input_buffer ), _
BYVAL L, _
BYVAL STRPTR( output_buffer ), _
BYVAL L, _
BYVAL &h2000, _
pOutBuffSize, _
BYVAL STRPTR(Need_Buff) )
output_buffer = LEFT$( output_buffer, pOutBuffSize )
' function returns output buffer size, zero if error
FUNCTION = pOutBuffSize
END FUNCTION
FUNCTION PBMAIN () AS LONG
LOCAL sz, rt AS LONG, file, src_buff, dest_buff AS STRING
file = "data.txt"
' Compression test
OPEN file FOR BINARY AS 1: sz = LOF(1): GET$ 1, sz, src_buff: CLOSE 1
rt = RTL_COMPRESS ( src_buff, dest_buff )
OPEN file+".lz" & file FOR BINARY AS 1: PUT$ 1, dest_buff: CLOSE 1
? "Compression Done." & $CR & "Output buffer size=" & STR$(rt)
' Decompression test
RESET src_buff
rt = RTL_DECOMPRESS ( dest_buff, src_buff )
OPEN file+".unlz" FOR BINARY AS 1: PUT$ 1, src_buff: CLOSE 1
? "DeCompression Done." & $CR & "Output buffer size=" & STR$(rt)
END FUNCTION
If you want to open a ZIP file, then you must use code from Jean-Loup Gailly.
http://gailly.net/
...
I did some experiment with RtlCompressBuffer & co.
In the example above the output_buffer size is set to LEN(input_buffer) * 12.5
I do not know where 12.5 come from but the compression ratio can be
a lot bigger than that, like if we compress STRING$(1,000,000 "A").
So I used a loop to grow the buffer step by step to near
the PowerBASIC string size limit if needed.
Also, Patrice, about very small string compression,
if one want to avoid using a header.
As we know, the compressed result may be longer than the uncompressed source,
normal with a one byte string source.
But only by 3 bytes according to my tests under XP-32 and Se7en-64.
So increasing unCompressedBufferSize by 3 bytes resolve the problem
and keep compressed string structure intact.
'http://www.jose.it-berater.org/smfforum/index.php?topic=3467.0
#COMPILE EXE '#Win 8.04# Thank to Aslan Babakhanov
#DIM ALL
#OPTION VERSION5 'Not 2000 but XP in fact, to be lazy or not to be lazy...
#INCLUDE "WIN32API.INC"
#INCLUDE "NTSTATUS.INC"
%MaxStringLen = 2147483000
GLOBAL sBuffer AS STRING
DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer"( _
BYVAL CompressionFormat AS LONG, _
BYVAL SourceBuffer AS DWORD PTR, _
BYVAL SourceBufferLength AS LONG, _
BYVAL DestinationBuffer AS DWORD, _
BYVAL DestinationBufferLength AS DWORD PTR, _
BYVAL Unknow AS LONG, _
pDestinationSize AS DWORD, _
BYVAL WorkspaceBuffer AS DWORD PTR) AS LONG
DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize"( _
BYVAL CompressionFormat AS LONG, _
pNeededBufferSize AS LONG, _
pUnknown AS LONG) AS LONG
DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer"( _
BYVAL CompressionFormat AS LONG, _
BYVAL DestinationBuffer AS DWORD PTR, _
BYVAL DestinationBufferLength AS LONG, _
BYVAL SourceBuffer AS DWORD PTR, _
BYVAL SourceBufferLength AS LONG, _
pDestinationSize AS LONG) AS LONG
'______________________________________________________________________________
FUNCTION RTL_DECOMPRESS(BYVAL sCompressedBuffer AS STRING, BYREF sUncompressedBuffer AS STRING) AS LONG
LOCAL FinalUncompressedSize AS DWORD
LOCAL unCompressedBufferSize AS DWORD
LOCAL RetVal AS LONG
unCompressedBufferSize = LEN(sCompressedBuffer) * 5
IF unCompressedBufferSize = 0 THEN 'Nothing to decompress, 0 * 5 = 0
RetVal = %STATUS_NO_DATA_DETECTED '%STATUS_INVALID_PARAMETER
ELSE
DO
sUncompressedBuffer = NUL$(unCompressedBufferSize)
RetVal = RtlDecompressBuffer(%COMPRESSION_FORMAT_LZNT1, _
BYVAL STRPTR(sUncompressedBuffer), _
BYVAL unCompressedBufferSize, _
BYVAL STRPTR(sCompressedBuffer), _
BYVAL LEN(sCompressedBuffer), _
FinalUncompressedSize)
SELECT CASE RetVal
CASE %STATUS_INVALID_PARAMETER
sBuffer = sBuffer & "%STATUS_INVALID_PARAMETER" & $CRLF
CASE %STATUS_UNSUPPORTED_COMPRESSION
sBuffer = sBuffer & "%STATUS_UNSUPPORTED_COMPRESSION" & $CRLF
CASE %STATUS_SUCCESS
sBuffer = sBuffer & "STATUS_SUCCESS" & $CRLF
EXIT LOOP
CASE %STATUS_BAD_COMPRESSION_BUFFER
sBuffer = sBuffer & "%STATUS_BAD_COMPRESSION_BUFFER with " & _
FORMAT$(unCompressedBufferSize, "#,###") & " bytes." & $CRLF & _
"Increasing buffer size and retry..." & $CRLF
'Dynamic strings can contain up to approximately 2 Gb (2^31) characters.
IF unCompressedBufferSize >= %MaxStringLen THEN EXIT LOOP
unCompressedBufferSize = MIN(unCompressedBufferSize * 20, %MaxStringLen)
END SELECT
LOOP
END IF
IF RetVal = %STATUS_SUCCESS THEN
sUncompressedBuffer = LEFT$(sUncompressedBuffer, FinalUncompressedSize)
FUNCTION = FinalUncompressedSize
ELSE
sUncompressedBuffer = ""
FUNCTION = 0
END IF
END FUNCTION
'______________________________________________________________________________
FUNCTION RTL_COMPRESS(BYVAL sUncompressedBuffer AS STRING, BYREF sCompressedBuffer AS STRING) AS LONG
LOCAL RetVal AS LONG
LOCAL CompressionFormatAndEngine AS DWORD
LOCAL unCompressedBufferSize AS DWORD
LOCAL CompressedBufferSize AS DWORD
LOCAL FinalCompressedSize AS DWORD
LOCAL CompressBufferWorkSpaceSize AS DWORD
LOCAL CompressFragmentWorkSpaceSize AS DWORD
LOCAL unCompressedChunkSize AS DWORD
LOCAL CompressBufferWorkSpace AS STRING
CompressionFormatAndEngine = MAK(DWORD, %COMPRESSION_FORMAT_LZNT1, %COMPRESSION_ENGINE_MAXIMUM)
RtlGetCompressionWorkSpaceSize(CompressionFormatAndEngine, _ '%COMPRESSION_ENGINE_STANDARD
CompressBufferWorkSpaceSize, _
CompressFragmentWorkSpaceSize)
CompressBufferWorkSpace = NUL$(CompressBufferWorkSpaceSize)
unCompressedBufferSize = LEN(sUncompressedBuffer)
CompressedBufferSize = unCompressedBufferSize + 3 'See line below
'Add 3 in case output is bigger than input witch appen witth small buffer
sCompressedBuffer = NUL$(CompressedBufferSize)
unCompressedChunkSize = 4096
'Can be 512, 1024, 2048, or 4096. The os uses 4096 and it's also the recommended value.
RetVal = RtlCompressBuffer(BYVAL CompressionFormatAndEngine, _
BYVAL STRPTR(sUncompressedBuffer), _
BYVAL unCompressedBufferSize, _
BYVAL STRPTR(sCompressedBuffer), _
BYVAL CompressedBufferSize, _
BYVAL unCompressedChunkSize, _
FinalCompressedSize, _
BYVAL STRPTR(CompressBufferWorkSpace))
SELECT CASE RetVal
CASE %STATUS_SUCCESS
sBuffer = sBuffer & "STATUS_SUCCESS" & $CRLF
CASE %STATUS_BUFFER_ALL_ZEROS
sBuffer = sBuffer & "%STATUS_BUFFER_ALL_ZEROS" & $CRLF
CASE %STATUS_INVALID_PARAMETER
sBuffer = sBuffer & "%STATUS_INVALID_PARAMETER" & $CRLF
CASE %STATUS_UNSUPPORTED_COMPRESSION
sBuffer = sBuffer & "%STATUS_UNSUPPORTED_COMPRESSION" & $CRLF
CASE %STATUS_NOT_SUPPORTED
sBuffer = sBuffer & "%STATUS_NOT_SUPPORTED" & $CRLF
CASE %STATUS_BUFFER_TOO_SMALL
sBuffer = sBuffer & "STATUS_BUFFER_TOO_SMALL" & STR$(FinalCompressedSize) & $CRLF
END SELECT
sCompressedBuffer = LEFT$(sCompressedBuffer, FinalCompressedSize)
FUNCTION = FinalCompressedSize
END FUNCTION
'______________________________________________________________________________
FUNCTION PBMAIN() AS LONG
LOCAL hFile AS DWORD
LOCAL UnCompressedLen AS DWORD
LOCAL DeCompressedLen AS DWORD
LOCAL CompressedLen AS DWORD
LOCAL sUncompressed AS STRING
LOCAL sCompressed AS STRING
LOCAL sDeCompressed AS STRING
LOCAL zFileName AS ASCIIZ * %Max_Path
zFileName = "C:\Windows\System32\Shell32.DLL" '<-- Change
hFile = FREEFILE : OPEN zFileName FOR BINARY AS hFile : GET$ hFile, LOF(hFile), sUncompressed : CLOSE hFile
UnCompressedLen = LEN(sUncompressed)
sBuffer = zFileName & $CRLF & "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "2 first bytes = " & $DQ & LEFT$(sUncompressed, 2) & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "2 first bytes = " & $DQ & LEFT$(sDeCompressed, 2) & $DQ
MSGBOX sBuffer
sUncompressed = "A"
UnCompressedLen = LEN(sUncompressed)
sBuffer = "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sUncompressed & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes "
sBuffer = sBuffer & "(" & HEX$(ASC(sCompressed, 1), 2) & $SPC & _
HEX$(ASC(sCompressed, 2), 2) & $SPC & _
HEX$(ASC(sCompressed, 3), 2) & $SPC & _
HEX$(ASC(sCompressed, 4), 2) & ")" & $CRLF
sUncompressed = ""
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sDeCompressed & $DQ & $CRLF
MSGBOX sBuffer
sUncompressed = STRING$(150, "B") 'sUncompressed = STRING$(155, 0) 'sUncompressed = ""
UnCompressedLen = LEN(sUncompressed)
sBuffer = "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sUncompressed & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "(" & HEX$(ASC(sCompressed, 1), 2) & $SPC & _
HEX$(ASC(sCompressed, 2), 2) & $SPC & _
HEX$(ASC(sCompressed, 3), 2) & $SPC & _
HEX$(ASC(sCompressed, 4), 2) & $SPC & _
HEX$(ASC(sCompressed, 5), 2) & $SPC & _
HEX$(ASC(sCompressed, 6), 2) & ")" & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sDeCompressed & $DQ & $CRLF
MSGBOX sBuffer
END FUNCTION
'______________________________________________________________________________
'
Pierre,
The header i am using, is part of my encryption protection, and lossless image compression ;)
...