string size limit

[mattallen37]

I am having huge amounts of trouble getting strings of 2000 or so characters to work properly. Normally it will just lock the NXT, but sometimes the programs exits with error -5 which means "Insufficient memory available". I am trying to write a single-line file with a string of 2110+ characters, but things are acting up big time.

This works relatively well (it doesn't crash at least):

Code: Select all

void WriteFileSingle(string file_name, string message)
{
  DeleteFile(file_name);
  byte file_handle;
  unsigned long file_size = strlen(message);
  CreateFile(file_name, file_size, file_handle);
  unsigned long amount_written;
  WriteString(file_handle, message, amount_written);
  CloseFile(file_handle);
}

task main(){
  string Message;
  
  for(int i = 0; i<100; i++){
    Message = Message + NumToStr(i) + " ";
  }
  string name = "Test file.txt";
  WriteFileSingle(name, Message);
}

However, it locks up the NXT if you make it loop for 500.

How should I write files that are 2 to 8 kB long?

Edit: I have recently fragmented the file system of the NXT, and the NXT has 95k + of flash available.

[afanofosc]

The problem is that you are using up all the available memory in your for loop (I'm guessing). But I can see no reason whatsoever for gathering multiple bits of data into a single string before writing it to a file. Is there any particular reason why you would want to do something like that? You can easily write to a "single line" using multiple write operations.

If you reveal what your intent is I can tell you how to do it right and as optimally as it can be done with the NXT.

John Hansen

[mattallen37]

Then is there a way to release variable space for variables I am done with? There is no reason I should be using up anywhere near that much RAM in the for loop. Maybe a couple temporary strings, and a 2 byte variable... at very most 2500 bytes (assuming it used a temporary string for the additions). Comparing that to the 32k of RAM, it doesn't seem like all that much IMO.

I am working on a library that would allow you to create monochrome .bmp files on the NXT. I need a way that the NXT can have much more than a resolution of 100 x 64, but I only need it to be a single-shot deal. With some rough work-around I have been able to create 128x128 BMP files, but I don't want to have to do weird things, and I would like to be able to have at least 256x256, but ideally 400x400 or more. A 128x128 file is 2110 (2048 bit-map + 62 header and color table) bytes.

I currently have a function that returns a string of the data needed for making a BMP file (both of the headers, color map, and bit-map). I then feed that string into the file write function to create the file. At 128x128, the bit-map area is a 2048 element byte array. Unless I mix the file writing in with the BMP creator function, or I piece out the data at write time, there is no way to decrease the amount of data written at a time.

Is the problem with the writing? or is it the fact that I have a string with 2110 characters?

Do I need to re-design the file write function to tear apart the string, and only write several hundred characters at a time?

Edit: and in addition to my first question about releasing memory space (RAM), I already ArrayInit all unused user arrays to 0,1 and I set all unused user strings to "". unused meaning I am done with them.

[muntoo]

Using the first post method might actually be slower than simply writing data to a preallocated file one number at a time. (write 41, write 42, write 43, etc) This is because each time you add text to Message, it must be reallocated. That's a hundred unnecessary allocations of large amounts of data, and (loosely speaking) something equivalent to O(n^2).

If you want to lessen the number of writes and allocations, try writing ~10 numbers to a buffer and then flushing it.

Code: Select all

for(t = 0; t < 100; t += 10)
{
    buffer = "";

    for(o = 0; o < 10; ++o)
    {
        buffer += NumToStr(t+o) + " ";
    }

    Write(buffer);
}

I don't know if this method is any faster or slower, but it should lessen memory usage and writes.

[muntoo]

Then is there a way to release variable space for variables I am done with?

The NBC code uses temporary variables, but IIRC, it doesn't clean them up after use. The only way to guarantee no extra memory is being used is to write the critical parts of your code in NBC.

I suppose we could feature request for a #ifdef RELEASE_TEMP that tells the compiler to release the memory occupied by the temporary variables?

I've run into this problem before, IIRC. I wasn't able to determine how all that much memory was being used up.

[mattallen37]

The for loop program was just an example to demonstrate the problem.

[afanofosc]

If you #import a bitmap into a byte array and then successfully write it to a file it proves the problem is not in the size of the array/string.

You absolutely should not use the string type for a binary format like a bitmap. And don't call WriteString when you are really writing a byte array containing binary data. Just use Write or WriteBytes or WriteBytesEx.

Could you please just post your code. I don't think there's any reason to keep it a secret. Then all the experts can contribute to make it the best possible code.

John Hansen

[mattallen37]

Why not use a string for binary data? I need to merge pieces together, and being able to do things like string x = y + z is so much easier than resizing an array, and copying in the data. I haven't ever tried returning an array from a function, but I know I can return a string. Also, I didn't realize that there was the possibility of writing anything but a string to a file.

Sorry, I didn't mean to be keeping the code a secret. It's just that normally when someone posts a large program with #include s, you ask for an example program that demonstrates the problem.

Here is the main program:

Code: Select all

#include "File BMP Mono lib.nxc"
#include "File.nxc"

#define Width 128
#define Hight 128
#define Size  2048

#define uint32_t unsigned long
#define AdvSize 1.40625           // convert 0-255 to 0-359

uint32_t gray_encode(uint32_t b)
{
    return b ^ (b >> 1);
}

uint32_t gray_decode(uint32_t g)
{
    for (uint32_t bit = 1 << 31; bit > 1; bit >>= 1)
    {
        if (g & bit) g ^= bit >> 1;
    }
    return g;
}

void PointOutBMP(unsigned int x, unsigned int y, unsigned int width, unsigned int hight, byte & data[], unsigned int options = DRAW_OPT_NORMAL){
  byte Bytes_Per_Line = (width + 7) / 8;     // Get the minimum number of bytes used.
  while(Bytes_Per_Line%4){                   // While it isn't a multiple of 4 ...
    Bytes_Per_Line ++;                       // ... keep adding to it
  }
  unsigned int Y_Offset = y * Bytes_Per_Line;
  unsigned int X_Offset = x / 8;
  unsigned int data_pointer = Y_Offset + X_Offset;
  byte mask = 0x80 >> (x % 8);
  switch (options){
    case DRAW_OPT_NORMAL:
      data[data_pointer] &= (~mask);
    break;
    case DRAW_OPT_CLEAR:
      data[data_pointer] |= (0xFF&mask);
    break;
  }
}

void ClearScreenBMP(unsigned int size, byte & data[]){
  ArrayInit(data, 0xFF, size);
}

task main(){

  byte BMP[];
  ClearScreenBMP(Size, BMP);

  for(float i=0; i<10; i+=0.5){
    byte gray = gray_encode(i);
    for(int ii=0; ii<8; ii++){
      if((gray >> ii) & 1){
        PointOutBMP((ii*3+43)*sind(i*AdvSize) + 64, (ii*3+43)*cosd(i*AdvSize) + 64, Width, Hight, BMP);
      }
    }
  }

  string BMP_String = CreateMonochromeBMP(BMP, Width, Hight);
  ArrayInit(BMP, 0, 1);
  NumOut(0, LCD_LINE1, strlen(BMP_String));
  PlayTone(3500,100);
  Wait(100);

  until(ButtonPressed(BTNCENTER, false));
  until(!ButtonPressed(BTNCENTER, false));

  WriteFileSingle("test image.bmp", BMP_String);

  PlayTone(5000,100);
  Wait(100);
  until(ButtonPressed(BTNCENTER, false));
  until(!ButtonPressed(BTNCENTER, false));
}

and "File BMP Mono lib.nxc":

Code: Select all

string CreateBMP_Header(unsigned long TotalSize, unsigned long OffsetToBitmap){
  byte data[14] = {
    0x42, 0x4D,
    0x00, 0x00, 0x00, 0x00,  // will get replaced
    0x00, 0x00,
    0x00, 0x00,
    0x00, 0x00, 0x00, 0x00   // will get replaced
  };

  data[2] = TotalSize;
  data[3] = TotalSize >> 8;
  data[4] = TotalSize >> 16;
  data[5] = TotalSize >> 24;

  data[10] = OffsetToBitmap;
  data[11] = OffsetToBitmap >> 8;
  data[12] = OffsetToBitmap >> 16;
  data[13] = OffsetToBitmap >> 24;

  return FlattenVar(data);
}

string CreateDIB_Header(
  unsigned long DIB_Len,
  unsigned long Width,
  unsigned long Hight,
  unsigned int Color_Planes,
  unsigned int Bits_Per_Pixel,
  unsigned long Compression,
  unsigned long PixelArrSize,
  unsigned long HorzRes,
  unsigned long VertRes
  unsigned long Colors_In_Palette,
  unsigned long Important_colors){

  byte data[40];

  data[0]  = DIB_Len;
  data[1]  = DIB_Len >> 8;
  data[2]  = DIB_Len >> 16;
  data[3]  = DIB_Len >> 24;

  data[4]  = Width;
  data[5]  = Width >> 8;
  data[6]  = Width >> 16;
  data[7]  = Width >> 24;

  data[8]  = Hight;
  data[9]  = Hight >> 8;
  data[10] = Hight >> 16;
  data[11] = Hight >> 24;

  data[12] = Color_Planes;
  data[13] = Color_Planes >> 8;

  data[14] = Bits_Per_Pixel;
  data[15] = Bits_Per_Pixel >> 8;

  data[16] = Compression;
  data[17] = Compression >> 8;
  data[18] = Compression >> 16;
  data[19] = Compression >> 24;

  data[20] = PixelArrSize;
  data[21] = PixelArrSize >> 8;
  data[22] = PixelArrSize >> 16;
  data[23] = PixelArrSize >> 24;

  data[24] = HorzRes;
  data[25] = HorzRes >> 8;
  data[26] = HorzRes >> 16;
  data[27] = HorzRes >> 24;

  data[28] = VertRes;
  data[29] = VertRes >> 8;
  data[30] = VertRes >> 16;
  data[31] = VertRes >> 24;

  data[32] = Colors_In_Palette;
  data[33] = Colors_In_Palette >> 8;
  data[34] = Colors_In_Palette >> 16;
  data[35] = Colors_In_Palette >> 24;

  data[36] = Important_colors;
  data[37] = Important_colors >> 8;
  data[38] = Important_colors >> 16;
  data[39] = Important_colors >> 24;

  string Sdata = FlattenVar(data);
  return Sdata;
}

string CreateMonochromeColorTable(){
  byte data[8] = {
    0x00, 0x00, 0x00, 0x00,
    0xFF, 0xFF, 0xFF, 0x00
  };
  return FlattenVar(data);
}

string CreateMonochromeBMP(byte bitmap_data[], unsigned long width, unsigned long hight){
  unsigned long length = ArrayLen(bitmap_data) + 62;
  string BuildingResult = "";
  BuildingResult += CreateBMP_Header(length, 62);
  BuildingResult += CreateDIB_Header(40, width, hight, 1, 1, 0, 0, 2835, 2835, 0, 0);
  BuildingResult += CreateMonochromeColorTable();
  BuildingResult += FlattenVar(bitmap_data);
  Wait(10);
  string Result = BuildingResult;
  BuildingResult = "";
  Wait(10);
  return Result;
}

and "File.nxc":

Code: Select all

void WriteFileSingle(string file_name, string message)
{
  DeleteFile(file_name);
  byte file_handle;
  unsigned long file_size = strlen(message);
  CreateFile(file_name, file_size, file_handle);
  unsigned long amount_written;
  WriteString(file_handle, message, amount_written);
  CloseFile(file_handle);
}

void ReadFileSingle(string file_name, string & message)
{
  byte File_Handle;
  int File_Size;
  bool Success = false;

  if(OpenFileRead(file_name, File_Size, File_Handle) == NO_ERR)
  {
    Success = ReadLnString(File_Handle, message);
  }
  CloseFile(File_Handle);
}
/*
task main(){
  string Message;
  
  for(int i = 0; i<500; i++){
    Message = Message + NumToStr(i) + " ";
  }
  string name = "Test file.txt";
  WriteFileSingle(name, Message);
}

It's supposed to create a BMP image file on the NXT file system. If it creates it successfully, you can see what it's supposed to look like (or look on FaceBook for a similar image I posted earlier).

[afanofosc]

Matt,

I have revised your code a little. I hope you like it.

bmptest.nxc

(1.2 KiB) Downloaded 932 times

bmp_lib.nxc

(5 KiB) Downloaded 993 times

The problem with the original code is that there is a lot of copying going on and it does not take much copying at 8k (256x256/8) per copy to reach 32k which is the total memory available for all variables in any program running on the NXT virtual machine. The above code does zero copying of the bitmap pixel data. Because of that it can work with bitmaps approaching 512x512 (which would use all 32k of memory so you can't get that large). The idea is that anyone who wants to work with a bitmap has to first acquire a handle to a bitmap. In this case there is only one bitmap and only one handle to give out so whoever acquires a bitmap handle gets to use it until they release the handle, at which time somebody else can acquire the bitmap handle.

If you have any questions about this don't hesitate to ask.

John Hansen

[mattallen37]

Thanks a lot!

I must say, I am impressed by it! You obviously did some research on the BMP file format

I have waded through just about all of it already. I do have a question though: Why do put NULL terminators on __bmp_magic and __bmp_bw_color_table? They should be 2 and 8 bytes, not 3 and 9.

Also, what does the "safecall" keyword mean? Does it mean the the VM will not go on to the next task until that function is complete?