Memory management of NXC strcat

[ejvaughan]

How exactly does strcat manage memory in NXC? I'm writing a program to receive packets of data over Bluetooth from a computer, and I'm simply concatenating each new packet that comes in to a single string. The problem is that the program crashes after the string's length exceeds a little over 6000. I saw somewhere else on Mindboards that an NXC program has 32kB of memory available during execution, so I came to the conclusion that strcat must be the source of the problem since the string is not large enough to be using up that amount of memory. So what is the best way to combine the packets of data so that the program does not exceed its memory limitations?

[muntoo]

Yeah, must be strcat(). Watch:

Code: Select all

task main()
{
	string a, b, c;

	c = strcat(a, b);
}

Gives the following NBC output:

Code: Select all

dseg	segment
	a	byte[]
	b	byte[]
	strcat_dest	byte[]
	strcat_src	byte[]
	retval	byte[]
	bufstrcat	byte[]
	strbuf	byte[]
dseg   ends

thread main
	mov strcat_dest, a
	strcat strbuf, b
	mov strcat_src, strbuf
	strcat bufstrcat, strcat_dest, strcat_src
	mov strcat_dest, bufstrcat
	mov a, bufstrcat
	mov retval, bufstrcat
	strcat strbuf, retval
	exit -1, -1
endt

Which translates to:

Code: Select all

byte a[];
byte b[];
byte strcat_dest[];
byte strcat_src[];
byte retval[];
byte bufstrcat[];
byte strbuf[];

task main()
{
	strcat_dest = a;
	strbuf = b;
	strcat_src = strbuf;
	bufstrcat = strcat(strcat_dest, strcat_src);
	strcat_dest = bufstrcat;
	a = bufstrcat;
	retval = bufstrcat;
	strbuf = retval;
}

-----

Here it is with the + operator:

Code: Select all

task main()
{
	string a, b, c;

	c = a + b;
}

NBC:

Code: Select all

dseg	segment
	a	byte[]	
	b	byte[]	
	strcat_dest	byte[]	
	strcat_src	byte[]	
	strbuf	byte[]	
dseg	ends

thread main
	mov strcat_dest, a
	strcat strbuf, b
	mov strcat_src, strbuf
	strcat strbuf, strcat_dest, strcat_src
	exit -1, -1
endt

Which translates to:

Code: Select all

byte a[];
byte b[];
byte strcat_dest[];
byte strcat_src[];
byte strbuf[];

task main()
{
	strcat_dest = a;
	strbuf = b;
	strcat_src = strbuf;
	strbuf = strcat(strcat_dest, strcat_src);
}

[spillerrec]

As shown in muntoos post, you have to be very careful when you are dealing with large strings/arrays because it ends up making temporary variables.
To make matters worse, in NBC there is no such thing as scope, all variables are global/static so even when it goes out of scope in NXC it does not get cleared. And it is "impossible" to clear them, since you don't have access to them. (Well, there is a dirty trick, run the function again but with an empty array, depending on the function you might be able to clear most of them.)

[h-g-t]

I wonder, would writing the input to a file whilst it is coming in be too slow?

[afanofosc]

I would recommend using a simple NBC asm block that uses the strcat opcode to concatenate the new input string to your buffer. You will need one temporary byte array to hold the output of strcat and then copy that into your original buffer. After that you can empty the temporary with arrinit.

John Hansen

[muntoo]

I would recommend using a simple NBC asm block that uses the strcat opcode to concatenate the new input string to your buffer. You will need one temporary byte array to hold the output of strcat and then copy that into your original buffer. After that you can empty the temporary with arrinit.

Wouldn't this work:

Code: Select all

task main()
{
    string a = "abcd";
    string b = "efgh";
    string c;

    asm
    {
        // c = a + b;
        strcat c, a, b
    }
}

Do I really need the extra buffer...?

[afanofosc]

In the OP's case he has something like this:

Code: Select all

task main()
{
  string large_buffer, tmp;
  string new_data;
  // read data from bluetooth
  //append it to large_buffer
  asm {
    strcat tmp, large_buffer, new_data
    mov large_buffer, tmp
    arrinit tmp, 0, 0
  }
}

So he has to have a third buffer to store the results of adding new_data to the end of large_buffer and then he needs to copy that back into a larger version of large_buffer. You can't use large_buffer as both an input and an output in strcat or you will overwrite your source before you copy it to your destination. The problem with strcat is that it is an inline function that takes 2 string input arguments and returns a string so you have the 2 variables you pass in, the 2 local variables that get a copy of your inputs, and the local return value variable and then a string buffer temporary that holds the return value of all string functions before the value is assigned to the LHS of the string expression. If I had a better optimizer these temporaries/locals would all be removed. If I were really clever and ambitious I could add compiler code that empties out any local arrays/strings when they go out of scope or are "released" by the compiler.

John Hansen

[muntoo]

You can't use large_buffer as both an input and an output in strcat or you will overwrite your source before you copy it to your destination.

Would arrbuild work, then? I use this in NXC:

Code: Select all

#define push(out,val) ArrayBuild(out, out, val);

Or does it have the same "problems"?

OT: Say, have you read the 'Dragon Book'? Apparently, it's the must read book for compiler writers. (It's on my to-read list.)

[afanofosc]

arrbuild is the same as strcat except that one is designed to work with null terminated byte arrays (strcat) while the other is designed to work with any old kind of array. They both have the same limitation that you can't use any of the inputs as the output or it will be overwritten (actually reallocated to a different memory location) before you copy the input to the output.

John Hansen

[ejvaughan]

In the OP's case he has something like this:

Code: Select all

task main()
{
  string large_buffer, tmp;
  string new_data;
  // read data from bluetooth
  //append it to large_buffer
  asm {
    strcat tmp, large_buffer, new_data
    mov large_buffer, tmp
    arrinit tmp, 0, 0
  }
}

So he has to have a third buffer to store the results of adding new_data to the end of large_buffer and then he needs to copy that back into a larger version of large_buffer. You can't use large_buffer as both an input and an output in strcat or you will overwrite your source before you copy it to your destination. The problem with strcat is that it is an inline function that takes 2 string input arguments and returns a string so you have the 2 variables you pass in, the 2 local variables that get a copy of your inputs, and the local return value variable and then a string buffer temporary that holds the return value of all string functions before the value is assigned to the LHS of the string expression. If I had a better optimizer these temporaries/locals would all be removed. If I were really clever and ambitious I could add compiler code that empties out any local arrays/strings when they go out of scope or are "released" by the compiler.

John Hansen

Hmm. I tried your code and it ends up freezing my NXT. I have no clue why. Here is my code:

Code: Select all

task main ()
{
	string URL = "http://rss.cnn.com/rss/cnn_topstories.rss";
	string proceed;

	int URLlength = StrLen(URL);
	string URLlengthAsString = NumToStr(URLlength);
	SendResponseString(0, URLlengthAsString);

	for (int idx = 0; idx < URLlength; idx += 58) {
		string msg = SubStr(URL, idx, 58);
		SendResponseString(0, msg)
		while (ReceiveMessage(0, true, proceed) != LDR_SUCCESS);
	}
	
	string sizeAsString;

	while(ReceiveMessage(0, true, sizeAsString) != LDR_SUCCESS);
	
	long size = StrToNum(sizeAsString);
	NumOut(0, LCD_LINE1, size);

	long numBytesWritten = 0;

	string message, packet, tmp;

	while(numBytesWritten < size) {
		while(ReceiveMessage(0, true, packet) != LDR_SUCCESS);
		asm {
			strcat tmp, message, packet
			mov message, tmp
			arrinit tmp, 0, 0
		}
		numBytesWritten += StrLen(packet);
	}

	NumOut(0, LCD_LINE4, StrLen(message), true);
}