Bluebonnet, light-weight .NET on Java

Bluebonnet is a partial implementation of the .NET platform on top of the Java Virtual Machine, and compatible with Android runtime. The Bluebonnet bytecode compiler translates .NET CIL into Java bytecode in Java classes, and additional run-time support is provided by the Baselib library.

The Bluebonnet platform is compatible with Java on all API levels of the Android platform. This means Android apps can be written in languages such as C# and F#. After conversion, they run as 100% Java code on Android, with no native code libraries. They require only a thin, light-weight support library — the Baselib library.

The Baselib library is itself written in C# and compiled to Java using Bluebonnet. It provides support for .NET features such as types (primitive types, value types, generic types), reflection, collections, and some basic functionality from the .NET Base Class Libraries (BCL).

It should be emphasized that Bluebonnet does not aim to be a full implementation of .NET, and that large parts of the standard .NET libraries are not implemented in Baselib at this time. Having said that, it is possible to create simple Android apps with Bluebonnet. For an example, take a look at the Unjum proof-of-concept game (also on Github).

This game relies on BNA (Github link), which is a partial implementation of the XNA 4 library for Android. The game was developed in F# in Visual Studio on Windows using the original implementation of XNA 4 from Microsoft. Then the game binary was run through Bluebonnet, combined with BNA (as well as with Baselib and the F# core library) and packaged into an APK, ready to run on Android.

For more information, and to download Bluebonnet, please visit the Bluebonnet Github page.

Loulabelle, a Lua to JavaScript compiler

Loulabelle

version 5203, released May 2018

Compiler/transpiler from Lua 5.2 to JavaScript.
Written in Lua 5.2 and can be compiled to JavaScript.

Project home page:

https://www.spaceflint.com/loulabelle

https://www.spaceflint.com/?p=161

Online playground:

https://www.spaceflint.com/loulabelle/playground

Reference manual:

https://www.spaceflint.com/loulabelle/manual.html

Source code download:

https://www.spaceflint.com/loulabelle/loulabelle-5203.zip

GitHub page:

https://github.com/spaceflint7/loulabelle

Using ptrace on OS X

Documentation on the subject of using ptrace in OS X seems somewhat lacking. This text attempts to summarize the steps necessary to trace another process. In OS X, the ptrace service itself offers little more than attaching to and detaching from the target process. Other system calls are necessary to accomplish additional jobs, such as accessing target memory and registers.

task_for_pid

The first of these system calls is task_for_pid, which gets a “task port” for the target process. (Some parts of the OS X kernel use the term “task” for a process.) This port (or handle) is used in other system calls to query or manipulate the target process.


#include <mach/mach.h>

task_for_pid(mach_task_self(),
             target_pid,
             &target_task_port);

Note that the first argument specifies which process receives the new task port, with mach_task_self() indicating the current process, and the third argument points to the variable which actually receives the port.

To release the port, at the end of the ptrace session:


mach_port_deallocate(mach_task_self(),
                     target_task_port);

The task_for_pid system call is protected by the gatekeeper security framework, and will fail (with error code 5, KERN_FAILURE) unless the calling process is digitally signed, and its Info.plist includes the SecTaskAccess key.

An example of a minimal Info.plist:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>SecTaskAccess</key>
    <array>
        <string>allowed</string>
        <string>debug</string>
    </array>
</dict>
</plist>

If desired, this Info.plist can be embedded into the executable by specifying the sectcreate option in the link step:


gcc -sectcreate __TEXT __info_plist Info.plist -o program object1 object2...

Finally, digitally signing the executable is accomplished with codesign(1) utility. This text will not go into the details of using this utility, or generating a self-signed certificate. For more information, please see Mac OS X and Task_for_pid() Mach Call.

Mach exceptions

As part of controlling a process using ptrace, the controlling process expects to be notified about signals in the target process. Using ptrace with the PT_ATTACH request, these notifications are delivered via SIGCHLD signals sent to the controlling process.

However, OS X version 10.2 introduced the PT_ATTACHEXC form of ptrace, which delivers mach exceptions (as opposed to just UNIX signals) via mach messages. And in recent versions of OS X, the old PT_ATTACH form is marked deprecated. This text will therefore focus on the newer PT_ATTACHEXC form.

Mach exception are enumerated in /usr/include/mach/exception_types.h and deal with various faults and traps. For example, EXC_BAD_ACCESS is a memory access fault, and EXC_BREAKPOINT is a software breakpoint trap. One of the exceptions, EXC_SOFTWARE (with code EXC_SOFT_SIGNAL) is used to represent a UNIX signal as a mach exception.

Requesting mach exceptions

To receive mach exceptions, the controlling process must register an “exception port” with the target process, to request notifications on a set of exception types. The controlling process should then listen on the port to receive notifications about exceptions (including UNIX signals) that occur in the target process.

The set of exception types is defined in exception_types.h as EXC_MASK_ALL, but note that previous versions of OS X may not support all exception types, and will fail requests that specify exception types which are not recognized. Note also that LLDB ignores EXC_RESOURCE so it may be advisable to do the same.

It is useful to save the set of exception ports already registered for the target process, before replacing them. This makes it easier to detach the process, as will be discussed in a later section of this document.

Example Code:


/* save the set of exception ports registered in the process */

exception_mask_t       saved_masks[EXC_TYPES_COUNT];
mach_port_t            saved_ports[EXC_TYPES_COUNT];
exception_behavior_t   saved_behaviors[EXC_TYPES_COUNT];
thread_state_flavor_t  saved_flavors[EXC_TYPES_COUNT];
mach_msg_type_number_t saved_exception_types_count;

task_get_exception_ports(target_task_port,
                        EXC_MASK_ALL,
                        saved_masks,
                        &saved_exception_types_count,
                        saved_ports,
                        saved_behaviors
                        saved_flavors);

/* allocate and authorize a new port */

mach_port_allocate(mach_task_self(),
                   MACH_PORT_RIGHT_RECEIVE,
                   &target_exception_port);

mach_port_insert_right(mach_task_self(),
                       target_exception_port,
/* and again */        target_exception_port,
                       MACH_MSG_TYPE_MAKE_SEND);

/* register the exception port with the target process */

task_set_exception_ports(target_task_port,
                         EXC_MASK_ALL,
                         target_exception_port,
                         EXCEPTION_DEFAULT | MACH_EXCEPTION_CODES,
                         THREAD_STATE_NONE);

Attaching to the target process

With an exception port registered on the target process, it is finally possible to attach to the process with ptrace.


/* attach to the target process */

ptrace(PT_ATTACHEXC, target_pid, 0, 0);

As part of the attach sequence, the target process will be signalled with SIGSTOP, and this signal will be delivered to the controlling process as a mach “soft signal” exception, that is, exception type EXC_SOFTWARE with code EXC_SOFT_SIGNAL, and sub-code SIGSTOP.

Receiving mach exceptions

When an exception occurs in some thread in the target process, the kernel pushes an exception message into a queue associated with the exception port. The thread then blocks (in the kernel) while waiting for a reply to the exception message. This wait is indefinite; it does not include a timeout.

If the controlling process is not listening on the exception port at the time when an exception is sent, the exception message will remain in the exception port queue, and the thread which sent the exception will remain blocking while waiting for a reply.

While a thread is waiting for a reply in this way, other threads in the process continue to execute, and can generate exceptions, which are handled in the same way as described above.

This queueing mechanism allows the controlling process to have a main loop in which it waits for an exception message, processes the exception, sends a reply message, and then restarts the loop. Processing the exception may take any length of time.

If the exception is a UNIX signal, the UNIX status of the process is set to stopped, before the thread sends the message. This does not prevent other threads in the process from continuing to execute. However, delivery of additional UNIX signal exceptions will be suspended while the UNIX status of the process remains stopped.


/* wait indefinitely to receive an exception message */

wait_for_exception:

char req[128], rpl[128];			/* request and reply buffers */

mach_msg((mach_msg_header_t *)req,	/* receive buffer */
		 MACH_RCV_MSG,				/* receive message */
		 0, 						/* size of send buffer */
		 sizeof(req),				/* size of receive buffer */
		 target_exception_port,		/* port to receive on */
		 MACH_MSG_TIMEOUT_NONE,		/* wait indefinitely */
		 MACH_PORT_NULL);			/* notify port, unused */

/* suspend all threads in the process after an exception was received */

task_suspend(target_task_port);

Parsing mach exceptions

Functions for handling the exception message are defined in /usr/include/mach/mach_exc.defs and generated using the mig(1) program. The kernel-side calls the generated mach_exception_raise function when sending the exception.

The controlling process, listening on the exception port, is the receiving end, and receives the exception as a message buffer. It then calls mach_exc_server to extract the data from the message buffer and convert these to function arguments, which are used in the call to the catch_mach_exception_raise function.

The catch_mach_exception_raise function is provided by the program code of the controlling process. In other words, the stub generated by mig(1) calls catch_mach_exception_raise.

Note that mach_exc.defs defines exception_raise, exception_raise_state, and exception_raise_state_identity. A flag passed to task_set_exception_ports (EXCEPTION_DEFAULT in the code fragment above) selects which exception function is used.

To generate the mach_exc_server function for the receiving end, mig(1) is invoked:


mig /usr/include/mach/mach_exc.defs

This generates mach_excUser.c, mach_excServer.c, and mach_exc.h. The second file, mach_excServer.c, contains the mach_exc_server function. This file should be compiled into the executable of the controlling process.


if (kret_from_mach_msg == KERN_SUCCESS) {

	/* mach_exc_server calls catch_mach_exception_raise */
	
	boolean_t message_parsed_correctly =
	
				mach_exc_server((mach_msg_header_t *)req,
								(mach_msg_header_t *)rpl);
	
	if (! message_parsed_correctly) {
	
		kret_from_catch_mach_exception_raise =
				((mig_reply_error_t *)rpl)->RetCode;
	}
}

/* resume all threads in the process before replying to the exception */

task_resume(target_task_port);

/* reply to the exception */

mach_msg_size_t send_sz = ((mach_msg_header_t *)rpl)->msgh_size;

mach_msg((mach_msg_header_t *)rpl,	/* send buffer */
		 MACH_SEND_MSG,				/* send message */
		 send_sz,					/* size of send buffer */
		 0,							/* size of receive buffer */
		 MACH_PORT_NULL,			/* port to receive on */
		 MACH_MSG_TIMEOUT_NONE,		/* wait indefinitely */
		 MACH_PORT_NULL);			/* notify port, unused */

/* target process is now running.  wait for a new exception */

goto wait_for_exception;

The last remaining piece is implementing catch_mach_exception_raise,
which is called by the mach_exc_server function after it has parsed the incoming exception message and extracted the message arguments.

As discussed in a preceding section, if the exception is a UNIX soft signal exception, an additional step is required. Before sending the soft signal exception message to the controlling process, the kernel sets the process status to stopped (p_stat = SSTOP).

Similarly, before replying to the message, the controlling process must tell the kernel to reset the process status (p_stat = SRUN). This is accomplished with a call to ptrace with the PT_THUPDATE request, which also modifies the signal number.

By changing the signal number, the controlling process can hide the signal from the target process, by modifying the signal number to zero. For example, for the case of handling SIGSTOP. Or the controlling process can leave the signal to be handled by the target process, or request that some other signal be handled instead.


/* catch_mach_exception_raise */

kern_return_t catch_mach_exception_raise(
    mach_port_t exception_port, 
    mach_port_t thread_port,
    mach_port_t task_port,
    exception_type_t exception_type,
    mach_exception_data_t codes,
    mach_msg_type_number_t num_codes)
{
	/* exception_type is defined in exception_types.h 						*/

	/* an exception may include a code and a sub-code.  num_codes specifies */
	/* the number of enties in the codes argument, between zero and two.    */
	/* codes[0] is the code, codes[1] is the sub-code.                      */

	if (exception_type == EXC_SOFTWARE && code[0] == EXC_SOFT_SIGNAL) {
	
		/* handling UNIX soft signal: 							     		*/
		/* this example clears SIGSTOP before resuming the process.  		*/
	
		if (codes[2] == SIGSTOP)
			codes[2] = 0;

		ptrace(PT_THUPDATE,
			   target_pid,
			   (caddr_t)(uintptr_t)thread_port,
			   codes[2]);
	}
	
	return KERN_SUCCESS;
}

/* catch_mach_exception_raise_state */

kern_return_t catch_mach_exception_raise_state(
    mach_port_t exception_port, 
    exception_type_t exception,
    const mach_exception_data_t code, 
    mach_msg_type_number_t codeCnt,
    int *flavor, 
    const thread_state_t old_state,
    mach_msg_type_number_t old_stateCnt, 
    thread_state_t new_state,
    mach_msg_type_number_t *new_stateCnt)
{
	/* not used because EXCEPTION_STATE is not specified in the call */
	/* to task_set_exception_ports, but referenced by mach_exc_server */
	
    return MACH_RCV_INVALID_TYPE;
}

/* catch_mach_exception_raise_state_identity */

kern_return_t catch_mach_exception_raise_state_identity(
    mach_port_t exception_port, 
    mach_port_t thread, 
    mach_port_t task,
    exception_type_t exception, 
    mach_exception_data_t code,
    mach_msg_type_number_t codeCnt, 
    int *flavor, 
    thread_state_t old_state,
    mach_msg_type_number_t old_stateCnt, 
    thread_state_t new_state,
    mach_msg_type_number_t *new_stateCnt)
{
	/* not used because EXCEPTION_STATE_IDENTITY is not specified in the   */
	/* call to task_set_exception_ports, but referenced by mach_exc_server */
	
    return MACH_RCV_INVALID_TYPE;
}

Detaching target process

As discussed in a preceding section, an exception in the target process causes the thread that triggered the exception to block until the controlling process send a reply message. Detaching from the target process does not release such threads, so all pending exceptions should be handled before detaching.

The saved exception port information should be restored, to prevent new exceptions being sent to the controlling process. However, it is possible the exception port queue contains additional exceptions that have not yet been handled, and must be handled by replying to them.

Detaching from the process can only be requested if the process has a stopped status, as discussed above in the context of UNIX signals. At that point ptrace can be called with request PT_DETACH (or PT_KILL) to detach (or kill) the target process.

Note that PT_THUPDATE resets the stopped status, so it conflicts with PT_DETACH and PT_KILL. If the controlling process wishes to use the stopped status (as a result of an exception) to detach or kill the target, then it should not use PT_THUPDATE in the context of handling that same exception.

The target process may not have a stopped status at the time detaching is requested: The process may be running, or it may be stopped due to an exception that is not a UNIX soft signal exception. In either case, it would be necessary for the controlling process to signal the target process with SIGSTOP, which would generate a UNIX soft signal exception. Then the controlling process continues to handle exceptions, while waiting to receive a UNIX soft signal exception. When it is received, the target process can be detached.

Alternatively, if the target process is to be killed rather than detached, it is possible to handle all pending exceptions, and then send a SIGKILL to the target process, without taking into account whether the target process has a stopped status or not.


/* restore saved exception ports */

for (uint32_t i = 0; i < saved_exception_types_count; ++i) {

	task_set_exception_ports(target_task_port,
							 saved_masks[i],
							 saved_ports[i],
							 saved_behaviors[i],
							 saved_flavors[i]);
}

/* issue PT_DETACH if there is a pending exception */

if (last_exception_type was a UNIX soft signal) {

	if (reply not yet sent for last exception) {
	
		ptrace(PT_DETACH, target_pid, 0, 0);
	}
}

/* process all pending exceptions */

kern_return_t kret_recv = 0;

for (kret_recv == 0) {

	/* reply to last exception */

	mach_msg((mach_msg_header_t *)rpl,	/* send buffer */
			 MACH_SEND_MSG,				/* send message */
			 send_sz,					/* size of send buffer */
			 0,							/* size of receive buffer */
			 MACH_PORT_NULL,			/* port to receive on */
			 MACH_MSG_TIMEOUT_NONE,		/* wait indefinitely */
			 MACH_PORT_NULL);			/* notify port, unused */
	
	/* get next exception, with timeout option.   when the result   */
	/* is timeout, it means the exception port queue was exhausted. */
	/* no new messages will be queued, because the exception port   */
	/* is no longer associated with the target process.				*/

	kret_recv =
	mach_msg((mach_msg_header_t *)req,		/* receive buffer */
		 MACH_RCV_MSG | MACH_RCV_TIMEOUT,	/* receive message */
		 0, 								/* size of send buffer */
		 sizeof(req),						/* size of receive buffer */
		 target_exception_port,				/* port to receive on */
		 1,									/* wait one millisecond */
		 MACH_PORT_NULL);					/* notify port, unused */
}

/* release exception port */

mach_port_deallocate(mach_task_self(), target_exception_port);

/* if detach was not possible, kill the target process */

if (! ptraced_detached_called) {

	kill(target_pid, SIGKILL);
}

Conclusion

This concludes the basic overview of dealing with exception ports in OS X. Please note that some aspects in the example code were simplified for sake of brevity and clarity. As an example, a real debugger would let the user decide how handle an exception, and therefore would not issue ptrace(PT_THUPDATE) in catch_mach_exception_raise immediately after receiving the exception.

As mentioned in the opening paragraph, ptrace on OS X is limited. The reader may wish to look further into thread_get_state and thread_set_state for access to the register state in the target process, and the mach_vm_* functions for access to the virtual memory space of the target process.

References

https://github.com/opensource-apple/xnu

http://web.archive.org/web/20090627062246/http://www.matasano.com/log/1100/what-ive-been-doing-on-my-summer-vacation-or-it-has-to-work-otherwise-gdb-wouldnt/

http://os-tres.net/blog/2010/02/17/mac-os-x-and-task-for-pid-mach-call/

http://uninformed.org/index.cgi?v=4&a=3&p=14

http://unixjunkie.blogspot.co.il/2006/01/darwin-ptrace-and-registers.html

http://stackoverflow.com/questions/2824105/handling-mach-exceptions-in-64bit-os-x-application

KeePassX 2.0 Alpha 6 Plus Additional Fixes From 2014

Download KeePassX binary executable.

KeePassX

Compiled from Git source code, pulled on 28 March 2015.
Includes all fixes in 2014, with latest commit 835c411d12.

DOWNLOAD

https://www.spaceflint.com/KeePassX

HOW TO INSTALL:

1.  Install KeePassX version 2.0 Alpha 6 if you don’t already have it.
2.  Use Finder to open your Applications folder.
3.  Right click on KeePassX.app and select Show Package Contents.
4.  Use Finder to open the Contents folder, then MacOS folder.
5.  Rename original file KeePassX to a differet name, KeePass-orig or similar.
6.  Paste the downloaded KeePassX file into this folder.
7. Make the pasted file executable. Open a Terminal window and paste the command:
chmod +x /Applications/KeePassX.app/Contents/MacOS/KeePassX
8.  Test your installation by starting the KeePassX application.

Absolutely no changes were introduced to the source code that was pulled from Git.

I am not associated with KeePassX, its authors or contributors.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR “AS IS” AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

THE FULL STORY

After switching to Mac from Windows, I was pleased to find KeePassX.  The refering web site recommended version 2.0 Alpha 6, which was great except for not saving column widths across invocations of the program.  A quick search revealed this issue was already fixed in source code, but not yet released as a compiled package.

I know how to compile software from source code, but not yet familiar enough with OSX that I was confident that I can release a complete app package.  But I figured that if I can build KeePassX against the same library dependencies that version 2.0 Alpha 6 was built against, then I could just replace the binary executable and leave everything else as it was.

Those library dependencies were QT 4.8.4, and GNU libgcrypt 1.5.0 and libgpg-error 1.10.  Due to changes in the OSX line, these packages do not build cleanly on OSX Yosemite, and I first had to research various tweaks and fixes to get them to build.

(Please note that these tweaks and fixes were only necessary to build these intermediate libraries on my machine, so I could then compile the new KeePassX with the exact same version numbers as the old KeePassX.  There is no need to replace these libraries in your KeePass application folder.)

Once the dependencies were ready, the KeePassX source code that I pulled from Git compiled quickly and successfully, and did not require any changes.  After copying the newly compiled binary to the KeePass application folder, I was pleased to find that this later version indeed remembers column widths across invocations.

Compiled on OSX Yosemite 10.10.2, Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn) Target: x86_64-apple-darwin14.1.0 Thread model: posix.