There are many Near Field Communication (NFC)-enabled phones (devices) available on the consumer market. LG, Huawei, Motorola, Samsung, HTC, Nokia, ZTE, Sony, RIM, Amazon, and Apple are among the companies that manufacture and sell mobile phones with NFC technology.
There are many common denominators at the hardware and software levels among all the phones that have NFC technology. First, all mobile phones contain a main (application) processor, and if the phone is manufactured by one of the companies listed above, the application processor is an ARM microprocessor. There may be one or two exceptions, but if 150 current phones were sampled across all of these manufacturers, one would find that 99% of the phones contain ARM technology. Operating systems that run on the ARM Powered processors contained within phones from these manufacturers are similar in that they support preemptive multitasking. Most of these devices run the Linux operating system or a UNIX-based operating system.
An ARM Powered smartphone with NFC technology contains other microprocessors on its printed circuit board (PCB). These other microprocessors consist of the radio frequency (RF) controller or contactless front end (CLF) and an optional embedded secure element (eSE). The CLF is responsible for managing contactless communication at 13.56MHz. The eSE and SE (external UICC-based or microSD card-based) are responsible for secure application execution and the secure storage of sensitive payment or identification information. A trusted OS runs on the eSE and/or SE. A mobile phone may contain both an eSE and a UICC-based or microSD-based SE. For such a configuration where multiple secure elements exist on the phone, different trusted operating systems may run on the phone's secure elements, and different trusted applications (client applications) may run on the trusted operating systems. One trusted application may be responsible for secure VISA payments, and another trusted application may be responsible for electronic identification (eIdentification).
The RF controller and the embedded secure element (eSE) are soldered directly to the PCB, typically within the same IC package. The RF controller may exist on the circuit board without the eSE. The eSE is an optional hardware component on a mobile phone that contains NFC technology. The RF controller and eSE are soldered to the PCB by the phone manufacturer. Alternatively, the secure element used for NFC may be external. An external secure element is one that is located on the mobile phone's UICC or microSD card. When a mobile phone is purchased from a local cell phone store, a UICC card is usually inserted into the phone by the person at the store. If purchased from one of the leading operators in the United States, the UICC card contains a secure element. It is important to note that the existence of an embedded secure element and a UICC-based or microSD-based secure element on the same mobile phone is not mutually exclusive. The secure elements may serve different purposes.
As mentioned above, ARM Powered smartphones with NFC technology may contain a single type of secure element. Such a phone may only contain a CLF or RF controller. Mobile phones with a CLF and no secure element may emulate the software running on the secure element via the operating system and application software running on the application processor. This is called host card emulation (HCE).
There are numerous configurations of NFC technology on mobile phones. Each of the described components exposes one or more hardware interfaces for communication and interconnection with neighboring components on the printed circuit board. Hardware bus protocols and software communication protocols run on top of the wires that connect the individual components on the printed circuit board. These include standard bus protocols such as I²C, S2C (NFC-WI), SWP, and SPI (more on this later). It is important to note that a typical ARM-powered smartphone with NFC technology has a CLF that is connected to the application processor via both I²C and UART. This is good because I²C device drivers are not complicated, and reading/writing to a tty on an ARM-powered smartphone is straightforward. Lastly, the CLF typically contains an optimized 80C51 processor. The CLF may be packaged with a secure element, and the secure element may contain an optimized 8051 core or newer RISC-based core. The secure element will also contain dedicated crypto co-processors for performing asymmetric or symmetric cryptographic operations (i.e., FameX). Custom firmware runs on the CLF, and it can usually be updated via the host operating system over UART. 80C51 firmware for the CLF is usually C code (more on this later). A trusted OS (e.g., JCOP) and trusted applications (applets) run on the secure element. Lastly, if the CLF and secure element are contained within a single IC soldered onto the PCB, they are usually connected via S2C (NFC-WI) internally. As a side note, ARM designs and manufactures ARM Keil development boards. Arm Keil and Ashling are two of the most prevalent companies that sell compilers and development toolkits for embedded microprocessors such as the 80C51. Ashling sells an excellent series of in-circuit emulators (ICE) for debugging SmartMX and SmartMX2 microcontrollers.
Saturday, December 13, 2014
Tuesday, December 9, 2014
The ARM Cortex-A9 Processor - Real World Uses
The ARM Cortex-A processor is commonly used in compute-intensive applications. These processors are capable of running full or rich operating systems such as Linux or UNIX. In the previous blog post, I mentioned the ARM Cortex-A9 based MPCore. The ARM Cortex-A9 processor is highly configurable and is implemented by Freescale Semiconductor in their i.MX 6 processor. The i.MX 6 processor is available in a variety of configurations, including lite, single, dual, dual-lite, and quad-core, and is used in critical applications across multiple industries, including aerospace, medical, and industrial. i.MX processors can be found in devices such as Medical-CT scanners, ultrasound machines, automotive telematics systems, e-readers, and more. Due to their power efficiency, i.MX 6 processors are also suitable for wearables such as glasses and watches.
One example of a device powered by the i.MX 6 is the OrCam, a clip-on wearable designed for users with vision impairments. The OrCam clips onto a pair of eyeglasses and scans the user's field of vision, detecting objects and providing aural feedback via a bone conduction earpiece.
Another device that uses the i.MX 6 processor is the Kindle Fire HD 6 tablet. It's worth keeping an eye out for other wearables, watches, and glasses that use the i.MX 6 processor.
In terms of development, the i.MX 6 is an ideal choice. Although the manuals and data sheets for the processor are quite lengthy, the one-time programmable memory locations or eFuses on the i.MX 6 are highly configurable. eFuses are standard in higher-end microprocessors and can be read or blown (one-time write to an on-chip memory location) using u-boot or the Freescale manufacturing tool. They can also be pulled via the Linux sysfs interface. Typically, there are over 100 eFuses on higher-end microprocessors, and they store important values such as the MAC address, boot configuration parameters, high-assurance boot (HAB) configuration parameters, and master key. U-boot conveniently supports reading and writing (blowing) the eFuses on many different microprocessors. U-boot is open source and supports one-time programmable memory (eFuses or OTP) on ARM-based processors. The U-Boot bootloader also supports a large number of chipsets, including ARM, AVR32, Blackfin, MicroBlaze, MIPS, PPC, and x86.
One example of a device powered by the i.MX 6 is the OrCam, a clip-on wearable designed for users with vision impairments. The OrCam clips onto a pair of eyeglasses and scans the user's field of vision, detecting objects and providing aural feedback via a bone conduction earpiece.
Another device that uses the i.MX 6 processor is the Kindle Fire HD 6 tablet. It's worth keeping an eye out for other wearables, watches, and glasses that use the i.MX 6 processor.
In terms of development, the i.MX 6 is an ideal choice. Although the manuals and data sheets for the processor are quite lengthy, the one-time programmable memory locations or eFuses on the i.MX 6 are highly configurable. eFuses are standard in higher-end microprocessors and can be read or blown (one-time write to an on-chip memory location) using u-boot or the Freescale manufacturing tool. They can also be pulled via the Linux sysfs interface. Typically, there are over 100 eFuses on higher-end microprocessors, and they store important values such as the MAC address, boot configuration parameters, high-assurance boot (HAB) configuration parameters, and master key. U-boot conveniently supports reading and writing (blowing) the eFuses on many different microprocessors. U-boot is open source and supports one-time programmable memory (eFuses or OTP) on ARM-based processors. The U-Boot bootloader also supports a large number of chipsets, including ARM, AVR32, Blackfin, MicroBlaze, MIPS, PPC, and x86.
By
Bryan R. Hinton
Monday, December 8, 2014
ARM TrustZone technology - from Monitor Mode to Dedicated Security Co-Processing and the Secure Element(s)
"A design that places sensitive resources in the Secure world and implements robust software running on the secure processor cores can protect assets against many possible attacks, including those that are normally difficult to secure, such as passwords entered using a keyboard or touch-screen. By separating security-sensitive peripherals through hardware, a designer can limit the number of sub-systems that need to go through security evaluation and therefore save costs when submitting a device for security certification." - ARM.com
NOTE: There are variations in how software is implemented in the secure world - from a simple synchronous library of code to a full-blown operating system.
The execution of the normal OS and secure OS is interleaved over time via a context-switching mechanism called monitor mode. Monitor mode is responsible for time-slicing the execution of the normal OS and secure OS via context switching the state of each world on the physical processor. Monitor mode is explicitly triggered via a dedicated instruction or special type of exception. The explicit methods by which monitor mode is triggered contrast the typical scheduling algorithms that trigger context switching in modern-day preemptive operating systems.
There are varying levels of complexity regarding how the physical hardware in which the secure world runs is designed. These range from both worlds running on the same physical processor core to the secure world running on a completely separate processor core.
Another type of physical hardware design entails an additional microprocessor that is separate from the main processor. The secure world software stack (secure OS and secure applications) runs on a dedicated co-processor. This design is not exclusive to a secure OS running on the main ARM processor. The normal OS still runs on the main ARM processor, and a secure OS can still run on the main ARM processor if the main ARM processor has ARM TrustZone technology. A different secure OS and secure application software can run on the dedicated co-processor.
Client applications running on a secure OS can communicate with the main ARM processor via a set of APIs and commands. There are certain benefits to the secure OS always running on a dedicated security processor core or co-processor.
The operating system that runs on the co-processor can be optimized for just the co-processor. There are many types of dedicated co-processors. The ARM SecurCore microprocessor is one type of dedicated co-processor. ARM SecurCore microprocessors are used in systems that require dedicated processors for security-sensitive applications such as SIM cards, e-Government, Banking, and Identification. Designs that incorporate ARM SecurCore microprocessors can realize multiple key benefits including build performance improvements, energy efficiency, and physical security. Designing and building an operating system for a single chip means that the operating system can be built to use all of the features and only those features that the chip provides.
In summary, here are the key points:
NOTE: There are variations in how software is implemented in the secure world - from a simple synchronous library of code to a full-blown operating system.
The execution of the normal OS and secure OS is interleaved over time via a context-switching mechanism called monitor mode. Monitor mode is responsible for time-slicing the execution of the normal OS and secure OS via context switching the state of each world on the physical processor. Monitor mode is explicitly triggered via a dedicated instruction or special type of exception. The explicit methods by which monitor mode is triggered contrast the typical scheduling algorithms that trigger context switching in modern-day preemptive operating systems.
There are varying levels of complexity regarding how the physical hardware in which the secure world runs is designed. These range from both worlds running on the same physical processor core to the secure world running on a completely separate processor core.
Another type of physical hardware design entails an additional microprocessor that is separate from the main processor. The secure world software stack (secure OS and secure applications) runs on a dedicated co-processor. This design is not exclusive to a secure OS running on the main ARM processor. The normal OS still runs on the main ARM processor, and a secure OS can still run on the main ARM processor if the main ARM processor has ARM TrustZone technology. A different secure OS and secure application software can run on the dedicated co-processor.
Client applications running on a secure OS can communicate with the main ARM processor via a set of APIs and commands. There are certain benefits to the secure OS always running on a dedicated security processor core or co-processor.
The operating system that runs on the co-processor can be optimized for just the co-processor. There are many types of dedicated co-processors. The ARM SecurCore microprocessor is one type of dedicated co-processor. ARM SecurCore microprocessors are used in systems that require dedicated processors for security-sensitive applications such as SIM cards, e-Government, Banking, and Identification. Designs that incorporate ARM SecurCore microprocessors can realize multiple key benefits including build performance improvements, energy efficiency, and physical security. Designing and building an operating system for a single chip means that the operating system can be built to use all of the features and only those features that the chip provides.
In summary, here are the key points:
- Operating system software runs on the main ARM processor or application processor. Software applications run on the application processor. ARM processors in ARM-based mobile phones may or may not have TrustZone processor security technology. If the ARM processor has TrustZone processor security technology, then it may or may not be used.
- There are additional processors on mobile phones that act as dedicated security co-processors. These include the secure element on the UICC or SIM card (UICC-based SE) and the secure element that has been soldered on the printed circuit board. The secure element that has been soldered to the printed circuit board is called the embedded SE. The iPhone 6, iPhone 6 Plus, Samsung S5, Galaxy Nexus, Nexus S, Nexus 7, Sony Xperia series, and a host of others contain a secure element soldered onto the printed circuit board. If the phone has a secure element that has been soldered onto the printed circuit board, then it is most likely contained within the packaging of a larger SoC that also contains the Near Field Communication (NFC) Radio-Frequency (RF) controller. Last but not least, it is entirely possible that the phone contains a secure element on the microSD card.
- The embedded SE and UICC-based SE run a trusted OS. Trusted applications run on top of the trusted OS. In contrast to the trusted OS that runs within the secure world on a main ARM processor with ARM TrustZone processor security technology, the trusted OS that runs on the embedded SE and UICC-based SE does not share full hardware peripheral or direct normal world software access on the main application processor.
- The UICC-based SE and embedded SE are protected by cryptographic keys. Client software applications running on the trusted OS in a processor with ARM TrustZone architecture security extensions are also protected by cryptographic keys.
- There are multiple standards bodies that have established APIs, architecture documents, design documents, and so forth for the trusted operating system and trusted applications that run on the UICC-based SE and embedded SE. These entities are also responsible for the hardware interface on the physical secure element.
By
Bryan R. Hinton
ARM TrustZone technology - a Few Good Boards
ARM provides the Juno ARM Development Platform, a reference platform for software and hardware developers building systems based on ARM Cortex-A processors. This platform contains a board that houses an ARM Cortex-A57 processor and the ARM Cortex-A53 MPCore processor. Both processors are 64-bit and implement the ARMv8-A instruction set architecture (ISA). Developers can build a board support package for this board using OpenEmbedded/Yocto.
The Apple A7 and Apple A8 chips found in the iPhone 5c, iPhone 5s, iPhone 6, and iPhone 6 Plus are based on the ARM Cortex-A53 and the ARM Cortex-A57. Additionally, the Samsung Exynos 5433 Octa SoC contains the ARM Cortex-A57 and the ARM Cortex-A53 MPCore, while the Samsung Galaxy Note 4 has an 8-core Exynos 5433 processor.
The Nvidia TK1 development board, which has a quad-core ARM Cortex-A15 processor, is currently available for purchase. However, the latest board from Nvidia, known as "Denver", which is rumored to contain the ARM Cortex-A57 and the ARM Cortex-A53 MPCore, is not yet available for purchase.
The Freescale I.MX 6 processor has been widely adopted across various industries for a range of embedded products. Freescale offers the SABRE board for intelligent devices, a development board that features the I.MX 6 Quad-Core ARM Cortex-A9 processor. Boundary Devices also sells their variation of this board with the same ARM Cortex-A9 MPCore. Developers can build a board support package for both the Freescale SABRE board and the Boundary Devices board using OpenEmbedded/Yocto.
When working with ARM development boards, it is important to take into account a few critical features. Specifically, e-fuses should not be blown out of the box and should be left open. The fuses can be blown to fit a specific configuration.
Here is a quick overview of the processors and boards.
The Apple A7 and Apple A8 chips found in the iPhone 5c, iPhone 5s, iPhone 6, and iPhone 6 Plus are based on the ARM Cortex-A53 and the ARM Cortex-A57. Additionally, the Samsung Exynos 5433 Octa SoC contains the ARM Cortex-A57 and the ARM Cortex-A53 MPCore, while the Samsung Galaxy Note 4 has an 8-core Exynos 5433 processor.
The Nvidia TK1 development board, which has a quad-core ARM Cortex-A15 processor, is currently available for purchase. However, the latest board from Nvidia, known as "Denver", which is rumored to contain the ARM Cortex-A57 and the ARM Cortex-A53 MPCore, is not yet available for purchase.
The Freescale I.MX 6 processor has been widely adopted across various industries for a range of embedded products. Freescale offers the SABRE board for intelligent devices, a development board that features the I.MX 6 Quad-Core ARM Cortex-A9 processor. Boundary Devices also sells their variation of this board with the same ARM Cortex-A9 MPCore. Developers can build a board support package for both the Freescale SABRE board and the Boundary Devices board using OpenEmbedded/Yocto.
When working with ARM development boards, it is important to take into account a few critical features. Specifically, e-fuses should not be blown out of the box and should be left open. The fuses can be blown to fit a specific configuration.
Here is a quick overview of the processors and boards.
Processor | Manuf | ISA | Dev Board | TrustZone |
|---|---|---|---|---|
| ARM Cortex-A57 and ARM Cortex-A53 MPCore | ARM | ARMv8-A | Juno Ref Platform | Yes |
| ARM Cortex-A15 | Nvidia | ARMv7 | TK1 | Yes |
| ARM Cortex-A15 | Samsung | ARMv7 | Arndale Exynos 5420 | Yes |
| ARM Cortex-A9 MPCore | Freescale | ARMv7 | Freescale SABRE | Yes |
| ARM Cortex-A9 MPCore | Freescale | ARMv7 | Boundary Devices | Yes |
| ARM Cortex-A9 MPCore + Zync 7000 FPGA | Xilinx | ARMv7 | Zed Board | Yes |
| ARM Cortex-A9 MPCore + Zync 7000 FPGA | Xilinx | ARMv7 | Digilent | Yes |
By
Bryan R. Hinton
Saturday, November 22, 2014
C++ - Generative Programming
C++ IOStreams are a powerful mechanism for transforming input into output. Most programmers are at least familiar with C++ IOStreams in the context of reading and writing bytes to a terminal or file.
When a file or terminal is opened for reading or writing by a process, the operating system returns a numerical identifier to the process. This numerical identifier is known as a file descriptor. In turn, the file or terminal can be written to by the process via this file descriptor. The read and write system calls, which are implemented as wrappers in libc, are passed this numerical file descriptor.
Many layers of abstraction reside on top of the read and write system calls. These layers of abstraction are implemented in both C and C++. Examples of C-based layers of abstraction are fprintf and printf. Internally, these functions call the write system call. An example of a C++-based layer of abstraction is the IOStreams hierarchy. Out of the box, most C++ compiler toolchains provide an implementation of IOStreams. IOStreams are an abstraction on top of the read and write system calls. When data is written to a terminal via an IOStream, the IOStream implementation calls the write system call. Lastly, these layers of abstraction handle things such as buffering and file synchronization.
In UNIX, everything is a file. Consequently, network devices, virtual terminals, files, block devices, etc., can all be written to via a numerical file descriptor - this in turn is why UNIX is referred to as having a uniform descriptor space. With this being said, the basic IOStreams and printf abstractions I mentioned above are not designed to be used with network sockets, pipes, and the like. The lower-layer read and write system calls can be used, but there are a number of functions that must be called before writing raw bytes to an open file descriptor that points to a network socket.
The additional functionality that is needed for communicating with network sockets, shared memory, and the like can be implemented in classes that are derived from the C++ iostream class. It is for this reason that the IOStreams classes are extended via inheritance.
Over the years, several popular C++ libraries have implemented classes that are derived from the base classes in the iostreams hierarchy. The C++ Boost library is a popular example. However, this has not always been the case. Going back to 1999, the Boost library did not exist, and there were one or two examples on the entire Internet as to how to properly extend the C++ IOStreams classes.
In 1999, the source code for the GNU compiler toolchain that is available on gcc.gnu.org was obtained, and a class hierarchy was derived to support sockets, pipes, and shared memory. The methods in the classes derived from the base classes in the iostreams library were designed to be reentrant and easy to use. Generative programming techniques and template metaprogramming were used to create objects that could be instantiated using familiar C++ iostreams syntax and semantics. The library created was called mls, and it was licensed under version 2 of the GPL.
Since 1999, Boost has come a long way. It provides support for cryptographic IOStreams, sockets, and all kinds of other fancy stuff. It uses generative programming techniques.The GCC compiler toolchain can be obtained from gcc.gnu.org. Ctags can then be used to dig into the internals of the IOStreams hierarchy. The following book is recommended: Generative Programming - Methods, Tools, and Applications.
The gcc compiler toolchain can be obained from gcc.gnu.org. Ctags can then be used to dig into the internals of the iostreams hierarchy.. The following book is recommended.
Generative Programming - Methods, Tools, and Applications
Many layers of abstraction reside on top of the read and write system calls. These layers of abstraction are implemented in both C and C++. Examples of C-based layers of abstraction are fprintf and printf. Internally, these functions call the write system call. An example of a C++-based layer of abstraction is the IOStreams hierarchy. Out of the box, most C++ compiler toolchains provide an implementation of IOStreams. IOStreams are an abstraction on top of the read and write system calls. When data is written to a terminal via an IOStream, the IOStream implementation calls the write system call. Lastly, these layers of abstraction handle things such as buffering and file synchronization.
In UNIX, everything is a file. Consequently, network devices, virtual terminals, files, block devices, etc., can all be written to via a numerical file descriptor - this in turn is why UNIX is referred to as having a uniform descriptor space. With this being said, the basic IOStreams and printf abstractions I mentioned above are not designed to be used with network sockets, pipes, and the like. The lower-layer read and write system calls can be used, but there are a number of functions that must be called before writing raw bytes to an open file descriptor that points to a network socket.
The additional functionality that is needed for communicating with network sockets, shared memory, and the like can be implemented in classes that are derived from the C++ iostream class. It is for this reason that the IOStreams classes are extended via inheritance.
Over the years, several popular C++ libraries have implemented classes that are derived from the base classes in the iostreams hierarchy. The C++ Boost library is a popular example. However, this has not always been the case. Going back to 1999, the Boost library did not exist, and there were one or two examples on the entire Internet as to how to properly extend the C++ IOStreams classes.
In 1999, the source code for the GNU compiler toolchain that is available on gcc.gnu.org was obtained, and a class hierarchy was derived to support sockets, pipes, and shared memory. The methods in the classes derived from the base classes in the iostreams library were designed to be reentrant and easy to use. Generative programming techniques and template metaprogramming were used to create objects that could be instantiated using familiar C++ iostreams syntax and semantics. The library created was called mls, and it was licensed under version 2 of the GPL.
Since 1999, Boost has come a long way. It provides support for cryptographic IOStreams, sockets, and all kinds of other fancy stuff. It uses generative programming techniques.The GCC compiler toolchain can be obtained from gcc.gnu.org. Ctags can then be used to dig into the internals of the IOStreams hierarchy. The following book is recommended: Generative Programming - Methods, Tools, and Applications.
The gcc compiler toolchain can be obained from gcc.gnu.org. Ctags can then be used to dig into the internals of the iostreams hierarchy.. The following book is recommended.
Generative Programming - Methods, Tools, and Applications
namespace mls
{
template<class BufType, int direction, class BaseType=mlbuf> class mlstreamimpl;
template<class Parent, class BaseType=mlbuf> class mloutputimpl;
template<class Parent, class BaseType=mlbuf> class mlinputimpl;
template<class BufType, int direction, class BaseType=BufType>
struct StreamConfig;
template<class BufType, int direction, class BaseType>
struct StreamConfig
{
typedef typename SWITCH<(direction),
CASE<0,mlinputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
CASE<1,mloutputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
CASE<10,mlinputimpl<mloutputimpl<mlstreamimpl<BufType, direction, BaseType>,
BufType>, BufType >,
CASE<DEFAULT,mlinputimpl<mlstreamimpl<BufType, 10, BaseType>,
BufType > > > > > >::RET Base;
};
}
{
template<class BufType, int direction, class BaseType=mlbuf> class mlstreamimpl;
template<class Parent, class BaseType=mlbuf> class mloutputimpl;
template<class Parent, class BaseType=mlbuf> class mlinputimpl;
template<class BufType, int direction, class BaseType=BufType>
struct StreamConfig;
template<class BufType, int direction, class BaseType>
struct StreamConfig
{
typedef typename SWITCH<(direction),
CASE<0,mlinputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
CASE<1,mloutputimpl<mlstreamimpl<BufType, direction, BaseType>, BufType>,
CASE<10,mlinputimpl<mloutputimpl<mlstreamimpl<BufType, direction, BaseType>,
BufType>, BufType >,
CASE<DEFAULT,mlinputimpl<mlstreamimpl<BufType, 10, BaseType>,
BufType > > > > > >::RET Base;
};
}
By
Bryan R. Hinton
Monday, October 24, 2011
Android Command Line Dev with VI
Notes on developing Android apps from *NIX command line.
Building an Android application from the command line with VI can save time. Here are some notes on setting up Vim w/ tags and code completion for Android development. The relevant Ant commands for building Android apps from the command line are included. The example includes the commands for building and installing an Android app that links to a dependent java library which resides outside of the project source tree (in this case, the lvl lib), along with a C shared library that resides in the local jni/ directory.
Subsequently, start the activity with
adb shell 'am start -n com.mycohname.myproject/.BaseActivityName'
And finally, connect via jdb from the shell with
$ jdb -sourcepath $HOME/src/myproject -attach localhost:8700
and start debugging.
Useful Vim Plugins for Android Development
Place vjde.tgz in $HOME/.vim and tar -zxvf vjde.tgz from within $HOME/.vim. Change the permissions on $HOME/.vim/plugin/vjde/readtags as follows:
$ chmod +x $HOME/.vim/plugin/vjde/readtags
Open an empty editor: $ vim and enter the following in command mode:
:helptags $HOME/.vim/doc
:h vjde
will then pull up the help page.
That should take care of setting up vjde. Now cd to the Android project dir. Open a blank editor and input the following in command mode:
:Vjdeas .myproject.prj
:let g:vjde_lib_path='/<path_to_android_sdk_top_level_dir>/platforms/ \
<desired_sdk_target>/android.jar:bin/classes:build.classes'
:Vjdesave
:q!
Next, Open up a source file in the project and type :Vjdeload .myproject.prj in command mode (or script and/or add to .vimrc). Use <ctrl-x><ctrl-u> for code completion. For example: import android.<ctrl-x><ctrl-u> and a nice little dialog box for browsing the matching frameworks.
Next, run ctags over the java and native sources as follows:
$ ctags -R src gen jni
Once NERD tree and Taglist are placed in ~/.vim/plugin/, the following lines in .vimrc will allow the use of <ctrl-n> and
<ctrl-m> to toggle the file explorer and visual tag list.
nmap <silent> <c-n> :NERDTreeToggle<CR>
nnoremap <silent> <c-m> :TlistToggle<CR>
Also, for a status line:
set statusline=\ %{HasPaste()}%F%m%r%h\ %w\ \ CWD:\ %r%{CurDir()}%h\ \ \ Line:\ %l/%L:%c
function! CurDir()
let curdir = substitute(getcwd(), '/Users/myhomedir/', "~/", "g")
return curdir
endfunction
function! HasPaste()
if &paste
return 'PASTE MODE '
else
return "
endif
endfunction
Vim should be good to go at this point. cd back to $HOME/src/myproject. This particular example accounts for a dependent Java library (the lvl) that resides outside of the project source tree, a shared library (which consists of a few C files natively compiled), and plain java source files in the appropriate src/com/ package subdir.
From within the top level project dir (assuming that Eclipse was used, otherwise, android create can be used ...),
$ android update project --name myproject --target <desired_sdk_target> \ --path $HOME/src/myproject
$ android update project --target <desired_sdk_target> --path $HOME/src/myproject \ --library ../lvl_lib_dir
Make sure to check project.properties to ensure that the android.library.reference.1 variable now contains the relative pathname of the lvl lib directory.
Assuming that jni/Android.mk and jni/Application.mk are appropriately setup for the shared library, run ndk-build from the top level project directory.
ant debug should now handle the build and debug version of the application package file.
Start up an Emulator and then install the app with a
db -r install bin/myproject-debug.apk or use ant install.
Next, open the Dev tools application in the emulator and configure the following: set wait for debugger and select the application for debugging.
Next, run ddms & and check the debug port. It should be 8700.- Tag List
- Nerd Tree
- VIM JDE
Place vjde.tgz in $HOME/.vim and tar -zxvf vjde.tgz from within $HOME/.vim. Change the permissions on $HOME/.vim/plugin/vjde/readtags as follows:
$ chmod +x $HOME/.vim/plugin/vjde/readtags
Open an empty editor: $ vim and enter the following in command mode:
:helptags $HOME/.vim/doc
:h vjde
will then pull up the help page.
That should take care of setting up vjde. Now cd to the Android project dir. Open a blank editor and input the following in command mode:
:Vjdeas .myproject.prj
:let g:vjde_lib_path='/<path_to_android_sdk_top_level_dir>/platforms/ \
<desired_sdk_target>/android.jar:bin/classes:build.classes'
:Vjdesave
:q!
Next, Open up a source file in the project and type :Vjdeload .myproject.prj in command mode (or script and/or add to .vimrc). Use <ctrl-x><ctrl-u> for code completion. For example: import android.<ctrl-x><ctrl-u> and a nice little dialog box for browsing the matching frameworks.
Next, run ctags over the java and native sources as follows:
$ ctags -R src gen jni
Once NERD tree and Taglist are placed in ~/.vim/plugin/, the following lines in .vimrc will allow the use of <ctrl-n> and
<ctrl-m> to toggle the file explorer and visual tag list.
nmap <silent> <c-n> :NERDTreeToggle<CR>
nnoremap <silent> <c-m> :TlistToggle<CR>
Also, for a status line:
set statusline=\ %{HasPaste()}%F%m%r%h\ %w\ \ CWD:\ %r%{CurDir()}%h\ \ \ Line:\ %l/%L:%c
function! CurDir()
let curdir = substitute(getcwd(), '/Users/myhomedir/', "~/", "g")
return curdir
endfunction
function! HasPaste()
if &paste
return 'PASTE MODE '
else
return "
endif
endfunction
Vim should be good to go at this point. cd back to $HOME/src/myproject. This particular example accounts for a dependent Java library (the lvl) that resides outside of the project source tree, a shared library (which consists of a few C files natively compiled), and plain java source files in the appropriate src/com/ package subdir.
From within the top level project dir (assuming that Eclipse was used, otherwise, android create can be used ...),
$ android update project --name myproject --target <desired_sdk_target> \ --path $HOME/src/myproject
$ android update project --target <desired_sdk_target> --path $HOME/src/myproject \ --library ../lvl_lib_dir
Make sure to check project.properties to ensure that the android.library.reference.1 variable now contains the relative pathname of the lvl lib directory.
Assuming that jni/Android.mk and jni/Application.mk are appropriately setup for the shared library, run ndk-build from the top level project directory.
ant debug should now handle the build and debug version of the application package file.
Start up an Emulator and then install the app with a
db -r install bin/myproject-debug.apk or use ant install.
Next, open the Dev tools application in the emulator and configure the following: set wait for debugger and select the application for debugging.
Subsequently, start the activity with
adb shell 'am start -n com.mycohname.myproject/.BaseActivityName'
And finally, connect via jdb from the shell with
$ jdb -sourcepath $HOME/src/myproject -attach localhost:8700
and start debugging.
By
Bryan R. Hinton
Tuesday, September 6, 2011
Radius and 802.1X
Configure Radius and 802.1X.
1. Generate a new self-signed root CA, write the encrypted private key to CA/private/cakey.pem, and then write the Base-64,ASN.1-encoded, self-signed certificate to CA/cacert.pem. This certificate will be used for signing client and server certificates.# openssl req -new -x509 -extensions v3_ca -keyout CA/priv/cakey.pem -out CA/cacert.pem -days 730 -config openssl.cnf
# openssl x509 -in cacert.pem -noout -text
# openssl x509 -in cacert.pem -noout -dates
# openssl x509 -in cacert.pem -noout -purpose
# openssl x509 -in cacert.pem -noout -issuer
# openssl rsa -noout -modulus -in CA/priv/cakey.pem | openssl sha1
# openssl x509 -noout -modulus -in CA/cacert.pem | openssl sha1
Check the modulus and public exponent in the private key and certificate to make sure they match.# openssl rsa -noout -modulus -in CA/priv/cakey.pem | openssl sha1
# openssl x509 -noout -modulus -in CA/cacert.pem | openssl sha1
2. Export the root CA signing certificate to ASN.1, DER encoded format so that clients can import it.
2. Export the root CA signing certificate to ASN.1, DER encoded format so that clients can import it.
# openssl x509 -in CA/cacert.pem -outform DER -out clientCerts/myRootCA.der
2a. Convert the DER encoded CA back to pem format and place in a .crt file so that Android can read it. (This is an extra, un-needed step as cacert.pem can be copied and renamed to .crt). (Android does not understand pem files so write the DER encoded certificate to PEM format in a file with extension .crt).
# openssl x509 -inform der -in clientCerts/myRootCA.der -out clientCerts/myRootCA.crt
3. Generate radius server certificate (i.e. signing request) and private key in unencrypted format.
# openssl req -new -nodes -keyout tempCerts/radius_key.pem -out tempCerts/radius_req.pem -days 730 -config openssl.cnf
4. Sign the radius server certificate. note: Microsoft clients require the creation of an xpextensions file. Add '-extensions xpserver_ext -extfile ./xpextensions' to the following command.
# openssl ca -out tempCerts/radius_cert.pem -infiles tempCerts/radius_req.pem -config openssl.cnf
5. Install the root CA signing certificate, Radius server private key, and Radius server signed certificate.
# cp tempCerts/radius_cert.pem /etc/radwl/certs/server/
# cp tempCerts/radius_key.pem /etc/radwl/certs/server/
# cp CA/cacert.pem /etc/radwl/certs/server/
6. Create the client certificate (i.e. signing request) and private key. note: match the output file names with the client identity or common name.
# cp CA/cacert.pem /etc/radwl/certs/server/
6. Create the client certificate (i.e. signing request) and private key. note: match the output file names with the client identity or common name.
# openssl req -new -keyout tempCerts/myandroid_key.pem -out tempCerts/myandroid_req.pem -days 730 -config openssl.cnf
7. Sign the client certificate.
# openssl ca -out tempCerts/myandroid_cert.pem -infiles tempCerts/myandroid_req.pem -config openssl.cnf
8. Export the signed client certificate and private key to pkcs#12 format.
# openssl pkcs12 -export -in tempCerts/myandroid_cert.pem -inkey tempCerts/myandroid_key.pem -out clientCerts/myandroid_cert.p12 -clcerts
9. Install the signed client certs on the Radius server.
# cp tempCerts/*_cert.pem /etc/radwl/certs/clients
10. Copy the client pkcs#12 certificate to appropriate device.
# cp clientCerts/myandroid_cert.p12 DEVICE
11. Copy the CA signing certificate to the same device.
# cp clientCerts/myRootCA.crt DEVICE
12. on OS X, use the following commands to add the freeradius user to the freeradius group. Also run chsh freeradius and set the shell to /sbin/nologin
# dscl . append /Groups/freeradius GroupMembership freeradius
By
Bryan R. Hinton
Tuesday, August 16, 2011
OpenSSH Security - Client Configuration
OpenSSH provides a suite of tools for encrypting traffic between endpoints, port forwarding, IP tunneling, and authentication. The below instructions outline a client side OpenSSH configuration where the client is running on OS X. The built in firewall, ipfw, is enabled on the client to restrict outbound and inbound traffic. Part II (currently on hold) of this guide will cover the configuration of OpenSSH on the server along with the available options and alternatives for authentication, authorization, and traffic encryption. The configuration will force AES 256 in Counter Mode and will restrict the available Message Authentication Algorithms that may be used between endpoints. Most of the options in the ssh configuration file on the server will be disabled, public key authentication will be used, password authentication will be disabled, and the ssh daemon will bind to a high number port. Multiple SSH sessions will use the same connection via the ControlMaster and ControlPath client configuration directive. Also, a server certificate will be generated and used to sign user public keys. The CA signed user public keys constitute a user certificate which the server will in turn use for client authentication. PF will be used on the server for stateful packet filtering, connection blocking, and connection throttling.
First and foremost, the client has ipfw enabled and the firewall ruleset is configured in /etc/ipfw.conf. ipfw has been configured to block all inbound traffic and block all outbound traffic except for the ports and IP addresses that are necessary for connecting to the OpenSSH server. The server is running FreeBSD 8.2.
FreeBSD 8.2 - sshd on a.b.c.d:21465 pf | <--------Internet----------> | ipfw OS X Lion - ssh client
To start with, install coreutils and apg on the client. coreutils and apg can be obtained from Mac ports and can be installed as follows:
client: $ sudo port install coreutils
client: $ sudo port install apg
Before generating a public/private keypair, generate a strong passphrase for the private key. It is important to store this passphrase in a secure location, not on a computer.
client: $ openssl rand -base64 1000 | shasum-5.12 -a 512 | apg -M SNCL -a 1 -m 20 -x 20
Depending on the version of OpenSSH (should be using latest stable for the OS), ECDSA may be used in addition to DSA and RSA. Certificates may also be used for user and host authentication. See the ssh-keygen man page for details. Generate the keypair using the following command. When prompted for the passphrase, use the output from the above command.
client: $ ssh-keygen -b 4096 -t rsa -C"$(id -un)@$hostname)-$(gdate --rfc-3339=date)"
Here is an example of how to use ssh-keygen to generate a public/private keypair using the Eliptic Curve Digital Signature Algorithm. Both the client and server must be running a version of OpenSSH >= 5.7.
client: $ ssh-keygen -b 521 -t ecdsa -C"$(id -un)@$hostname)-$(gdate --rfc-3339=date)"
Now, we need to push the public key to the server and place it in the authorized_keys file of the user that we are going to log in as over ssh.
The ssh-copy-id command can be used to automate this process. On the OS X client, the ssh-copy-id command does not come preinstalled with SSH. The ssh-copy-id command can be obtained from http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/ports/security/ssh-copy-id/files/ssh-copy-id?rev=1.1;content-type=text%2Fplain.
First and foremost, the client has ipfw enabled and the firewall ruleset is configured in /etc/ipfw.conf. ipfw has been configured to block all inbound traffic and block all outbound traffic except for the ports and IP addresses that are necessary for connecting to the OpenSSH server. The server is running FreeBSD 8.2.
FreeBSD 8.2 - sshd on a.b.c.d:21465 pf | <--------Internet----------> | ipfw OS X Lion - ssh client
To start with, install coreutils and apg on the client. coreutils and apg can be obtained from Mac ports and can be installed as follows:
client: $ sudo port install coreutils
client: $ sudo port install apg
Before generating a public/private keypair, generate a strong passphrase for the private key. It is important to store this passphrase in a secure location, not on a computer.
client: $ openssl rand -base64 1000 | shasum-5.12 -a 512 | apg -M SNCL -a 1 -m 20 -x 20
Depending on the version of OpenSSH (should be using latest stable for the OS), ECDSA may be used in addition to DSA and RSA. Certificates may also be used for user and host authentication. See the ssh-keygen man page for details. Generate the keypair using the following command. When prompted for the passphrase, use the output from the above command.
client: $ ssh-keygen -b 4096 -t rsa -C"$(id -un)@$hostname)-$(gdate --rfc-3339=date)"
Here is an example of how to use ssh-keygen to generate a public/private keypair using the Eliptic Curve Digital Signature Algorithm. Both the client and server must be running a version of OpenSSH >= 5.7.
client: $ ssh-keygen -b 521 -t ecdsa -C"$(id -un)@$hostname)-$(gdate --rfc-3339=date)"
Now, we need to push the public key to the server and place it in the authorized_keys file of the user that we are going to log in as over ssh.
The ssh-copy-id command can be used to automate this process. On the OS X client, the ssh-copy-id command does not come preinstalled with SSH. The ssh-copy-id command can be obtained from http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/ports/security/ssh-copy-id/files/ssh-copy-id?rev=1.1;content-type=text%2Fplain.
After downloading the script, change its permissions and place it in the path.
At this point, the server should be running OpenSSH on port 22 with the default configuration. Transfer the public key with the following command:
client: $ ssh-copy-id -i ~/.ssh/id_xxxyy.pub bryan@a.b.c.d \
It is time to setup connection sharing. Create the following file if it does not currently exist.
client: $ ls -l ~/.ssh/config -rw------- 1 bryan scclp 104 Aug 13 10:55 config
At this point, the server should be running OpenSSH on port 22 with the default configuration. Transfer the public key with the following command:
client: $ ssh-copy-id -i ~/.ssh/id_xxxyy.pub bryan@a.b.c.d \
It is time to setup connection sharing. Create the following file if it does not currently exist.
client: $ ls -l ~/.ssh/config -rw------- 1 bryan scclp 104 Aug 13 10:55 config
The file should contain these lines.
ServerAliveInterval 60 Host a.b.c.d ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h:%p
The goal is to only allow connections to the server in AES 256 Counter mode, with umac-64 or hmac-ripemd160 MACs, and compression, on a non-standard SSH port from a designated IP range using public key authentication. Connections will also be throttled and SSHGuard along with a few custom PF rules on the server will be used to block and log attackers. The commands that the client will use to connect to the server will look like this:client:
$ alias sshconnect="ssh -l bryan a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 client:
$ alias sshtunnel="ssh -v -ND 8090 bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 client:
$ alias sshmonitor="yes | pv | ssh -l bryan a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 \"cat > /dev/null\"" client:
$ alias sshportforward="ssh -f bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 -L 15478:localhost:15479 -N" client:
$ alias sshportforward2="ssh -f bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 -L 17293:localhost:17294 -N"
Alternatively, Ciphers, MACs, and compression can be specified in the user config file as follows:
ServerAliveInterval 60
Host host.name.com
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h:%p
Port 21465
User bryan
Ciphers aes256-ctr
Compression yes
MACs umac-64@openssh.com,hmac-ripemd160
StrictHostKeyChecking yes
User and Host certificates provide a more convenient method of authentication for multiple clients (users) and servers (hosts). Certificate revocation can also provide an easier method of quickly invalidating user access.A certificate authority key pair is first generated as follows. The ca is then placed in the /etc/ssh directory on the host.
ca $ ssh-keygen -t ecdsa -b 521 -f user_ca server $ sudo mv user_ca* /etc/ssh/
On the client, generate a public/private key pair and then copy the public key to the server so that it can be signed with the ca. Make sure to set the validity period of the certificate. Alternatively, a host key may be signed with a ca key that is stored in a PKCS11 token. OpenSSH supports ca keys stored PCKS11 tokens. Check the version of SSH and see ssh-keygen for more information.client
client $ ssh-keygen -t ends -b 521 -f ~/.ssh/id_ecdsa
client $ scp .ssh/id_ecdsa.pub bryan@server-ca:~/user_public_keys
server-ca $ ssh-keygen -s /etc/ssh/user_ca \
-O source-address=clientip
-O permit-pty
-O no-port-forwarding
-O no-user-rc
-O no-x11-forwarding \ -V -1d:+52w1d -z 6739301351 -I "bryan" -n bryan,clienthostname id_ecdsa.pub
id "bryan" serial 6739301351 for bryan,clienthostname valid from 2011-08-18T15:05:24 to 2012-08-17T15:05:24
Copy the signed user cert back to the client.
client $ scp bryan@server:~/user_public_keys/id_ecdsa-cert.pub ~/.ssh/
Setup TrustedUserCAKeys and AuthorizedPrincipalsFile files. Subsequently, set appropriate options in /etc/ssh/sshd_config on the server.
server-ca $ sudo cat /etc/ssh/user_ca.pub > /etc/ssh/trusted_user_ca_keys
Modify /etc/ssh/authorized_principals to include the following lines.bryan from="clientip" bryan
Modify /etc/ssh/sshd_config on the server to include the following lines
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys
AuthorizedPrincipalsFile /etc/ssh/authorized_principals
Now, restart sshd on the server and add an appropriate host configuration for certificate authentication to ~/.ssh/config on the client.
Last of all, setup a host certification via the -h option with ssh-keygen when signing a host key.
It is important to always keep OpenSSH updated with the latest, stable version that has been released for the operating system.
ServerAliveInterval 60 Host a.b.c.d ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h:%p
The goal is to only allow connections to the server in AES 256 Counter mode, with umac-64 or hmac-ripemd160 MACs, and compression, on a non-standard SSH port from a designated IP range using public key authentication. Connections will also be throttled and SSHGuard along with a few custom PF rules on the server will be used to block and log attackers. The commands that the client will use to connect to the server will look like this:client:
$ alias sshconnect="ssh -l bryan a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 client:
$ alias sshtunnel="ssh -v -ND 8090 bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 client:
$ alias sshmonitor="yes | pv | ssh -l bryan a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 \"cat > /dev/null\"" client:
$ alias sshportforward="ssh -f bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 -L 15478:localhost:15479 -N" client:
$ alias sshportforward2="ssh -f bryan@a.b.c.d -p 21465 -C -c aes256-ctr -m umac-64@openssh.com,hmac-ripemd160 -L 17293:localhost:17294 -N"
Alternatively, Ciphers, MACs, and compression can be specified in the user config file as follows:
ServerAliveInterval 60
Host host.name.com
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h:%p
Port 21465
User bryan
Ciphers aes256-ctr
Compression yes
MACs umac-64@openssh.com,hmac-ripemd160
StrictHostKeyChecking yes
User and Host certificates provide a more convenient method of authentication for multiple clients (users) and servers (hosts). Certificate revocation can also provide an easier method of quickly invalidating user access.A certificate authority key pair is first generated as follows. The ca is then placed in the /etc/ssh directory on the host.
ca $ ssh-keygen -t ecdsa -b 521 -f user_ca server $ sudo mv user_ca* /etc/ssh/
On the client, generate a public/private key pair and then copy the public key to the server so that it can be signed with the ca. Make sure to set the validity period of the certificate. Alternatively, a host key may be signed with a ca key that is stored in a PKCS11 token. OpenSSH supports ca keys stored PCKS11 tokens. Check the version of SSH and see ssh-keygen for more information.client
client $ ssh-keygen -t ends -b 521 -f ~/.ssh/id_ecdsa
client $ scp .ssh/id_ecdsa.pub bryan@server-ca:~/user_public_keys
server-ca $ ssh-keygen -s /etc/ssh/user_ca \
-O source-address=clientip
-O permit-pty
-O no-port-forwarding
-O no-user-rc
-O no-x11-forwarding \ -V -1d:+52w1d -z 6739301351 -I "bryan" -n bryan,clienthostname id_ecdsa.pub
id "bryan" serial 6739301351 for bryan,clienthostname valid from 2011-08-18T15:05:24 to 2012-08-17T15:05:24
Copy the signed user cert back to the client.
client $ scp bryan@server:~/user_public_keys/id_ecdsa-cert.pub ~/.ssh/
Setup TrustedUserCAKeys and AuthorizedPrincipalsFile files. Subsequently, set appropriate options in /etc/ssh/sshd_config on the server.
server-ca $ sudo cat /etc/ssh/user_ca.pub > /etc/ssh/trusted_user_ca_keys
Modify /etc/ssh/authorized_principals to include the following lines.bryan from="clientip" bryan
Modify /etc/ssh/sshd_config on the server to include the following lines
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys
AuthorizedPrincipalsFile /etc/ssh/authorized_principals
Now, restart sshd on the server and add an appropriate host configuration for certificate authentication to ~/.ssh/config on the client.
Last of all, setup a host certification via the -h option with ssh-keygen when signing a host key.
It is important to always keep OpenSSH updated with the latest, stable version that has been released for the operating system.
By
Bryan R. Hinton
Sunday, February 27, 2011
Androidified
I picked up a Motorola Xoom tablet on February 24th, the day of its release. To start with, the Xoom is equipped with a Dual-Core ARM Cortex A9 processor called the Nvidia Tegra 2. The Xoom runs Android 3.0, Honeycomb. There are no third party UI replacements or additions. The Xoom runs pure Android 3.0 and it is very fast.
One of the nice things that you can do is customize the keyboard. For a 10 inch tablet, it is convenient to be able to type with your left and right thumbs and Thumb Keyboard is available on the Android Marketplace just for this.
Thumb Keyboard greatly speeds up typing on the Xoom.
Next, let's build a statically linked Busybox binary so that you can do something useful with the device.
One of the nice things that you can do is customize the keyboard. For a 10 inch tablet, it is convenient to be able to type with your left and right thumbs and Thumb Keyboard is available on the Android Marketplace just for this.
Thumb Keyboard greatly speeds up typing on the Xoom.
Next, let's build a statically linked Busybox binary so that you can do something useful with the device.
# wget http://busybox.net/downloads/busybox-1.18.3.tar.bz2 # wget http://www.busybox.net/downloads/busybox-1.18.3.tar.bz2.sign # http://busybox.net/~vda/vda_pubkey.gpg # gpg --check-sigs vda.linux # gpg --verify busybox-1.18.3.tar.bz2.sign # sha1sum busybox-1.18.3.tar.bz2 # md5sum busybox-1.18.3.tar.bz2 # bzip2 -cd busybox-1.18.3.tar.bz2 | tar xf - # cd busybox-1.18.3 # make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- defconfig # make CFLAGS=--static LDFLAGS=--static ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- # adb remount # adb push busybox /system/xbin/busybox # adb shell chmod 0755 /system/xbin/busybox # adb shell # cd /system/xbin # ./busybox --install -s /system/xbin # cd /mnt/sdcard # echo "export PATH="export PATH=/system/xbin:/sbin:/vendor/bin:/system/sbin:/system/bin" > profile # echo "ENV=/mnt/sdcard/profile /system/xbin/ash" > /system/bin/alsh # chmod 0755 /system/bin/alsh # alsh # uname -ia # Linux localhost 2.6.36.3-g2a65edc #1 SMP PREEMPT Mon Feb 7 15:24:33 PST 2011 armv7l GNU/Linux
By
Bryan R. Hinton
Saturday, July 31, 2010
Agile Data modeling, de-normalization, and other important factors
This subject probably deserves an entire book, but in the essence of keeping this blog post short, I've tried to highlight some of the more important data modeling tasks related to conceptual, logical, and physical data model design. Please note, this is not meant to be exhaustive. Relational models, object-relational models, and object models are not always the right solution for the problem. There are many great solutions available in the NoSQL area, and there is nothing wrong with a hybrid solution.
Software applications that are compiled on specific machine architectures often store and retrieve information from database software systems. Web server platforms often interpret machine-independent script that stores and retrieves information from database software systems. Java bytecode is executed by a Java Virtual Machine, and data is stored and retrieved from database software systems.
Efficient storage and retrieval of information from database software systems are certainly dependent on underlying hardware and operating system software characteristics. However, efficient storage and retrieval of information is also dependent on the structure and efficient execution of query statements that are used to store and retrieve the information. In a purely academic world, data is modeled and normalized in 3NF, 4NF, and 5NF forms. However, higher-order normalized forms do not always promote the creation of optimal query statements. Therefore, certain parts of a data model are often denormalized to improve the read performance of the database software system.
At a high level, joins between multiple, large relations are generally expensive, so denormalization reduces this cost. Reducing the cost of something typically results in inconsistencies or redundancies in the data, so the developer(s) are left with the job of minimizing the amount of redundant data. A formal process typically alleviates this strain.
Defining the I/O characteristics of a database is an important precursor to data modeling. Often times, the I/O characteristics of the database are not defined before modeling the data. Nevertheless, there are a few basic tasks and questions that can help define the I/O characteristics of the database. What is the purpose of the database and what types of problem(s) does the database solve? What types of operations (read, write) will be performed on the database? How many users will the database support? Define user stories - short, succinct 1-3 sentence action statements. Define as many as needed from both the developer perspective and the user perspective. The user stories will help define the conceptual data model for the domain from which the logical data model can be created. A clear definition of the purpose of the system will aid in defining the read/write (I/O) characteristics of the database software system, thus forming a picture of the system. For instance, OLTP databases typically support short transactions, so it is important that write latency is minimized. The answers to the questions above will also help determine the type of hardware storage configuration needed. Another important question when determining the type of hardware configuration is the following. How much downtime can be afforded, and how much failover is needed?
At this point, the types of queries can be seen that will be run on the database with heavy consideration given to the I/O characteristics of the database and the types of joins and scans preferred for the query optimizer to ideally use.
Agile data modeling can be defined as continual iterative feedback with the ability to add entities and relationships, or slice off sections of a data model in various sizes, at any point in the project. This may seem difficult but can be accomplished. Whether the database architecture adheres to the relational model, object-relational model, or object model, the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow the addition and subtraction of small and large chunks of relations from the data model throughout the development process; for a specific domain. Formal algorithms can be used to map entity relationship models to relations.
And how is this done? Continually modeling entities within the real world. Abstracting classes, re-factoring, keeping the hierarchy semi-flat, avoiding deep polymorphic behavior, and always embracing changes to the data model while never generalizing with a one size fits all approach.
It's important to note that data modeling is a crucial step in designing a database software system that efficiently stores and retrieves information. This process involves defining the I/O characteristics of the database, understanding the purpose of the system, and creating a conceptual data model for the domain. From there, a logical data model can be created, followed by a physical data model that considers the hardware storage configuration needed. Agile data modeling allows for continual iterative feedback, and the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow for the addition and subtraction of small and large chunks of relations from the data model throughout the development process. Overall, data modeling is a critical aspect of creating an efficient and effective database software system.
Software applications that are compiled on specific machine architectures often store and retrieve information from database software systems. Web server platforms often interpret machine-independent script that stores and retrieves information from database software systems. Java bytecode is executed by a Java Virtual Machine, and data is stored and retrieved from database software systems.
Efficient storage and retrieval of information from database software systems are certainly dependent on underlying hardware and operating system software characteristics. However, efficient storage and retrieval of information is also dependent on the structure and efficient execution of query statements that are used to store and retrieve the information. In a purely academic world, data is modeled and normalized in 3NF, 4NF, and 5NF forms. However, higher-order normalized forms do not always promote the creation of optimal query statements. Therefore, certain parts of a data model are often denormalized to improve the read performance of the database software system.
At a high level, joins between multiple, large relations are generally expensive, so denormalization reduces this cost. Reducing the cost of something typically results in inconsistencies or redundancies in the data, so the developer(s) are left with the job of minimizing the amount of redundant data. A formal process typically alleviates this strain.
Defining the I/O characteristics of a database is an important precursor to data modeling. Often times, the I/O characteristics of the database are not defined before modeling the data. Nevertheless, there are a few basic tasks and questions that can help define the I/O characteristics of the database. What is the purpose of the database and what types of problem(s) does the database solve? What types of operations (read, write) will be performed on the database? How many users will the database support? Define user stories - short, succinct 1-3 sentence action statements. Define as many as needed from both the developer perspective and the user perspective. The user stories will help define the conceptual data model for the domain from which the logical data model can be created. A clear definition of the purpose of the system will aid in defining the read/write (I/O) characteristics of the database software system, thus forming a picture of the system. For instance, OLTP databases typically support short transactions, so it is important that write latency is minimized. The answers to the questions above will also help determine the type of hardware storage configuration needed. Another important question when determining the type of hardware configuration is the following. How much downtime can be afforded, and how much failover is needed?
At this point, the types of queries can be seen that will be run on the database with heavy consideration given to the I/O characteristics of the database and the types of joins and scans preferred for the query optimizer to ideally use.
Agile data modeling can be defined as continual iterative feedback with the ability to add entities and relationships, or slice off sections of a data model in various sizes, at any point in the project. This may seem difficult but can be accomplished. Whether the database architecture adheres to the relational model, object-relational model, or object model, the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow the addition and subtraction of small and large chunks of relations from the data model throughout the development process; for a specific domain. Formal algorithms can be used to map entity relationship models to relations.
And how is this done? Continually modeling entities within the real world. Abstracting classes, re-factoring, keeping the hierarchy semi-flat, avoiding deep polymorphic behavior, and always embracing changes to the data model while never generalizing with a one size fits all approach.
It's important to note that data modeling is a crucial step in designing a database software system that efficiently stores and retrieves information. This process involves defining the I/O characteristics of the database, understanding the purpose of the system, and creating a conceptual data model for the domain. From there, a logical data model can be created, followed by a physical data model that considers the hardware storage configuration needed. Agile data modeling allows for continual iterative feedback, and the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow for the addition and subtraction of small and large chunks of relations from the data model throughout the development process. Overall, data modeling is a critical aspect of creating an efficient and effective database software system.
By
Bryan R. Hinton
Close to Optimal Data Modeling
Agile Data modeling, de-normalization, and other important factors
This subject probably deserves an entire book but in the essence of keeping this blog post short, I've tried to highlight some of the more important data modeling tasks related to conceptual, logical, and physical data model design. Please note, this is not meant to be exhaustive. Relational models, object relational models, and object models are not always the right solution for the problem. There are many great solutions available in the noSQL area and there is nothing wrong with a hybrid solution.
Software applications that are compiled on specific machine architectures often store and retrieve information from database software systems. Web server platforms often interpret machine independent script that stores and retrieves information from database software systems. Java byte code is executed by a Java Virtual Machine and data is stored and retrieved from database software systems.
Efficient storage and retrieval of information from database software systems is certainly dependent on underlying hardware and operating system software characteristics. However; efficient storage and retrieval of information is also dependent on the structure and efficient execution of query statements that are used to store and retrieve the information. In a purely academic world, data is modeled and normalized in 3NF, 4NF, and 5NF forms. However; higher order normalized forms do not always promote the creation of optimal query statements. Therefore, certain parts of a data model are often de-normalized to improve the read performance of the database software system. At a high level, joins between multiple, large relations are generally expensive so de-normalization reduces this cost. Rreducing the cost of something typically results in inconsistencies or redundancies in the data so the developer(s) is left with the job of minimizing the amount of redundant data. Formal process typically alleviates this strain.
Defining the I/O characteristics of a database is an important precursor to data modeling. Often times, the I/O characteristics of the database are not defined before modeling the data. Nevertheless, there are a few basic tasks and questions that can help define the I/O characteristics of the database. What is the purpose of the database and what types of problem(s) does the database solve? What types of operations (read, write) will you be performing on the database? How many users will the database support? Define user stories - short, succinct 1-3 sentence action statements. Define as many as you need from both the developer perspective and the user perspective. The user stories will help define the conceptual data model for the domain from which the logical data model can be created. A clear definition of the purpose of the system will aid in defining the read/write (I/O) characteristics of the database software system. Thus forming a picture of the system. For instance, OLTP databases typically support short transactions so it is important that write latency is minimized. The answers to the questions above will also help determine the type of hardware storage configuration needed. Another important question when determining the type of hardware configuration is the following. How much downtime can we afford and how much failover do we need?
At this point, we can begin to see the types of queries that will be run on the database with heavy consideration given to the I/O characteristics of the database and the types of joins and scans that we would like for the query optimizer to ideally use.
Agile data modeling - Continual, iterative feedback. The ability to add entities and relationships, or slice off sections of a data model in various sizes, at any point in the project. This may seem difficult but can be accomplished. Whether the database architecture adheres to the relational model, object-relational model, or object model, the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow the addition and subtraction of small and large chunks of relations from the data model throughout the development process; for a specific domain.
Formal algorithms can be used to map entity relationship models to relations.
And how is this done? Continually modeling entities within the real world. Abstracting classes, re-factoring, keeping the hierarchy semi-flat, avoiding deep polymorphic behavior, and always embracing changes to the data model while never generalizing with a one size fits all approach.
This subject probably deserves an entire book but in the essence of keeping this blog post short, I've tried to highlight some of the more important data modeling tasks related to conceptual, logical, and physical data model design. Please note, this is not meant to be exhaustive. Relational models, object relational models, and object models are not always the right solution for the problem. There are many great solutions available in the noSQL area and there is nothing wrong with a hybrid solution.
Software applications that are compiled on specific machine architectures often store and retrieve information from database software systems. Web server platforms often interpret machine independent script that stores and retrieves information from database software systems. Java byte code is executed by a Java Virtual Machine and data is stored and retrieved from database software systems.
Efficient storage and retrieval of information from database software systems is certainly dependent on underlying hardware and operating system software characteristics. However; efficient storage and retrieval of information is also dependent on the structure and efficient execution of query statements that are used to store and retrieve the information. In a purely academic world, data is modeled and normalized in 3NF, 4NF, and 5NF forms. However; higher order normalized forms do not always promote the creation of optimal query statements. Therefore, certain parts of a data model are often de-normalized to improve the read performance of the database software system. At a high level, joins between multiple, large relations are generally expensive so de-normalization reduces this cost. Rreducing the cost of something typically results in inconsistencies or redundancies in the data so the developer(s) is left with the job of minimizing the amount of redundant data. Formal process typically alleviates this strain.
Defining the I/O characteristics of a database is an important precursor to data modeling. Often times, the I/O characteristics of the database are not defined before modeling the data. Nevertheless, there are a few basic tasks and questions that can help define the I/O characteristics of the database. What is the purpose of the database and what types of problem(s) does the database solve? What types of operations (read, write) will you be performing on the database? How many users will the database support? Define user stories - short, succinct 1-3 sentence action statements. Define as many as you need from both the developer perspective and the user perspective. The user stories will help define the conceptual data model for the domain from which the logical data model can be created. A clear definition of the purpose of the system will aid in defining the read/write (I/O) characteristics of the database software system. Thus forming a picture of the system. For instance, OLTP databases typically support short transactions so it is important that write latency is minimized. The answers to the questions above will also help determine the type of hardware storage configuration needed. Another important question when determining the type of hardware configuration is the following. How much downtime can we afford and how much failover do we need?
At this point, we can begin to see the types of queries that will be run on the database with heavy consideration given to the I/O characteristics of the database and the types of joins and scans that we would like for the query optimizer to ideally use.
Agile data modeling - Continual, iterative feedback. The ability to add entities and relationships, or slice off sections of a data model in various sizes, at any point in the project. This may seem difficult but can be accomplished. Whether the database architecture adheres to the relational model, object-relational model, or object model, the proper classification of entities and attributes, in conjunction with a balance of normalized and de-normalized groups of data, will allow the addition and subtraction of small and large chunks of relations from the data model throughout the development process; for a specific domain.
Formal algorithms can be used to map entity relationship models to relations.
And how is this done? Continually modeling entities within the real world. Abstracting classes, re-factoring, keeping the hierarchy semi-flat, avoiding deep polymorphic behavior, and always embracing changes to the data model while never generalizing with a one size fits all approach.
By
Bryan R. Hinton
Saturday, July 24, 2010
High Performance Database Development
Fast, High performance, database driven, application architectures that support true rapid development are little talked about. AOLServer, OpenACS, PostgreSQL, and TCL are such a combination.
I was introduced to AOLServer and OpenACS in 2002 by a close friend who I grew up with. We built a very robust application architecture over the course of the following 6 years.
PostgreSQL was sitting behind the AOLServer/OpenACS instance and AOLServer nicely managed the database connection pools. AOLServer is multi-threaded and internally handles TCL interpretation and execution. TCL makes heavy use of lists (implementation notes aside), so many LISP programmers have enjoyed working with it.
The OpenACS developer toolbar is a tool that provides developers with access to various debugging and profiling information about their OpenACS application. It is a feature that is built into the OpenACS application framework and can be enabled or disabled depending on the needs of the developer.
Once enabled, the developer toolbar is accessible via a web browser and provides a range of information about the current page, including SQL queries, response times, and server resources. This information can be used to optimize and debug an OpenACS application, allowing developers to quickly identify and fix performance issues.
The OpenACS developer toolbar is just one of the many tools available to developers using the AOLServer, OpenACS, PostgreSQL, and TCL stack. Together, these technologies provide a fast, high-performance, database-driven application architecture that supports rapid development.
I was introduced to AOLServer and OpenACS in 2002 by a close friend who I grew up with. We built a very robust application architecture over the course of the following 6 years.
PostgreSQL was sitting behind the AOLServer/OpenACS instance and AOLServer nicely managed the database connection pools. AOLServer is multi-threaded and internally handles TCL interpretation and execution. TCL makes heavy use of lists (implementation notes aside), so many LISP programmers have enjoyed working with it.
The OpenACS developer toolbar is a tool that provides developers with access to various debugging and profiling information about their OpenACS application. It is a feature that is built into the OpenACS application framework and can be enabled or disabled depending on the needs of the developer.
Once enabled, the developer toolbar is accessible via a web browser and provides a range of information about the current page, including SQL queries, response times, and server resources. This information can be used to optimize and debug an OpenACS application, allowing developers to quickly identify and fix performance issues.
The OpenACS developer toolbar is just one of the many tools available to developers using the AOLServer, OpenACS, PostgreSQL, and TCL stack. Together, these technologies provide a fast, high-performance, database-driven application architecture that supports rapid development.
By
Bryan R. Hinton
Tuesday, July 20, 2010
Dtrace on FreeBSD 8
Dtrace on FreeBSD 8 - new server with no load
# uname -msri
FreeBSD 8.0-RELEASE-p3 amd64 DEV_A64_KERNEL
# /usr/local/share/DTTraceToolkit/syscallbypid.d Tracing... Hit Ctrl-C to end. 77913 perl sigprocmask 48 1267 svscan stat 51 45001 httpd read 52 26063 python2.4 recvfrom 61 26922 sshd ioctl 69 27104 more read 69 26888 postgres getppid 78 26888 postgres gettimeofday 78 26888 postgres select 78 26889 postgres getppid 78 26889 postgres select 78 45001 httpd poll 81 27102 postgres read 82 27103 postgres read 82 27105 postgres read 82 27101 dtrace clock_gettime 97 26063 python2.4 select 102 45001 httpd writev 118 44966 python2.4 read 121 26890 postgres read 162 26063 python2.4 read 179 26063 python2.4 _umtx_op 186 27104 more write 205 26063 python2.4 write 208 26891 postgres write 300 27101 dtrace ioctl 301 26922 sshd write 304 26922 sshd read 306 27054 psql poll 378 27054 psql recvfrom 378 27055 postgres sendto 379 26922 sshd select 609 26922 sshd sigprocmask 1218 27054 psql write 3203
# /usr/local/share/DTTraceToolkit/rwbypid.d 78168 sshd R 39 97062 squid R 58 45164 httpd W 81 97062 squid W 95 97474 python2.4 R 98 555 pflogd R 132 45164 httpd R 186
# dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
CPU ID FUNCTION:NAME
0 25202 open:entry svscan .
0 25202 open:entry imapd ./cur
0 25202 open:entry imapd ./new
0 25202 open:entry imapd ./courierimapkeywords/:list
0 25202 open:entry imapd ./courierimapkeywords
0 25202 open:entry svscan .
2 25202 open:entry postgres pg_stat_tmp/pgstat.tmp
7 25202 open:entry postgres global/pg_database
7 25202 open:entry postgres pg_stat_tmp/pgstat.stat
7 25202 open:entry postgres pg_stat_tmp/pgstat.stat
7 25202 open:entry postgres pg_stat_tmp/pgstat.stat
4 25202 open:entry tinydns data.cdb
5 25202 open:entry tinydns data.cdb
0 25202 open:entry svscan .
0 25202 open:entry svscan .
0 25202 open:entry svscan .Syscalls count by process# dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'
28820 qmailmrtg7 158
28825 qmailmrtg7 158
27504 postgres 159
28776 cron 234
28853 bash 245
28861 bash 245
28869 bash 245
28877 bash 245
28798 qmailmrtg7 264
28777 newsyslog 293
28772 dtrace 322
28778 sh 327
28281 httpd 542
28900 svn 691
28903 svn 740
28902 svn 748
28904 svn 748
28901 svn 758
28780 perl 3023
# Files opened by process
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
CPU ID FUNCTION:NAME
4 25202 open:entry svscan .
0 25202 open:entry squid /usr/local/plone/<domain-name>-dev.com/var/squidstorage/08/0C
4 25202 open:entry svscan .
4 25202 open:entry svscan .
4 25202 open:entry svscan .
0 25202 open:entry imapd ./.Spam/tmp/1279591336.M11620P234573_courierlock.dev.<domain-name>.com
0 25202 open:entry imapd ./.Spam/tmp/1279591336.M11620P234573_courierlock.dev.<domain-name>.com
0 25202 open:entry imapd ./.Spam/tmp
0 25202 open:entry imapd ./.Spam/tmp/1279591336.M11867P28573_imapuid_15.dev.<domain-name>.com
0 25202 open:entry imapd ./.Spam/courierimapuiddb
0 25202 open:entry imapd ./.Spam/tmp/1279591336.M113467P28573_imapuid_15.dev.<domain-name>.com
0 25202 open:entry imapd ./.Spam/cur
0 25202 open:entry imapd ./.Spam/new
0 25202 open:entry imapd ./.Spam/courierimapkeywords/:list
0 25202 open:entry imapd ./.Spam/courierimapkeywords
0 25202 open:entry squid /usr/local/plone/abcdexcrfdsf-dev.com/var/squidstorage/09/0C
4 25202 open:entry svscan .
By
Bryan R. Hinton
Thursday, May 27, 2010
bsdtalk186 - Soft Updates w/ Journaling & ULE Sched
FreeBSD Soft Updates with Journaling, ULE Scheduler
Very good short interview with Jeff Roberson, FreeBSD Committer.Discussion on the addition of journaling to soft updates and the development of the ULE scheduler.You can also listen on your Android device. Just download Google Listen from the marketplace. After you setup Google Listen on your Android Device, search for bsdtalk or FreeBSD from within Google Listen.
Very good short interview with Jeff Roberson, FreeBSD Committer.Discussion on the addition of journaling to soft updates and the development of the ULE scheduler.You can also listen on your Android device. Just download Google Listen from the marketplace. After you setup Google Listen on your Android Device, search for bsdtalk or FreeBSD from within Google Listen.
By
Bryan R. Hinton
Wednesday, May 5, 2010
Single Thread Object Communication
When objects in a single thread are loosely coupled and need to communicate with each other, Notifications allow you to broadcast messages on a synchronous or asynchronous basis and still retain object independence.
When you implement Store Kit in your application, it is most likely a reasonable assumption that you want to re-use your in-app store in a future iPhone application without rewriting the object(s) that implement the SKPaymentTransactionObserver and SKProductsRequestDelegate protocols. We know that these objects receive asynchronous notifications via the delegate protocol(s) that they implement. So in a nutshell, the object that implements the SKProductsRequestDelegate protocol will receive a response asynchronously when the product identifier request that it previously sent is successfully processed. The payment transaction observer object, which is added as an observer to the payment queue upon application launch, again receives asynchronous notifications when there's stuff to do in the payment queue.
When your transaction observer receives notifications from the payment queue that there are updated transactions, completed transactions, or failures, it is most likely the responsibility of the observer object to process these notifications. The processing of these notifications most likely entails application specific logic. So rather than loading up your transaction observer with application specific logic, the NSNotification center provides a clean way of synchronously or asynchronously(via Notification Queues) notifying other objects within the same thread. There is also an NSDistributedNotificationCenter for broadcasting messages to objects in other threads.
By using the NSNotification center, you can separate the application-specific logic from the observer object that receives notifications from the payment queue. This allows for a cleaner and more modular design, as each object can focus on its specific responsibilities without being tightly coupled to other objects. In addition, the NSNotification center provides a flexible way to communicate between objects within the same thread or across different threads using the NSDistributedNotificationCenter. This makes it easier to reuse and refactor code, which can save time and effort in the long run.
By
Bryan R. Hinton
Monday, May 3, 2010
Creating a Mobile App for the iPhone
In the coming weeks, I plan on covering the development process from start to finish. I will be talking about code organization, memory management, useful xcode features, core data, performance optimization, debugging, iterative development, and any other useful topics that I come across.
iLogMiles was approved by Apple for sale on the App store, on April 6, 2010. iLogMiles is free to download. The app quickly hit the #2 spot on the top free business apps list. The user can upgrade to the full version and purchase themes from within the free application. We utilized the Apple Store Kit framework to provide the in-app store. The Store Kit framework provides the necessary means to securely process payments from within the application. Products are identified via product identifiers. Apple clearly defines the types of products that may be sold via the Store Kit framework. More on this later but keep in mind that it is very important to read through all of the documentation.
Over the next few weeks, we will be talking about the mobile application development process. To give you some background on me, I started with C++ in 1997. At the time, I mostly used the GNU compilers on FreeBSD and Linux and Sun workshop on a SPARCstation. And for the C++ courses I was taking, Borland in a Win environment. To get up to speed on iPhone OS, I purchased the WWDC 2009 sessions and watched them all (at the gym on my iPhone). I attended the iPhone tech talk tour in San Jose and I listened to any and all of the available iPhone development podcasts (at the gym) that I could find. I poured through all of the books that I could get my hands on. I read the iPhone OS programming guides on the iPhone OS developer connection. XCode also has integrated documentation for the Apple API's. I've always used vi/vim with ctags, so I had to force myself to watch the xcode screencasts (available via third-party on the Internet). There are lots of great xcode tips out there and you can print out a summary page with all of the cute shortcuts - if you are so inclined to do so. I found it quite helpful to print the shortcuts page. Design patterns are at the core of how the Cocoa frameworks interact. Apple provides solid documentation on Cocoa design patterns in the iPhone OS Reference Library and these patterns can be understood at a high-level without delving into any code. This is my suggested approach. If you are not familiar with design patterns, the gang of four book (the GOF book as most often termed in the software world) - Design Patterns: Elements of Reusable Object-Oriented Software, by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, is the official source for design patterns. The theory behind Cocoa's design patterns can be found in this book and for that matter, the use of design patterns in any modern object-oriented software system can be found in this book. The GOF book describes design patterns in a language independent manner. Familiarizing yourself with Cocoa's design patterns from a language independent perspective is a key starting point for building an iPhone application.
Continuing on the topic of cocoa design patterns, Apple's frameworks are designed for re-use. There are folks out there who are very experienced with design patterns and are able to layer design patterns on top of the cocoa frameworks. These individuals understand all of the memory implications. They understand MVC backward and forward and they understand the Cocoa design patterns backward and forward. If you do not fall into this category, then keep it simple. If there is one problem that I have seen across forums and code samples on the Internet, it is that of "fighting the apis". It is very important to not fight the apis. The apple documentation describes the proper use of the iPhone OS frameworks and associated API's. The Apple developer connection provides plenty of sample code so it is important to read through this code to understand the proper use of the iPhone OS APIs.
From inception to app store approval, we spent 5 months developing iLogMiles. While this may seem longer than the standard "2 month" development lifecycle, I can assure you that iLogMiles uses a wide variety of the iPhone OS frameworks. In the coming weeks, we plan on covering the development process from start to finish. I will be talking about code organization, memory management, xcode features, core data, performance optimization, debugging, iterative development, and any other topics that we see useful.
Apple, the Apple logo, iPhone, iPod touch, and iTunes are trademarks of Apple Inc., registered in the U.S. and other countries. iPad is a trademark of Apple Inc. App Store is a service mark of Apple Inc.
iLogMiles was approved by Apple for sale on the App store, on April 6, 2010. iLogMiles is free to download. The app quickly hit the #2 spot on the top free business apps list. The user can upgrade to the full version and purchase themes from within the free application. We utilized the Apple Store Kit framework to provide the in-app store. The Store Kit framework provides the necessary means to securely process payments from within the application. Products are identified via product identifiers. Apple clearly defines the types of products that may be sold via the Store Kit framework. More on this later but keep in mind that it is very important to read through all of the documentation.
Over the next few weeks, we will be talking about the mobile application development process. To give you some background on me, I started with C++ in 1997. At the time, I mostly used the GNU compilers on FreeBSD and Linux and Sun workshop on a SPARCstation. And for the C++ courses I was taking, Borland in a Win environment. To get up to speed on iPhone OS, I purchased the WWDC 2009 sessions and watched them all (at the gym on my iPhone). I attended the iPhone tech talk tour in San Jose and I listened to any and all of the available iPhone development podcasts (at the gym) that I could find. I poured through all of the books that I could get my hands on. I read the iPhone OS programming guides on the iPhone OS developer connection. XCode also has integrated documentation for the Apple API's. I've always used vi/vim with ctags, so I had to force myself to watch the xcode screencasts (available via third-party on the Internet). There are lots of great xcode tips out there and you can print out a summary page with all of the cute shortcuts - if you are so inclined to do so. I found it quite helpful to print the shortcuts page. Design patterns are at the core of how the Cocoa frameworks interact. Apple provides solid documentation on Cocoa design patterns in the iPhone OS Reference Library and these patterns can be understood at a high-level without delving into any code. This is my suggested approach. If you are not familiar with design patterns, the gang of four book (the GOF book as most often termed in the software world) - Design Patterns: Elements of Reusable Object-Oriented Software, by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, is the official source for design patterns. The theory behind Cocoa's design patterns can be found in this book and for that matter, the use of design patterns in any modern object-oriented software system can be found in this book. The GOF book describes design patterns in a language independent manner. Familiarizing yourself with Cocoa's design patterns from a language independent perspective is a key starting point for building an iPhone application.
Continuing on the topic of cocoa design patterns, Apple's frameworks are designed for re-use. There are folks out there who are very experienced with design patterns and are able to layer design patterns on top of the cocoa frameworks. These individuals understand all of the memory implications. They understand MVC backward and forward and they understand the Cocoa design patterns backward and forward. If you do not fall into this category, then keep it simple. If there is one problem that I have seen across forums and code samples on the Internet, it is that of "fighting the apis". It is very important to not fight the apis. The apple documentation describes the proper use of the iPhone OS frameworks and associated API's. The Apple developer connection provides plenty of sample code so it is important to read through this code to understand the proper use of the iPhone OS APIs.
From inception to app store approval, we spent 5 months developing iLogMiles. While this may seem longer than the standard "2 month" development lifecycle, I can assure you that iLogMiles uses a wide variety of the iPhone OS frameworks. In the coming weeks, we plan on covering the development process from start to finish. I will be talking about code organization, memory management, xcode features, core data, performance optimization, debugging, iterative development, and any other topics that we see useful.
Apple, the Apple logo, iPhone, iPod touch, and iTunes are trademarks of Apple Inc., registered in the U.S. and other countries. iPad is a trademark of Apple Inc. App Store is a service mark of Apple Inc.
By
Bryan R. Hinton
Monday, February 1, 2010
Objective-C Categories
Categories are a powerful tool for extending the functionality of existing classes in Objective-C. However, it's important to follow Apple's guidelines when using them to avoid any undefined behavior.
Categories allow you to extend the functionality of an existing class without subclassing. Before I get into an example, here are a few guidelines to follow when using categories. Apple recommends to not use categories to override methods in other classes, inherited classes, or other categories. Categories are not meant to add instance variables to classes either. If used in such contexts, the behavior of a program is undefined. Apple recommends using categories for adding functionality to large, existing classes and then grouping that functionality accordingly. Overriding existing Cocoa methods that are inherited from within the Cocoa hierarchy is a no-no as is extending methods that are already part of a category within the Cocoa hierarchy. Categories allow the addition of methods to existing classes while retaining the scope of class instance variables. So here's an example. Categories are quite simple to create and very powerful.
myCategory.h
@interface UIViewController(myCategory) - (void)messageName:(double)arga keywordA:(double)argb keywordB:(NSString *)argc;myCategory.m
#import "myController.h"
@implementation UIViewController(myCategory)
- (void)messageName:(double)arga keywordA:(double)argb keywordB:(NSString *)argc {
// process arguments
// do something with ivar
// ...
}myController.h#import "myCategory.h"
@interface myController : UIViewController {
someType ivar;
}
- (void) someMessage;
...
@endmyController.m#import "myController.h"
@implementation
- (void)someMessage {
[self messageName]; // send message to self - i.e. call method defined by category
}
@end
By
Bryan R. Hinton
Monday, January 25, 2010
Core Data
I was fortunate to attend the iPhone Tech Talk conference, hosted by Apple Inc. in San Jose last October, 2009. One of the sessions I attended was on Core Data. At a very high level, Core Data is an Apple™ software framework for storing data on the computer - namely, the iPhone and the Mac. Please note that I don't plan on covering database design theory in this post so I'm assuming a familiarity with it.
Core data is an Object graph management and persistence framework. Core data maintains a graph of objects (otherwise known as a managed object graph or managed object context). References to managed objects are stored in the object graph. Core Data serializes these objects and persists them to a data store. Core data can use several different underlying data stores - xml, binary, sqlite. The managed object context is a key concept in Core Data, as it provides a proxy between the application objects and the underlying data store(s). The persistent store coordinator is responsible for managing the relationship between the managed object context and the persistent store(s). A managed object context wraps a collection of managed objects and sits on top of a persistent store coordinator which houses one or more persistent stores. A managed object context is created via an instance of the NSManagedObjectContext class.It's important to note that the Core Data sqlite file should not be accessed directly from within code, as the schema may change and it's not intended for direct manipulation.
In Summary, a managed object context is a collection of managed objects. Managed objects are instances of the entities that are defined in the data model. A managed object represents a row of data in a table. The managed objects context sits on top of a persistent store coordinator. The managed object context is a proxy between the objects in the application and the underlying data store(s). A persistent object store is represented by a xml, binary, or sqlite file. The persistent store coordinator sits between the managed object context and the persistent store(s). The persistent store coordinator in turn abstracts one or more persistent object stores as a single aggregate. Please note, a persistent store coordinator can only be associated with one managed object context. But a persistent store coordinator can have many persistent object stores. The managed object context talks directly to the persistent store coordinator. The managed object context is a scratchpad of managed objects.
After working with Core Data, it is a worthwhile exercise to open the core data sqlite file and dig through the tables. A quick look through the tables will explain how Apple is handling referential integrity. The Core Data sqlite file is "not meant" to be accessed directly from within code. Do not modify it or attempt to write queries against it. There is no guarantee that the schema will not change. Core Data is meant to abstract implementation dependent details (i.e. the implementation may change upon any future releases).
The following notes were derived from three sources: Apple WWDC 2009, the iPhone Core Data session in San Jose last October, and my experience working with object/relational databases.
After working with Core Data, it is a worthwhile exercise to open the core data sqlite file and dig through the tables. A quick look through the tables will explain how Apple is handling referential integrity. The Core Data sqlite file is "not meant" to be accessed directly from within code. Do not modify it or attempt to write queries against it. There is no guarantee that the schema will not change. Core Data is meant to abstract implementation dependent details (i.e. the implementation may change upon any future releases).
The following notes were derived from three sources: Apple WWDC 2009, the iPhone Core Data session in San Jose last October, and my experience working with object/relational databases.
Creating a Core Data Stack
- Load the Data Model
- Create the persistent store coordinator and set the model
- Add the data store
- Create a managed object context and set the persistent store coordinator
Core Data Model
- A Core Data model does not need to be normalized. Define data contracts early on. Don't over tighten the data model - design with usage patterns in mind. Define entities or tables with Xcode's visual editor. (core data objects represent rows in these tables). In other words, a Core Data object (row of data) of type PERSON is an instance of the PERSON entity in the Core Data model. Of course, a person object will have relationships with other objects in the database. A Core Data Managed object context manages these objects, their relationships, and persists them to the underlying data store (via the persistent store coordinator). Core Data is simple to setup. Don't complicate it.
- NSManagedObjectContext defines the verbs (CRUD or fetch/insert/delete/save) that act on managed objects (nouns)
- NSManagedObjectContext tracks changes to properties in the data model
- Core Data fits very well in the MVC architecture.
- An NSPersistentStoreCoordinator can have multiple persistent stores(NSPersistentStore) and multiple Managed Object Contexts can use a single NSPersistentStoreCoordinator
- It is possible to delete the sqlite store file if the stored data needs to be deleted. This file is stored in $HOME/Library/Application Support/iPhone Simulator/User/Applications/SOMEAPPLICATIONGUID/Documents
- Use Core Data migration or built-in versioning for model changes. easy to use. zero code.
Multithreading
- Core data assigns one managed object context per thread; use notifcations for changes made to managed objects within a managed object context.
- never pass managed objects between threads, pass object IDs (actually the primary key)
- within a thread, create a new Managed object context and then tell the new Managed object context what objects IDs to fetch
Fetching
- Use NSFetchedResultsController (break up fetches into sections) - it is fast, efficient, and easy.
- Create an NSFetchedResultsController
- Set the fetch request, predicate, and sort descriptors. fetch request provides predicate and sort descriptors). fetch request and predicate are immutable.
- the keypath returns the section name (section information is cached)
- set yourself as its delegate and implement your tableview methods
- Create separate controllers for different data sets
NSFetchRequest
- set the entity that you want to fetch against
- create NSEntityDescription - provide managed object context and entity you want to work with
- set the predicate (if applicable)
- Fetch - pass in fetch and error
Batching
- setup fetch request (only need some of the objects at a time so setBatchSize:)
- set batch size
- you get back an NSArray of results
Prefetching
- use setRelationshipKeyPathsForPrefetching:
Managed Object Interaction
- NSManagedObject
- Use accessors
- Objective-C properties
- To Many relationships are sets
- NSManagedObjects provides Key value coding/observing out of the box
- Managed Object Context observes object changes. Leverage change tracking notifcations
- Register for Object Level Changes (KVO)
- Register for Graph Level Changes (NSNotifications)
By
Bryan R. Hinton
Sunday, January 24, 2010
Iterate over Custom UITableViews
UITableViews on the iPhone often contain a variable number of cells. The following code snippet should be helpful for those who need to iterate over a variable length table and subsequently read or set data model and cell state.
//loop through cells in each section and make sure the data model is in sync
NSInteger numSections = [self numberOfSectionsInTableView:someTable];
for (NSInteger s = 0; s < numSections; s++) {
NSLog(@"Section %d of %d", s, numSections);
NSInteger numRowsInSection = [self tableView:someTable numberOfRowsInSection:s];
for (NSInteger r = 0; r < numRowsInSection; r++) {
NSLog(@"Row %d of %d", r, numRowsInSection);
MyCell *cell = (MyCell *)[someTable cellForRowAtIndexPath:
[NSIndexPath indexPathForRow:r inSection:s]];
DataModelObject *theObject = (DataModelObject *)[fetchedResultsController objectAtIndexPath:
[NSIndexPath indexPathForRow:r inSection:s]];
[theObject setValue:[NSNumber numberWithBool:MyCell.someProperty] forKey:@"someEntityKey"];
}
}
By
Bryan R. Hinton
Monday, December 7, 2009
Mobile Technology - Where Are We Going?
OS X and iOS contain open source components.
- Open Source Components that ship with Mac OS X 10.6.2 and their corresponding Open Source Projects (src avail for dl/browse) - http://www.apple.com/opensource/
- Open Source Components that ship with Mac OS X 10.6.2, iPhone OS 3.1.2 and Developer Tools 3.2.1 (source avail for dl) Also browseable by prior versions - http://opensource.apple.com
On to Android...so what is it? Android is an open-source mobile operating system. It uses a modified Linux 2.6 Kernel - tuned for embedded devices. To quote the Android Web site - "Android relies on Linux version 2.6 for core system services such as security, memory management, process management, network stack, and driver model. The kernel also acts as an abstraction layer between the hardware and the rest of the software stack." (http://developer.android.com/guide/basics/what-is-android.html).
Linux is known for its stability, security, and flexibility. It has become a popular choice for servers, workstations, and even desktops. Linux is widely used in the cloud computing industry and powers many of the world's largest websites and services. It has a strong developer community and a vast selection of open-source software available for free. Additionally, Linux is highly customizable, allowing users to modify and tailor their operating system to their specific needs.
Linux (GNU/Linux) is a UNIX-like operating system that comes in many flavors or distributions. The distributions typically differ in how package (applications) distribution is handled. Oftentimes, a custom or stripped down kernel is shipped with the distribution. There are over 600 GNU/Linux distributions. They run on a myriad of devices from embedded controllers to large scale grids and supercomputers. GNU/Linux has been around a while - Linuz Torvalds coined the term in '91. The GNU Project, created by Richard Stallman in 1983, had the goal of creating a complete Unix-compatible software system composed entirely of free software". Hence came GNU/Linux. Read more here - http://www.gnu.org/.
To clarify, the GNU/Linux operating system consists of the Linux kernel (created by Linus Torvalds in 1991) and GNU software, which was developed by the GNU Project (launched by Richard Stallman in 1983) and includes tools such as the GNU C Compiler (GCC), GNU Debugger (GDB), and Bash shell.
Linux (GNU/Linux) is a UNIX-like operating system that comes in many flavors or distributions. The distributions typically differ in how package (applications) distribution is handled. Oftentimes, a custom or stripped down kernel is shipped with the distribution. There are over 600 GNU/Linux distributions. They run on a myriad of devices from embedded controllers to large scale grids and supercomputers. GNU/Linux has been around a while - Linuz Torvalds coined the term in '91. The GNU Project, created by Richard Stallman in 1983, had the goal of creating a complete Unix-compatible software system composed entirely of free software". Hence came GNU/Linux. Read more here - http://www.gnu.org/.
To clarify, the GNU/Linux operating system consists of the Linux kernel (created by Linus Torvalds in 1991) and GNU software, which was developed by the GNU Project (launched by Richard Stallman in 1983) and includes tools such as the GNU C Compiler (GCC), GNU Debugger (GDB), and Bash shell.
The Linux kernel was based on the Minix operating system and Unix principles, but it was not directly derived from SVR4 or BSD. It was created as a free and open-source alternative to commercial operating systems and has since become widely used on servers, desktops, and mobile devices.
The process, thread, and memory models in GNU/Linux are based on the Unix principles and have been tested and refined over several decades. They are known for their stability and scalability, making GNU/Linux a popular choice for high-performance computing and enterprise applications.
The process, thread, and memory models in GNU/Linux are based on the Unix principles and have been tested and refined over several decades. They are known for their stability and scalability, making GNU/Linux a popular choice for high-performance computing and enterprise applications.
Many GNU userland tools, such as Bash, GCC, GNU coreutils, and GNU Emacs, are included in most GNU/Linux distributions. These tools are often used by developers and system administrators to perform various tasks, such as writing and compiling code, managing files and directories, and configuring system settings. The availability of these tools on GNU/Linux systems is a result of the close relationship between the GNU Project and the Linux kernel.
Android uses a modified version of the Linux kernel, which is a Unix-like operating system kernel, but its userland libraries are not based on GNU/Linux. Instead, Android uses a custom, BSD-like userland developed by Google. This allows Android to have more control over the code and reduce dependencies on GNU/Linux. However, Android still maintains compatibility with many GNU/Linux tools and libraries.
In addition, Android is built with security in mind. It includes various security features such as app sandboxing, which isolates each application and its data from other applications, and permissions, which allow users to control what data and resources an application can access on their device.
As opposed to Objective-C, which developers use to write applications for the iPhone, Java is used for writing applications on Android devices. Android provides libraries on top of the kernel for 2D and 3D rendering, type support, sqlite access, media, libc (actually called Bionic), etc. While the Linux kernel provides memory management, process and threading support, etc., Google uses a byte-code interpreter (well, not exactly, but similar) called DalvikVM which transforms java class files into a second type of byte code format - .dex. An Android application gets loaded into a single process and is allocated a DalvikVM instance. DavlikVM was chosen because of its memory/processor efficiency on embedded devices.
Android also provides an application framework that allows developers to use various pre-built components such as activities, services, content providers, and broadcast receivers to build their applications. These components help developers to manage the lifecycle of their applications and handle various system events, such as incoming phone calls or text messages.
The Android SDK (Software Development Kit) provides all the necessary tools and libraries for developers to build, test, and debug their applications. The SDK includes an emulator, which allows developers to test their applications on a virtual device before deploying them on an actual Android device.In addition, Android is built with security in mind. It includes various security features such as app sandboxing, which isolates each application and its data from other applications, and permissions, which allow users to control what data and resources an application can access on their device.
In 2007, Google created the open handset alliance with the following objective: "The Open Handset Alliance is a group of 47 technology and mobile companies who have come together to accelerate innovation in mobile and offer consumers a richer, less expensive, and better mobile experience. Together we have developed Android™, the first complete, open, and free mobile platform. We are committed to commercially deploy handsets and services using the Android Platform (http://www.openhandsetalliance.com)". By working together, the alliance aims to create a more open and innovative mobile ecosystem that benefits both developers and consumers.
The list of member companies is huge including but not limited to, TI, NVidia, Quallcomm, Acer, Sprint, Toshiba, LG, etc. Shortly thereafter, in 2008, Google released the Android source code under an Apache license. This move attracted and will continue to attract top developers from all over the world who are comfortable with the Java programming language in a Linux type environment.
By releasing the Android source code under an Apache license, Google allowed developers to modify and distribute the code, which led to the creation of numerous custom ROMs (modified versions of the Android operating system) and a thriving community of developers. It also allowed manufacturers to use the Android operating system on their devices without any licensing fees, which helped to accelerate the adoption of Android.
Google's initiative with the release of the Android source code and the formation of the Open Handset Alliance were huge moves. The Open Handset Alliance will aid its member companies with deploying the Android operating system on their respective devices. In other words, we will see the Android operating system on many different brands of mobile phones.
The open-source nature of Android and the formation of the Open Handset Alliance have allowed for the widespread adoption of the Android operating system across many different mobile devices from various manufacturers. This has helped to create a more competitive and diverse mobile market, which ultimately benefits consumers.
By the end of 2009, there will be over 15 mobile devices with the Android operating system. There are over 15,000 apps on the Android market (as of November 2009). There were 10,000 apps on the Android Market 2 months ago. There are over 100,000 apps on the Apple app store (as of November 2009). The Apple app store is 16 months old. There have been over 2 billion total downloads since inception.
The Android operating system is solid and it isn't going anywhere. Here are some good resources on the Internet.
Getting Started with Android Development
The GNU Operating System
Free Software Foundation
Open Handset Alliance
What is Android?
Dalvik Virtual Machine insights
Android Notes
Inside the Android Application Framework
By
Bryan R. Hinton