000webhost

Web hosting

Tuesday, July 3, 2012

iPod Nano Device Sanitisation

Continuing my work on my, "Cloud and Internet Security" report. I'll revise my earlier estimate and say that its likely to be 500+ pages though the scope will depend on other factors. One of the issues that I wanted to explore was mobile device backup and sanitisation. In particular my iPod Nano. It sounds easy but if you haven't got a recent backup,  you're aware of how strongly iTunes and iPod devices are integrated, and have taken a look at the file directory structure of an iPod you'll realise just how awkward this can be.

Actual music files are stored on the device in the following format:
/media/sdd1/iPod_Control/Music/F[:number:]{2}/[:alpha:]{4}.mp3

The command, "file ZRJR.mp3" yields the following:
ZRJR.mp3: Audio file with ID3 version 2.2.0, contains: MPEG ADTS, layer III, v1, 128 kbps, 44.1 kHz, JntStereo


knoppix@system:~/iPod_Control/iTunes/iTunes Library.itlp$ file *
Dynamic.itdb:       SQLite 3.x database
Extras.itdb:        SQLite 3.x database
Genius.itdb:        SQLite 3.x database, user version 2
Library.itdb:       SQLite 3.x database, user version 23
Locations.itdb:     SQLite 3.x database
Locations.itdb.cbk: data

http://www.macworld.com/article/1139974/itunes_files.html

Plugging the device in under Linux and mounting as as a USB mass storage device works but so does iPod/iPhone Explorer under Windows as well (though they have some quirks that need to be fixed (I picked up some unhandled exception during my experimentation)). They remind me a lot of Explore2FS which we had to use in the early days of Windows/Linux interoprability. They also allow you to drag and drop though conversion from stored file format to MP3 will take some time. 

TeraCopy (a more optimal version of Windows Explorer's copy functionality) sufferred from a number of stalls and the only notification that I had was a balloon popup on the toolbar indicating that it was due to the process of conversion. Would have liked to see something more informative regarding progress.

Some of the main database table structures extracted from various onboard files are provided below (This was extracted by using a SQLite Browser/Manager and then exporting it to SQL code.).

CREATE TABLE album (pid INTEGER NOT NULL, kind INTEGER, artwork_status INTEGER, artwork_item_pid INTEGER, artist_pid INTEGER, user_rating INTEGER, name TEXT, name_order INTEGER, all_compilations INTEGER, feed_url TEXT, season_number INTEGER, is_unknown INTEGER DEFAULT 0, has_songs INTEGER DEFAULT 0, has_music_videos INTEGER DEFAULT 0, sort_order INTEGER DEFAULT 0, artist_order INTEGER DEFAULT 0, has_any_compilations INTEGER DEFAULT 0, sort_name TEXT, artist_count_calc INTEGER DEFAULT 0 NOT NULL, PRIMARY KEY (pid));

CREATE TABLE artist (pid INTEGER NOT NULL, kind INTEGER, artwork_status INTEGER, artwork_album_pid INTEGER, name TEXT, name_order INTEGER, sort_name TEXT, is_unknown INTEGER DEFAULT 0, has_songs INTEGER DEFAULT 0, has_music_videos INTEGER DEFAULT 0, PRIMARY KEY (pid));

CREATE TABLE avformat_info (item_pid INTEGER NOT NULL, sub_id INTEGER NOT NULL DEFAULT 0, audio_format INTEGER, bit_rate INTEGER DEFAULT 0, sample_rate REAL DEFAULT 0, duration INTEGER, gapless_heuristic_info INTEGER, gapless_encoding_delay INTEGER, gapless_encoding_drain INTEGER, gapless_last_frame_resynch INTEGER, analysis_inhibit_flags INTEGER, audio_fingerprint INTEGER, volume_normalization_energy INTEGER, PRIMARY KEY (item_pid,sub_id));

CREATE TABLE category_map (id INTEGER NOT NULL, category TEXT NOT NULL, PRIMARY KEY (id), UNIQUE (category));
CREATE TABLE composer (pid INTEGER NOT NULL, name TEXT, name_order INTEGER, sort_name TEXT, is_unknown INTEGER DEFAULT 0, has_music INTEGER DEFAULT 0, PRIMARY KEY (pid));
CREATE TABLE genre_map (id INTEGER NOT NULL, genre TEXT NOT NULL, genre_order INTEGER DEFAULT 0, is_unknown INTEGER DEFAULT 0, has_music INTEGER DEFAULT 0, artist_count_calc INTEGER DEFAULT 0 NOT NULL, album_count_calc  INTEGER DEFAULT 0 NOT NULL, PRIMARY KEY (id), UNIQUE (genre));

CREATE TABLE item (pid INTEGER NOT NULL, revision_level INTEGER, media_kind INTEGER DEFAULT 0, is_song INTEGER DEFAULT 0, is_audio_book INTEGER DEFAULT 0, is_music_video INTEGER DEFAULT 0, is_movie INTEGER DEFAULT 0, is_tv_show INTEGER DEFAULT 0, is_ringtone INTEGER DEFAULT 0, is_voice_memo INTEGER DEFAULT 0, is_book INTEGER DEFAULT 0, is_rental INTEGER DEFAULT 0, is_itunes_u INTEGER DEFAULT 0, is_digital_booklet INTEGER DEFAULT 0, is_podcast INTEGER DEFAULT 0, date_modified INTEGER DEFAULT 0, date_backed_up INTEGER DEFAULT 0, year INTEGER DEFAULT 0, content_rating INTEGER DEFAULT 0, content_rating_level INTEGER DEFAULT 0, is_compilation INTEGER, is_user_disabled INTEGER DEFAULT 0, remember_bookmark INTEGER DEFAULT 0, exclude_from_shuffle INTEGER DEFAULT 0, part_of_gapless_album INTEGER DEFAULT 0, chosen_by_auto_fill INTEGER DEFAULT 0, artwork_status INTEGER, artwork_cache_id INTEGER DEFAULT 0, start_time_ms REAL DEFAULT 0, stop_time_ms REAL DEFAULT 0, total_time_ms REAL DEFAULT 0, total_burn_time_ms REAL, track_number INTEGER DEFAULT 0, track_count INTEGER DEFAULT 0, disc_number INTEGER DEFAULT 0, disc_count INTEGER DEFAULT 0, bpm INTEGER DEFAULT 0, relative_volume INTEGER, eq_preset TEXT, radio_stream_status TEXT, genius_id INTEGER DEFAULT 0, genre_id INTEGER DEFAULT 0, category_id INTEGER DEFAULT 0, album_pid INTEGER DEFAULT 0, artist_pid INTEGER DEFAULT 0, composer_pid INTEGER DEFAULT 0, title TEXT, artist TEXT, album TEXT, album_artist TEXT, composer TEXT, sort_title TEXT, sort_artist TEXT, sort_album TEXT, sort_album_artist TEXT, sort_composer TEXT, title_order INTEGER, artist_order INTEGER, album_order INTEGER, genre_order INTEGER, composer_order INTEGER, album_artist_order INTEGER, album_by_artist_order INTEGER, series_name_order INTEGER, comment TEXT, grouping TEXT, description TEXT, description_long TEXT, collection_description TEXT, copyright TEXT, track_artist_pid INTEGER DEFAULT 0, physical_order INTEGER, has_lyrics INTEGER DEFAULT 0, date_released INTEGER DEFAULT 0, PRIMARY KEY (pid));

CREATE TABLE item_to_container (item_pid INTEGER, container_pid INTEGER, physical_order INTEGER, shuffle_order INTEGER);

CREATE TABLE video_info (item_pid INTEGER NOT NULL, has_alternate_audio INTEGER, has_subtitles INTEGER, characteristics_valid INTEGER, has_closed_captions INTEGER, is_self_contained INTEGER, is_compressed INTEGER, is_anamorphic INTEGER, is_hd INTEGER, season_number INTEGER, audio_language INTEGER, audio_track_index INTEGER, audio_track_id INTEGER, subtitle_language INTEGER, subtitle_track_index INTEGER, subtitle_track_id INTEGER, series_name TEXT, sort_series_name TEXT, episode_id TEXT, episode_sort_id INTEGER, network_name TEXT, extended_content_rating TEXT, movie_info TEXT, PRIMARY KEY (item_pid));


My guess is that they used the:

/media/sdd1/iPod_Control/Music/F[:number:]{2}/[:alpha:]{4}.mp3

structure as a means of avoiding filename clashes. iTunes basically manages various files and databases to keep track of where everything is. While its clear that there are alternatives they are not always perfect (I manually deleted a file from a my iPod and one of them was unable to correctly update the database. There has been some difficulty in the Open Source world with reverse engineering the details obviously and one of the solutions I've seen was adding/removing files using another program and then plugging it into iTunes to have it update its internal databases but you have to wonder about the logic of this as you're just adding another moving part to the already complex mix/interplay.) and have to be reverse engineered which can be time consuming and difficult as Apple continue to increase their use of anti-reverse engineering technologies (The more you dig the more you realise how drastically some organisation's thinking can veer off of what you believe to the most sensible/elegant option. During my sanitisation experiment a number of files kept on showing as being recoverable. My guess is that it has to do with the way in which files/directories are extracted and manipulated rather than remnant data though. You also obviously need to factor in design compromises and commercial considerations as well.).

CopyTransManager
CopyTransDoctor


It seems clear that the device itself is built off of industry standard software/hardware. Brilliantly integrated and packaged though. Underlying filesystem format is either HPFS or FAT32 depending on the operating system from which the device is setup from.

Tried using various wiping programs. However, it soon became clear that there  other issues at play here. Almost like there's another layer in between the operating system and the underlying filesystem. Not sure whether this is limited to my system (I have a Windows system that is dedicated to pure experimentation so I can't be entirely sure about it being true to other's experiences. I seemed to have no issues with interfacing with the underlying filesystem on Linux but that's not a 'standard environment'.) though.

AxCrypt - Shred and Delete
Shredding of ''????" failed,
A device attacked to the system is not functioning.

Eraser - Erase
Error - The file or directory is not a reparse point. (Exception from HRESULT: 0x80071126)

A file or directory can contain a reparse point, which is a collection of user-defined data. The format of this data is understood by the application which stores the data, and a file system filter, which you install to interpret the data and process the file. When an application sets a reparse point, it stores this data, plus a reparse tag, which uniquely identifies the data it is storing. When the file system opens a file with a reparse point, it attempts to find the file system filter associated with the data format identified by the reparse tag. If a file system filter is found, the filter processes the file as directed by the reparse data. If a file system filter is not found, the file open operation fails.

iPod Explorer - Simple Deletion
Obvious that one deletion of file is possible but it doesn't deal with updating the meta-data/database. It still appears in the menuing system but attempting to play results in nothing, not even an error.

Obviously, after deletion I tried various methods of file recovery to see just how effective some of the mechanisms I employed were. As with most standard FAT based filesystems if there were no other intervention 100% file recovery is almost always possible. You need to use a wiping program to truly defend your iPod or other mobile devices against attack from computer forensics programs. 

root@system:/media/sdc1# photorec /log /debug /d photorec
PhotoRec 6.11, Data Recovery Utility, April 2009
Christophe GRENIER
http://www.cgsecurity.org

Disk /dev/sdd - 7889 MB / 7523 MiB (RO) - Apple iPod
     Partition                  Start        End    Size in sectors
 1 P FAT32                    0   1  3   127 204 49    1926015 [USER'S IPOD]
168 files saved in photorec directory.
Recovery aborted by the user.
mp3: 148 recovered
tx?: 10 recovered
sqlite: 4 recovered
txt: 3 recovered
mov: 2 recovered
gz: 1 recovered
[ Quit ]

root@system:/media/sdc1# vim /etc/scalpel/scalpel.conf
# MPEG Video
mpg     y       50000000        \x00\x00\x01\xba        \x00\x00\x01\xb9
mpg     y       50000000        \x00\x00\x01\xb3        \x00\x00\x01\xb7

root@system:/media/sdc1# scalpel -v /dev/sdd1 -o /media/sdc1/scalpel
/dev/sdd1:   9.8% |********|  740.0 MB  2:05:55 ETAOPENING /media/sdc1/scalpel/mpg-1-0/00003091.mpg
/dev/sdd1:  10.0% |*******|  750.0 MB  2:06:05 ETAOPENING /media/sdc1/scalpel/mpg-1-0/00003093.mpg
CLOSING /media/sdc1/scalpel/mpg-1-0/00003091.mpg
^C^C^C
Kill signal detected. Cleaning up...
CLOSING /media/sdc1/scalpel/mpg-1-0/00003090.mpg
^C
Kill signal detected. Cleaning up...
CLOSING /media/sdc1/scalpel/mpg-1-0/00003089.mpg
^C
Kill signal detected. Cleaning up...
CLOSING /media/sdc1/scalpel/mpg-0-0/00000055.mpg
OPENING /media/sdc1/scalpel/mpg-1-0/00003092.mpg
CLOSING /media/sdc1/scalpel/mpg-1-0/00003092.mpg
/dev/sdd1:  10.1% |*********|  760.0 MB  2:04:56 ETACleaning up...
 Caught signal: Interrupt. Program is terminating early 

Erase Free Space (Eraser) option has reduced the significantly the chances of file recovery even though file integrity recovery indicators state (CopyTransDoctor) that 100% recovery is possible this is certainly not the case. My results indicate that while partial wiping is possible it requires specialised tools and/or knowledge (I've seen some mobile applications that are specifically designed for this purpose) and clearly if you have raw/jailbroken access to the file system you have many options/utilities available to you.

Not surprisingly given these issues, Apple and others have come up with their own wiping/encryption functionality. 


Simulated the following wiping (free space only. To run on the entire device just drop the digit.) algorithms,
- British HMG IS5 (3 passes)
- US DoD 5220.22-M (8-306./E)(3 passes)
by running the following:

dd if=/dev/zero of=/media/sde1/ipod.wipe
cat /dev/zero | tr '\0' '\1' | dd of=/media/sde1
dd if=/dev/urandom of=/media/sde1/ipod.wipe

Note that you will run into the 4GB filesystem limitation with FAT32 so just create a another file to get around this problem. Namely,

dd if=/dev/zero of=/media/sde1/ipod.wipe
dd if=/dev/zero of=/media/sde1/ipod.wipe1
dd if=/dev/zero of=/media/sde1/ipod.wipe2
etc...

It seemed to fairly obvious that a single wipe (British HMG IS5 (1 pass) or US DoD 5220.22-M (8-306./E)(1 pass) which basically amounts to a single pass of zeros) was enough to throw off the various programs that I was using to examine the device though (I was only using free and Open Source tools to simulate a worst case scenario.).

- scalpel
- photorec
- testdisk
- Recuva
- Glary Utilities
- diskinv

Not surprisingly, ports, and installations of Linux have been attempted. The main limitations seem to be the anti-reverse engineering technologies implemented by Apple though.



Given the fact that it basically is a USB storage device you can use a lot of standard utilities on the device such as DBAN and BCWipe.


Luckily recovery mode is there for you to be able to restore functionality most of the time (It's fairly easy to trigger. I obviously corrupted the filesystem a number of times through my experiments with dd and also attempting to reformat, use other programs, and operating systems on my device. Just plug it into a computer running iTunes and it'll do a complete reset of the device for you.).

http://www.tuaw.com/2010/12/31/discovering-ipad-nano-recovery-mode/

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , , , , ,

Friday, June 8, 2012

Crash Course in Security

My preliminary guesswork regarding my recent LibreOffice crashes were more correct accurate than I realised. What's actually happening is that in areas where the picture is larger/where strange cut/paste decisions are being made the program is actually having difficulties processing this data. This is reflected by a stall of several minutes (or eventual crash) in user space (workable under Microsoft Word but not under LibreOffice/OpenOffice at the moment). I've fiddled around with removing, changing the format, and compression level of the pictures themselves but obviously this does not scale well on large files (I'm not sure how long this is going to be yet.).

There does seem to be an option in Microsoft Word which will allow you to compress all images in the file though (there seems to be similar work occurring in LibreOffice/OpenOffice as well). I was able to achieve a size reduction from 5906KB to 5023KB. A substantial saving and one which has allowed me to re-open it (I've been standardising fonts/styles as well) and use it in OpenOffice/LibreOffice as well. Something else which I've realised is that where I've tracked changes I've made or the program has made odd formatting/styling decisions which have compounded the problem.

http://www.libreoffice.org/features/extensions/
http://user.services.openoffice.org/en/forum/viewtopic.php?t=6918
http://user.services.openoffice.org/en/forum/viewtopic.php?t=3665


In some other work had to deal with some browser compatibility issues (IE6). Something I came across was the history different in support levels between PNG as opposed to other formats in web browsers. If you've ever heard of or used ImagaMagick then you'll realise just how useful it is.

for i in *.jpg; do temp=`echo $i | sed s/jpg/png/`; convert $i $temp; done
for i in *.jpg; do temp=`echo $i | sed s/jpg/gif/`; convert $i $temp; done

Another thing you may notice is that even when using tools that are designed to deliberately compress image files depending on the algorithm there's only so far that you can take it.

http://pmt.sourceforge.net/pngcrush/

At a certain point it just comes down to the raw dimensions of the file itself. This is particularly the case with files that use lossless compression algorithms.

for i in *.jpg; do temp=`echo $i | sed s/jpg/png/`; convert -resize 25% $i $temp; done

The following are useful utilities for dealing with PDF files. I recently needed them to append multiple PDF's to one another/convert to other formats and didn't want to use a full blown editor. 

at
Labels: , , , , , ,

Monday, June 4, 2012

I'm not a Crash Test Dummy!

I was going to write the following.

Only a short while back I never thought that I would be able to reach this point (due to continual LibreOffice crashes) but my document on 'Cloud and Internet Security' has reached 90K+, 318+ pages without much of a problem. There's still a bit of work and a lot of editing to do though.

However, I'll have to change it to this.

The crashes resumed after copying/pasting in a new element. I can't be sure whether it was a picture or just pure text as the crash generally only occurs during automated autosaves so what is supposed to save you is actually causing you the problem. I've done a bit of research and its basically what I surmised earlier. It has something to do with formats/styles but not exactly in a way that you would think. It doesn't actually have to do with the content itself as the total number of formats/styles that are part of the document. The more you have the more likely the crashes are to occur.

Obviously, the cleanest way was to convert it to another format so that I could open it with a more reliable word processor. Ironically, even AbiWord, OpenOffice and CLI based tools had trouble with (opening, saving, and converting would result in timeouts or crashes) the document. Eventually, I recalled an add-on for Microsoft Word that would allow me to open ODT documents which seemed to work perfectly fine and also allowed me to convert to DOC format (converting to the more up to date DOCX format would result in freezes). I've since switched to the more stable, tested, and resilient Microsoft Word for now (LibreOffice still has problems with the new document but I've noticed that a lot of strange additional formatting decisions were made during cutting/pasting once I saw the Navigation Panel under Word. I will switch back to LibreOffice once they've figured some of these issues out.).

http://stackoverflow.com/questions/4017886/how-to-convert-odt-to-doc-rtf-without-openoffice-org
http://davelargo.blogspot.com.au/2012/05/libreoffice-live-better-crash.html
http://forums.opensuse.org/english/get-technical-help-here/applications/466453-libreoffice-writer-freezes-100-cpu-utilisation.html
http://scottnesbitt.net/ubuntublog/?p=1018

As an aside, I sometimes crimp/re-crimp my own network cables. Try to go after quality crimping tools/equipment. I just had a small batch of connectors just break. I suspect it partly had to do with the ambient temperature (very low) but I think its mostly down to QA problems. Please note that while stripping the network cable further back may allow you to more easily insert the cable into the connector it also means that it is more likely to lead to signal leakage/interference between individual wires in the long run (I've learnt this through personal experience).

Also, if you've ever come across the 'nm-applet' that sits on your taskbar and manages network connections (under UNIX/Linux) but it refuses to pickup a new connection try removing and reinserting the relevant network kernel module via 'rmmod' and 'modprobe'.

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , , , , ,

Monday, May 21, 2012

Temperamental Television

Until recently I didn't realise just how sensitive DTV (Digital Television) actually was. My experience tells me that an amplifier is highly recommended (unless your dongle/device/set top box is exceedingly good). The thing which has made me particularly curious though is the impact of local electrical appliances (such as power boards, heaters, and even networking devices. Removing or swapping particular devices can have an immediate impact on signal strength/integrity.) and the connections between the tuner, antenna, amplifier, etc... While I admit that some of the equipment in my setup could be of a higher standard the tolerances that we're talking about are borderline absurd. Changing the angle of connections is enough to change the signal strengh/integrity substantially. Moreover, signal integrity issues at certain points are extremely difficult to debug without specialised equipment (I've seen some digital TV strengh metres being sold but I haven't tried one as yet and the cost outweighs the possible gains. Admittedly, there are some signal strength measuring systems in the software itself but this is not ideal as it can not go into an arbitrary position in the pipeline.) and the only real feedback that you get is how 'choppy' the sound/picture is. I've found that moving the amplifier power point on to a less noisy circuit can be extremely helpful as well as changing the angle of the cable the being used to carry the signal into various devices (the amplifier is particularly susceptible to this problem and I've found it best to lay it on a flat surface). I may experiment with Microwave based transmitters (2.4/5GHz range) later on but early indications/reviews say that cable may actually be better. I've also been thinking about using an adapter to carry the signal over a non-coax type cable (or a medium that is less suspectible to issues related to cable bending) to hopefully reduce the impact of these problems but once again its a cost/benefit issue and I've seemed to have found a viable solution as is.

I've figured out that some of my recent problems with LibreOffice may also be due to issues with regards to externally inserted objects (such as images). I've been working on a new document (on 'Internet and Computer Security') and its 225+ pages/67K + words and there have been no random crashes as I've previously experienced when working on larger documents. I guess I'll have to write the text before hand, and add other objects in now from now on (at least until they fix the problem. This should also get me around another problem that I've found when editing text and there are images in the document (they don't move perfectly correctly with the text when cut/pasted.). The other alternative is to switch to another system of document management...

Have a theory about some connectivity issues I've been having lately. While playing around with MTU values has resulted in success I've also noticed something else. Sometimes, there seems to be a noticeable delay at certain critical points as though the traffic almost as though it is being buffered. Also, if there is other web based activity simultaneously this will help to get around the problem of stalls/stops at these particular points. Suspect there may be a timeout value that I may be able to tweak in my browser to help smooth out this intermittent problem (I've been dealing with it by dynamically shifting MTU values (depending on the circumstance) along with using HTTP/206 partial download capabilities but am thinking of building something more robust/automated/finding a better configuration for a more elegant solution.).

about:config
opera:config
http://100pulse.com/http-statuscode/206.jsp

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , , , , ,

Friday, May 18, 2012

Fujitsu Stylistic ST4120 Recovery without Recovery Disks

Someone recently brought me a laptop which was suffering from 'performance anxiety'. There were some obvious issues. File system fragmentation, low spec (P3 933MHz/768MB RAM/30GB HDD), power management issues (it was underclocking itself to a third of its maximum clock speed even when it was plugged into the local AC point), and general sub-standard upkeep (a lot of unrequired crud was installed on the system). I decided to re-install the system using the existing recovery partition by running D:\RECOVERY.EXE

Somehow, the original owner had managed to delete a critical file though (D:\INSTALL\WATERMRK.JPG) which was required for the program to run. After creating this file the program started but refused to run due to the recovery image being older than the currently installed operating system.

I initially tried to get around this problem by uninstalling SP3 (I discovered early in the piece that physical recovery disks were unavailable and the documentation said that you required a special boot disk to access the necessary files on the recovery partition. I didn't want to go with a clean XP Professional install because it woud be cleaner and more elegant if I could just get the recovery system to run.)(The process is similar to removing a Hotfix but more prolonged for obvious reasons).


However, I discovered that this was not enough (I found out that the recovery image was based on Windows XP Tablet PC Edition SP1) and that I needed to go further. Once SP3 was done removing itself I discovered that it also removed a critical uninstall file though. I tried to get around this through registry modifications of keys from the following two folders (I could only guess how the program was getting its version information because I couldn't load tools on to the system due to the system being so slow and unstable after removing SP3).

My Computer\HKLM\SOFTWARE\Microsoft\Updates\Windows XP\
My Computer\HKLM\SOFTWARE\Microsoft\Windows NT\Current Version\

Obviously, this was unsuccessful so I tried using a standard XP boot disk to boot to, 'Recovery Mode' (its three taps using the digitiser or the F12 button if you have a keyboard to activate the boot menu option). I noticed that changing into the various directories ("Access is denied.") and running D:\RECOVERY.EXE ("Command is not recognized.") was impossible though even though it was a FAT32 based filesystem partition and seemed to use standard PE32 executables on initial view (I expected to at least see the standard, "This program cannot be run in DOS mode." string and even saw this string in the initial header of the executable when viewing it via a text editor.).

user@system:/media/sdd1$ file RECOVERY.EXE
RECOVERY.EXE: PE32 executable for MS Windows (GUI) Intel 80386 32-bit


It seemed obvious that these were not standard executables though. I then tried using a Win 7 32-Bit recovery disk (Windows 7 recovery disks are temperamental with regards to the base system and the disk version (32-bit disk should be used with 32-bit base system) if you're curious). This allowed me to move into the required directory and also run D:\RECOVERY.EXE and kick off the recovery processes. Even though it seems possible or obvious that there may have been some co-operation between Microsoft and Fujitsu it looks as though the recovery process isn't quite as clean as it should be (progress meters aren't perfectly aligned, strange screens for particular processes, and so on). Nonetheless, it did what was required (it deleted/copied files to/from rather than formatting the existing Windows partition interestingly) and copied and ran the correct Windows installation files.

Some notes regarding this platform include the following:
- possible design flaw? On my particular system the power cable was quite simply too loose.
- compared to modern imaging techniques/systems (Norton Ghost) the restoration process was extremely slow.
- Windows Security button is the top left in portrait mode if you don't know. Hold it down for three seconds in order to activate Ctl-Alt-Del sequence on the Windows login screen.
- the left hand side of the screen in portrait mode can warm/heat up substantially as the hard drive is located directly underneath.
- even though it is extremely stylish, it seems to have a tiny battery and a high weight though this is synonymous with Windows tablets of this vintage.

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , , ,

Tuesday, April 17, 2012

Diverging Convergence

The 'Convergence Effect' report has become far larger than I had originally intended. Part of this was due to feedback which indicated that, "more detail" would make the document more useful, part of it has to do with me thinking of other ideas, and part it has to do with me working on a few side projects (some are related to 'Security') and this document provides me with a good outlet for some of my notes.

Another reason is that I've discovered that LibreOffice currently has some issues dealing with large documents (I'm around the 210 page/61000 word mark) that aren't in the native file format (Switching to working in the native 'odt' file format seems to work well but for reasons of compatibility I'd like access to other formats as well though.). The side effects of this can be data loss, inexplicable crashing, etc...

For these reasons, I will be splitting the 'Security' chapter (The new document will be more 'technically orientated' and will take offensive and defensive positions into consideration.) into a separate document (For reasons of coherence, I will maintain maintain much of the current content/state of the 'Convergence Effect' report.) and will consider these projects as largely, separate entities.

As an aside, I recently discovered a bug with cup-pdf which means that partial page loads aren't well supported under certain web browsers. I've worked around the issue for now by switching to a different browser.

I've also had some feedback that the following is a bit complex, http://dtbnguyen.blogspot.com/2011/03/eeepc-recovery-without-recovery.html
Since I recently damaged my recovery disk I've concur that the article can be a little difficult to read for someone who has never tried it before. Below is a simplified version:

1) Get 1005P recovery disc/image
2) Copy/extract contents to an 'arbitrary folder' which will be used to build a new image
3) Use ImgBurn to extract boot sector from recovery disc/image
4) Get recovery
5) Use a WIM utility to split the WIM image obtained from your 1015PD into sub 4GB files
6) Rename WIM files to asus.wim, asus2.wim, asus3.wim, etc... and drop them in the root directory of your 'arbitrary folder' where you stored the contents of the recovery image
7) Use ImgBurn to use boot image from disc/image on new recovery image
8) Create image
9) Write the image to an optical disc
10) Test whether it boots by using VM software such as VirtualBox or VMWare
at
Labels: ,

Friday, March 30, 2012

Automated Research

I never thought that I would be writing about this so soon. Apparently, the US Government announced an initiative  today/yesterday (depending on your location) that they would be pursuing an initiative into 'Big Data' and a form of automated research that I had been exploring in my 'Convergence' document (still working on it if you're wondering). While its certainly not a guarantee of success in this area I believe the implications may be far wider than any of us may ever imagine. Having algorithms which are capable of searching for masses of data, looking at it, extracting relationships and possibly other insights into it could result in an avalanche of human knowledge and discovery.

http://news.cnet.com/8301-11386_3-57406484-76/why-science-really-needs-big-data/
http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf

### Start Quote ####
OBAMA ADMINISTRATION UNVEILS BIG DATA INITIATIVE: ANNOUNCES $200 MILLION IN NEW R&D INVESTMENTS Aiming to make the most of the fast-growing volume of digital data, the Obama Administration today announced a Big Data Research and Development Initiative. By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the Nation's most pressing challenges. To launch the initiative, six Federal departments and agencies today announced more than $200 million in new commitments that, together, promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data. In the same way that past Federal investments in information-technology R&D led to dramatic advances in supercomputing and the creation of the Internet, the initiative we are launching today promises to transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security, said Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy. To make the most of this opportunity, the White House Office of Science and Technology Policy (OSTP) in concert with several Federal departments and agencies created the Big Data Research and Development Initiative to: Advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze, and share huge quantities of data. Harness these technologies to accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning; and Expand the workforce needed to develop and use Big Data technologies.
### End Quote ####
- as usual thanks to all of the individuals and groups who purchase and use my goods and services
at
Labels: ,

Thursday, March 29, 2012

Convergence Effect

If you've been watching this blog you may have noticed that there hasn't been a lot of activity lately. Part of this has to do with me working on other projects. One of these includes a report that I call the "Convergence Effect" which is basically a  follow up of "Building a Cloud Computing Service". If you're curious, both documents were/have been submitted to various organisations where more good can be done with them. Moreover, I consider both both works to be "WORKS IN PROGRESS" and I may make extensive alterations without reader notice. The latest versions are likely to be available here:



Convergence Effect
 
ABSTRACT
A while back I wrote a document called "Building a Cloud Service". It was basically a document detailing my past experiences and details some of the issues that a cloud company may face as it is being built and run. Based on what had transpired since, a lot of the concepts mentioned in that particular document are becoming widely adopted and/or are trending towards them. This is a continuation of that particular document and will attempt to analyse the issues that are faced as we move towards the cloud especially with regards to to media and IT convergence. Once again, we will use past experience, research, as well as current events trends in order to write this particular report. I hope that this document will prove to be equally useful and will provide an insight not only to the current state of affairs but will provide a blueprint for those who may be entering the sector as well as those who may be using resources/services from this particular sector. Please note that this document has gone through many revisions and drafts may have gone out over time. As such, there will be concepts that may have been picked up and adopted by some organisations (as was the case with the "Cloud" document with several technologies) while others may have simply broken cover while this document was being drafted and sent out for comment. It also has a more strategic/business slant when compared to the original document which was more technically orientated.

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , ,

Tuesday, February 14, 2012

Random Shuffling

I've recently been using a neglected version of a tool that did not possess randomisation capabilities. I decided to get around this problem by adding/creating a random shuffler so that subsequent results could not be as easily detected by automated systems. Some of the algorithms which I examined included the following.

 http://en.wikipedia.org/wiki/Shuffling
http://www.codinghorror.com/blog/2007/12/shuffling.html
http://amazoninterview.blogspot.com.au/2007/05/card-shuffle-algorithm.html
http://www.codeguru.com/forum/archive/index.php/t-339308.html
http://tekpool.wordpress.com/2006/10/06/shuffling-shuffle-a-deck-of-cards-knuth-shuffle/
http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=178050

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
at
Labels: , ,

Butter Fried Potatoes With Bacon Bits, Sour Cream, and Spring Onion

There used to be a dish at the Pancake Parlour that I took a particular liking to. Over time, I realised that there were 'economic' issues pertaining to this dish and I also thought that there were some modifications (basically changing the balance by using more of certain ingredients) that could be made to make for a better tasting recipe. Below are the results of my experimentation.

Ingredients
- butter
- olive oil
- bacon
- onion (optional)
- potatoes
- salt
- sour cream
- grated cheese
- spring onion

1) Brown/fry bacon bits or diced pieces of bacon in pan/skillet.
2) Remove from pan/skillet and place on plate (drain oil if desired using paper towel).
2) Chop potatoes/onions into cubes.
3) Use a combination of butter/oil (you can use butter only but I find that adding a little oil helps to change the burning point of the fat mixture which means that you can quick more quickly without having to worry as much about burning) to brown/fry the potatoes (if you prefer bigger potato chunks than you may want to boil them first to aide the cooking process). Note that even if you slightly burn the potatoes you can often just add another knob of butter and use slow, lower temperatures to 'fix the error'. Obviously, cooking it slowly at a lower temperature will soften the potatoes while cooking quickly at a higher temperature will make the potatoes crispier. Place onions in later because they cook more quickly.
4) Remove potatoes/onions from pan/skillet and place on plate (drain oil if desired using paper towel).
5) Drop grated cheese on top of potatoes/onions/bacon and put into microwave for 30 seconds to melt it or else use grill.
6) Salt to taste and use sour cream and chopped pieces of spring onion to garnish.
7) Goes very well with lemonade. Enjoy.

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/
at
Labels: , , ,

Wednesday, January 4, 2012

Personal Backup Solutions

I recently decided to implement an automated regime of backup (I tended to favour manual backups previously quite simply because there wasn't that much to backup). Below are some of my research notes.

BackupPC

Usability issues obviously that are similar to some of the issues that I've been facing working on a personal project. It feels like they've done a one to one translation of what's in the configuration file to the web interface and even though it works well it just feels a little rough around the edges. My expectations of user interface design is such that in most cases (especially consumer class applications) software should not require you to read a manual. In this case, though it seemed as though the some of the labels were confusing/contradictory and the only way to debug was to resort to the CLI. Setup was simple. Install using repo.

- chkconfig backuppc on
- service backuppc start
- service httpd start
- cd /etc/BackupPC
- htpasswd -cmb apache.users backuppc backuppc
- /etc/BackupPC/config.pl is actually a valid Perl file. Configure as required
- su -s /bin/bash backuppc
- ssh-keygen -t dsa
- ssh-copy-id -i .ssh/id_dsa.pub root@host.domain.com

It uses DNS to resolve 'viable hosts' and NetBIOS multicast thereafter. If all else fails though there is a fallback option to change the resolution mechanism via the config file via changes in parameter for the 'nmblookup' command. May need to change MAIL environment variable since by default 'backuppc' account has no configuration files and environment variables setup (I had to because I switched from ~/Maildir to /var/log/mail/*). Use /etc/aliases to forward email to another address. Documentation needs a bit of work. Some parts are skimmed over while others are quite verbose. Had some protocol mismatch issues when backup process was initiated. rsync man page indicated that it may be related to configuration files that may have been located (ssh remotehost /bin/true > out.dat to debug. If it contains non-zero material than obviously there are issues that need to be fixed. You see an error relating to only a certain amount of data being able to be received in the /var/log/BackupPC/* log file/s which is also viewable in the web interface.)

Massive performance issues on an i3 with 4GB once the backups started. The desktop environment actually began to suffer significant latency issues with the mouse cursor actually skipping halfway across the screen a number of times. Had to kill process eventually. Looked at other options such as using a different protocol and changing protocol (other than rsync). Thought about using a solution based on cpulimit, http://cpulimit.sourceforge.net/ that I built a while back (to deal with a similar issues with bacula and other pieces of software) which would basically act like an ABS brake on CPU utliisation and also automatically changing priority for the process via scripts. Further research indicated that this issue has been alleviated or fixed in subsequent revisions of BackupPC though.

Bacula

Prefer this solution over others because while its not perfect its still not a full blown backup solution that can can be unwieldly to deal with. I remember using bacula and there used to be some inexplicable errors in the database catalogue as well as some backup failures that couldn't be explained without delving overly deep into logfiles. Over time I figured out how to deal with them and achieved a perfect backup schedule but to be honest I just wanted a guarantee to know that it would work. One thing I did like about it though was the bconsole CLI interface. Single point from which to deal with mount/unmount/restore/backup of data.


Lightweight Options

Considered other lightweight (and even desktop) options such as rdiff-backup/backintime (basically a bunch of scripts) but a bit unwieldly and also didn't have the logging and diagnostics that BackupPC/bacula and other systems had.

Cloud/Filesystem Options

Have thought about cloud based and filesystem based solutions but have backed away for security/bandwidth reasons and would prefer to not to rely on the filesystem only.


Amanda

Remember Amanda from a while back. Had half configured it previously (basic setup in an experimental environment for possible use in production). This time I decided to do a more complete setup with 'virtual tapes/slots' in a 'virtual multi tape changer machine' setup. Installed using repos and copied relevant xinetd.* file to xinetd.d and ran the following as indicated in crontab sample file.

amcheck -m DailySet1
amdump DailySet1

Had "amanda client 10080 ack timeout fedora" errors. Packaging was slack. Provisions weren't made in xinetd.d/* files in order to properly locate amindex and other file/s causing port 10080 service not to be started. Perhaps it was just the 64-bit version?


Need to create 'virtual tapes/slots' under '/var/amanda/vtapes/slot?'
/dumps/amanda used as temporary storage prior to dumping to (in this case virtual) tape.

amtape
amlabel

  
/var/lib/amanda/.amandahosts works in a similar way to .rhosts bypass file to control who and which servers can backup/restore.


'strings /var/amanda/vtapes/slot0/* | less' gives you
'dd if=* bs=32k skip=1 | gzip -dc | tar -xpGf -'

Not pretty, even if you're doing it the 'proper way'.

amrecover
amrestore


As an aside, I restored to the /tmp directory. Somehow the permissions were erroneous which led to issues with Gnome (dealt with by setting correct permissions on /tmp directory). All the more reason to setup a seperate restoration area.


Logging layout could be streamlined. Spread out over a many different files and directories. Makes it easier to spot a particular time frame but complicates things. 'ls -al/multitail' are your friends here.


Long in the tooth and it shows. However, there does seem to be an effort to modernise judging by the website/Wiki. Zmanda (updated version of amanda with a web based management console) should definitely be at the back of your mind if you ever think of about using amanda.





at
Labels: , , , , ,
Subscribe to: Posts (Atom)

List Biggest Employers in New Zealand, Random Stuff, and More

#################### START new-zealand START #################### !!!!!!!!!!!!!!!!!!!! DATE Thu 13 Aug 12:54:23 AEST 2026 DATE !!!!!!!!!...