More Automated Research/Analysis

I've been examining the 'Automated Research/Analysis' concept further (especially in the context of real time analysis) so that I can design/build a prototype. One of the problems that was faced is the actual filtering of the actual content an underlying file format itself. As I've seen myself on a number of occassions depending on the format recovering the original data can be difficult if not impossible depending on how the developers created it (possibly compressed, packed, encrypted, has a proprietary format, or something else strange was done with it...) Moreover, depending on the nature of the programs data corruption or just mis-alignment (better word may be inconsistency)(through inadequate collection of data, incompetence, inadequte survey design, etc...) of data is possible. In the context of both quantitative and qualititive analysis 'outliers' are always a possibility but as the field of statistics is reasonably well explored (and predictive analysis)(although I admit it's still very much evolving) so I don't see too much of a problem in most circumstances (Still fleshing this out... My thoughts may change after seeing more data.).

Data collection can be any means. I've been playing around with speech recognition technology for a long while and it has come a long way (look at what they've done with Echelon), video surveillance technology has made significant strides (there are a group of researchers in Adelaide who are working on technology that allows you to scan and track a target in real time across multiple cameras though there are still some criticisms of facial recognition technology), and oscillators and data acquisition cards can be had for less than four/three figures from any number of sources and electronic stores.

http://en.wikipedia.org/wiki/Echelon_(signals_intelligence)
http://en.wikipedia.org/wiki/Onyx_(interception_system)
http://en.wikipedia.org/wiki/Frenchelon
http://en.wikipedia.org/wiki/ADVISE
http://arstechnica.com/security/2007/09/the-demise-of-advise-dhs-data-mine-boarded-up/
http://www.fas.org/irp/world/russia/soud/index.html
http://www.cvni.net/radio/nsnl/nsnl021/nsnl21ewar.html
http://en.wikipedia.org/wiki/Facial_recognition_system#Criticisms

After that it's a question of extracting useable/maleable data from the digital representation of the physical phenomena. One part of this may involve an intermediate data format. For instance, in the context of search engines data is often converted to a text format or otherwise a performance optimised, resilient, binary format that allows you to determine whether language in two text strings have similar taxonomy/meaning/context (whether across languages or inside a single language). In the context of facial recognition and images, you may use particular landmarks, shapes, and ratios to determine whether you may have something of interest...

http://en.wikipedia.org/wiki/Data_mining
https://en.wikipedia.org/wiki/Stemming
http://en.wikipedia.org/wiki/Conflation
http://en.wikipedia.org/wiki/Facial_recognition_system#Traditional

From the quantitative (even the qualititive side as well if we factor in taxonomy and semantic variations) analysis side a lot of the concepts that we require are already here. Determing relationships between sets of data is something that is done manually at high school level via algebra with more complex analysis of curves via calculus done at later high school and University level (complex graphing calculators were often used which could automatically define limited relationships between sets of data).

http://en.wikipedia.org/wiki/Student%27s_t-test
http://vassarstats.net/textbook/ch11pt1.html
http://en.wikipedia.org/wiki/Analysis_of_variance

http://en.wikipedia.org/wiki/Linear_algebra
http://en.wikipedia.org/wiki/Quadratic_equation
http://en.wikipedia.org/wiki/Linear_least_squares_%28mathematics%29

http://en.wikipedia.org/wiki/Fourier_transform
http://en.wikipedia.org/wiki/Fast_Fourier_transform
http://en.wikipedia.org/wiki/Fourier_series

Even if it isn't possible to determine a theory which works for the entire range of data it may still be possible to put together theories in series which include boundaries on which the data doesn't 'quite correlate'. For example, in Physics there is something known as the, 'Grand Unified Theory' which is a theory which attempts to model supposedly independent interactions, symmetries, and coupling constants into a single theory.

http://en.wikipedia.org/wiki/Grand_Unified_Theory
http://certifiedwaif.livejournal.com/389422.html

From this base it should be clear that we can lift the base and use it to work on all sorts of automated forms of analysis and research.

If we look at law enforcement/surveillance we have a history of real time facial recognition systems (which have had their fair share of criticism). But if we think about this further we don't need necessarily require real time analysis nor perfect facial recognition (I'm thinking about automated crime reporting rather than tracking people). If we are able to capture particular movements (literally and figuratively) then we can have a general idea of where a suspect is and what crime they have committed. For instance, if we look at the human body and examine a punching movement we have an arm (which is generally about 1/2 to 1/3 the length of the body from the tip of the hand to the top of the shoulder, has a hand that will generally be flesh coloured) which is moving at certain critical velocities with reference to the body (a punching movement will generally go up or across. Clothes are generally of a single colour and of very similar shapes which allow you to distinguish the body while the head is generally uncovered which will allow you to correctly identify most people (unless they are nude though there are laws against that (or they are wearing flesh covered clothing)) and in reference to the target body. Then it's simply a matter of periodically watching for specific relationships to show up particular data sets. For instance, if the punching movement was determined the by equation y = jx^2 + b + a^5cbz^3 and we found this particular relationship showing up in multiple points in our data set then we can be fairly sure that this particular event occurred. Obviously, we can extend the concept further to allow for unique equations that can represent other actions as well. As Quantum and High Performance Computing technology progresses the possiblity of real time analysis and a machine which roughly replicates 'The Machine' from 'Person of Interest' quickly becomes a reality. All you have to do is integrate surveillance, GPS, and time based information and you would be away. Time to think some more and flesh out other details....

http://www.businessweek.com/articles/2012-09-13/watch-out-google-facebook-s-social-search-is-coming#r=shared

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/

Automated Research and Some More Wireless Foo

One of the things that I sometimes wonder is whether or not Australia is lucky or unlucky in being so geographically isolated. Some industries (Airline and Motoring are probably the most recent and notable examples that come to mind) have not been provided with a level playing field when compared with the rest of the world. They've either been playing with one hand tied behind their back or others have. They haven't learnt to adapt. I think this is what happened with another industry that I recently had interactions with. Overseas, firms in this particular sector have advanced significantly over local competitors most particuarly in the clever use of technology. I've been thinking about this issue more though. Some of the concepts mentioned in my previous report (Convergence Effect)(such as 'Automated Research', http://dtbnguyen.blogspot.com/2012/03/automated-research.html) have had significant resources allocated to them but it's becoming clearer to me just how far we can push it. Data mining, searching, statistical analysis, and report writing are all processes that can be significantly automated with sufficient knowledge and application of technology (I'll provide a more detailed framework on how to achieve this in the context of Security analysis in my report on, 'Cloud and Internet Security' (about 557+ pages/155K+ words now) but obviously it can be used in other contexts as well).

http://www.businessweek.com/articles/2012-09-13/watch-out-google-facebook-s-social-search-is-coming#r=shared

In my recent experiments I've obviously been experimenting with various different USB wireless adapters. One of them was the D-Link DWA-125 A3 (rt5370sta), another was the NetGear WNDA4100 (rt3573sta, you need to modify the driver to add support for the new chipset, http://ubuntuforums.org/showthread.php?t=1942689), while another was Jaycar's YN8306 (all Ralink chipets funnily enough with decent Linux support, http://www.ralinktech.com/en/04_support/support.php?sn=501). Clearly, after doing a lot of experimental work I've discovered a lot about antenna technology as well as the current state of wireless technology out there. One of the things I didn't realise is how drastic some of the differences could be between revisions of both adapter and chipset. My original purchase of the YN8306 seemed to be reasonable but the performance was nothing spectacular (Ralink 3070L). After discovering the USB port was somewhat loose (it was constantly losing grip of the cable) I exchanged it for another one but discovered something similar so this time I opened up the device and made some modifications (extra solder to bolster the strength of the USB port). While I was in there I noticed that the chipset had been upgraded to the Ralink 3072L. My thoughts regarding the Rev A were that it was good but didn't have the level of sensitivity that I thought it would. Rev B was a bit different though. Based on initial observations it may be at least 20% stronger with regards to sensitivity when compared to the Rev A and transfer rates were far less likely to fluctuate. A significantly better device overall.

With my recent incursions into antenna design I should probably extrapolate on what exactly I mean. I've torn apart anything and everything that I could possibly get my hands on and have even resorted to modifying television equipment in combination with broad range amplifiers to boost signal strength (I know it's not 'tuned' to the correct frequency but I just wanted to see what would happen.). What have I learnt? With enough imagination a lot of weird and wonderful things are possible. While hooking up my wireless adapter to the home television antenna (yes, the one on the roof) prooved to be an interesting exercise it only yielded slightly better results than having no antenna at all. A broad spectrum amplification device brought it up to slightly better than the originally included 5dBi antenna and forays into using antenna systems extracted from a dead router only yielded neglible benefits. A 16dBi antenna purchased during a local swap meet did result in a noticeable (though still small) gain though. Using a sieve (it's actually quite difficult to find suitable parabolic/spherical sized shapes in the house (unless you're willing to tear up potentially valuable goods such as headlamps). I had to go out and purchase a sieve for $2.80 at a local 'value store') only helped in certain configurations (I later found out an antenna from an old eMac was actually able to significantly boost the sensitivity of the DWA-125 A3 depending on it's polarity.) unless I switched to a more forgiving Gregorian Satellite Dish design (I used another smaller sieve and some elastic bands to replicate the design).

http://en.wikipedia.org/wiki/Satellite_dish

I even tried extremely crude Cantenna and Yagi style antenna designs (I'm waiting for an order to come in). The difference was that I made mine out of solf drink bottles with the bottom cut out wrapped with aluminium foil and also something I call 'Ruler-Yagi'. Basically, an antenna that was made out of a wooden ruler (boom), gold paperclips (it's suprising what you can find these days at Officeworks), blu-tack, and connected to a wireless adapter using alligator clips. There was a significant boost in signal (50%) when my 'Softenna' was directed properly and surprisingly there was an actual gain for 'Ruler-Yagi' but once again it was only minimal (10-20% signal sensitivity increase).

In the context of cyber-intelligence/operations it interests me to see what sort of antenna systems we could possibly pursue. There is a high likelihood that given the right materials high gain antennas could be embedded or inserted inside of suit cases or even other devices with connectivity being provided via cable (or hopefully) wireless. For example, a supposedly long life battery could be modified to provide a genuinely higher gain antenna provided space/cells were cleared from the internal cavity. Alignment could be facilitated by an internal lens/reflective camera system.

- as usual thanks to all of the individuals and groups who purchase and use my goods and services
http://sites.google.com/site/dtbnguyen/
http://dtbnguyen.blogspot.com.au/

Automated Research

I never thought that I would be writing about this so soon. Apparently, the US Government announced an initiative  today/yesterday (depending on your location) that they would be pursuing an initiative into 'Big Data' and a form of automated research that I had been exploring in my 'Convergence' document (still working on it if you're wondering). While its certainly not a guarantee of success in this area I believe the implications may be far wider than any of us may ever imagine. Having algorithms which are capable of searching for masses of data, looking at it, extracting relationships and possibly other insights into it could result in an avalanche of human knowledge and discovery.

http://news.cnet.com/8301-11386_3-57406484-76/why-science-really-needs-big-data/
http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf

### Start Quote ####

OBAMA ADMINISTRATION UNVEILS BIG DATA INITIATIVE: ANNOUNCES $200 MILLION IN NEW R&D INVESTMENTS Aiming to make the most of the fast-growing volume of digital data, the Obama Administration today announced a Big Data Research and Development Initiative. By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help solve some the Nation's most pressing challenges. To launch the initiative, six Federal departments and agencies today announced more than $200 million in new commitments that, together, promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data. In the same way that past Federal investments in information-technology R&D led to dramatic advances in supercomputing and the creation of the Internet, the initiative we are launching today promises to transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security, said Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy. To make the most of this opportunity, the White House Office of Science and Technology Policy (OSTP) in concert with several Federal departments and agencies created the Big Data Research and Development Initiative to: Advance state-of-the-art core technologies needed to collect, store, preserve, manage, analyze, and share huge quantities of data. Harness these technologies to accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning; and Expand the workforce needed to develop and use Big Data technologies.

### End Quote ####

- as usual thanks to all of the individuals and groups who purchase and use my goods and services

http://sites.google.com/site/dtbnguyen/

http://dtbnguyen.blogspot.com.au/